The Birth of the Agile Manifesto
The Agile Manifesto was the result of a decade of cross-fertilization among many related incremental and iterative methods for software development, culminating in a seminal few days in February 2001 where the basic principles of the Manifesto were hammered out. Those principles addressed many of the challenges of software development of the 1980s and 1990s – but they weren’t the first time anyone did anything “agile”, and they aren’t necessarily the last word on how to achieve agility. A little history can be instructive.
What is Development, Anyways?
In the very early years of software, people just coded. And debugged. And coded and debugged some more. In the mid-1950s, people started to think about software development as more than just coding and debugging. The “stagewise” model appeared, in which requirements and design were defined as activities distinct from coding. Shortly, testing and debugging were considered as separate activities also. This new perspective allowed developers to imagine methods and techniques for more effective design and testing. Coding wasn’t the only skill any more.
There wasn’t yet any rule that all requirements had to be written before any coding could occur. Incremental development (taking one feature at a time completely through the requirements-design-code-test sequence) was not that unusual. In the 1960s, Project Mercury at NASA was running half-day iterations and even used test-first development. However, there was no common language to describe or compare different development methods. Without a common language, good methods didn’t spread rapidly.
The Waterfall Method Falls On Our Heads
In 1970, Winston Royce first described in print what was later dubbed the “waterfall” model. Royce’s original article included loopbacks to earlier stages, and warned that the model was quite risky without this iteration. Unfortunately Royce’s definition was very shortly re-interpreted as a strict phase-gate process without those loopbacks. The U.S. Department of Defense started requiring contractors to use this strict interpretation, and over the next decade the waterfall model rapidly became a standard.
Iterative Methods are Formally Described
Meanwhile, the concepts of developing in increments and iterating towards a solution continued to ferment. The 1980s saw the rise of language and definitions which enabled full discussion of these ideas, such as Barry Boehm’s definition of the spiral model in 1986, and Tom Gilb’s 1988 book on Evolutionary Delivery.
In the 1990s, an active international group of thinkers tried a wide variety of incremental and/or iterative methods. Evolutionary Delivery was joined by DSDM, the Rational Unified Process, Staged Delivery, Feature-Driven Development, Crystal, Scrum, XP, and others. Projects using these methods very frequently ran much faster and more successfully than projects using waterfall. Clearly something important was going on.
The Agile Manifesto
In 2001, 17 thought leaders gathered to discuss this variety of methods and why they worked. These leaders hammered out the Agile Manifesto and the 12 Principles of Agile, which were soon being cited as the definitive description of agile. They were a good description of agile as the authors of the Manifesto understood it at the time. A few of the familiar practices we know today, such as a regular cadence, didn’t appear reliably until a few years after the Agile Manifesto, but most of the key features were there. When applied, agile methods began producing excellent, sometimes even spectacular, results. Agile projects could predict their end dates, adapt rapidly to changing requirements, and produce reliable products.
Agile Craftsmanship – Context Matters
By 2005, Agile had become the new silver bullet of software engineering. However, many attempts to adopt Agile were unsuccessful. Often this was because the teams didn’t really understand why Agile worked. The 12 Principles had blossomed into a craft, with very specific instructions: do it exactly this way and it will work.
These instructions worked in contexts which were very similar to those of the original authors. Specifically, they worked well for single-team projects in fields where it was easy to release new software and the customer was readily accessible. That’s a good description of in-house IT development (the original context for XP) and for much of the early web development. But it’s not necessarily a good description of all software projects everywhere. For instance, in-house embedded software development – do you think of the end user as the customer, or is the customer really the hardware which the software drives? How do you collaborate with hardware development?
Agile as Engineering – Why Agile Works
In the second half of the 2000s, a few people (notably Mary Poppendieck, Dean Leffingwell, and Alan Shalloway) realized that the science underlying Lean also applies to Agile – and that understanding the science provides tools for adapting Agile to different contexts. Essentially, Agile splits up the development into small, relatively independent batches of work. These batches and their behavior can be managed using standard methods from queueing theory, which is a body of knowledge nearly 100 years old on how to optimize a workflow made of batches. More on this in What Makes Agile Work.
Here’s an example. One of the common mistakes in large-scale Agile adoptions is declaring stories “done” when the team sends their sprint results to system test, and blithely continuing with a new sprint as if they are all done. But system test does find defects, the team has to be interrupted, and their measured velocity isn’t predicting their capacity correctly. Queueing theory explains this – schedules are predictable only when each batch is truly complete before being considered “done”. See more (and what you can do about it) in this article: Fix Your Agile Project by Taking A Systems View.
Moving into the Future
Understanding queueing theory enables us to adapt Agile to more challenging contexts, such as large multi-team organizations. This also enables us to think about how to achieve agility in other business contexts, such as hardware development. Applying Agile software methods verbatim in fields outside software often doesn’t work, but thinking about the batch structure and applying the same concepts to the new field can produce very positive results.
Learn More
For more on queueing theory and how it applies to Agile, see What is Queueing Theory and Why Should I Care. Or read Lean-Agile Software Development by Shalloway, Beaver, and Trott or Scaling Software Agility or any subsequent book by Dean Leffingwell.
Learn about agile methods in hardware development: Agile for Hardware and Agile Gets Physical.
When Agile doesn’t quite work for your software project:
- When Classic Agile Doesn’t Work – and What to Do About It
- Rapid Learning Cycles: Agile for Architecture and COTS Investigations
- Fix Your Agile Project by Taking A Systems View
References on the early history: “Iterative and Incremental Development: A Brief History” by Craig Larman and Victor Basili in IEEE Computer; June 2003. Managing Software Quality and Business Risk; by Martyn Ould, 1999; The Complete Guide to Software Testing by Bill Hetzel, 1988.
First version published January 2021.