The Birth of the Agile Manifesto
The Agile Manifesto was the result of a decade of cross-fertilization among many related incremental and iterative methods for software development, culminating in a seminal few days in February 2001 where the basic principles of the Manifesto were hammered out. Those principles addressed many of the challenges of software development of the 1980s and 1990s – but they weren’t the first time anyone did anything “agile”, and they aren’t necessarily the last word on how to achieve agility. A little history can be instructive.
What is Development, Anyways?
In the very early years of software, people just coded. And debugged. And coded and debugged some more. In the mid-1950s, people started to think about software development as more than just coding and debugging. The “stagewise” model appeared, in which requirements and design were defined as activities distinct from coding. Shortly, testing and debugging were considered as separate activities also. This new perspective allowed developers to imagine methods and techniques for more effective design and testing. Coding wasn’t the only skill any more.
There wasn’t yet any rule that all requirements had to be written before any coding could occur. Incremental development (taking one feature at a time completely through the requirements-design-code-test sequence) was not that unusual. In the 1960s, Project Mercury at NASA was running half-day iterations and even used test-first development. However, there was no common language to describe or compare different development methods. Good methods couldn’t spread rapidly because we couldn’t talk about them coherently.
The Waterfall Method Falls On Our Heads
In 1970, Winston Royce first described in print what was later dubbed the “waterfall” model. Royce’s original article included loopbacks to earlier stages, and warned that the model was quite risky without this iteration. Unfortunately Royce’s definition was very shortly re-interpreted as a strict phase-gate process without those loopbacks. The U.S. government started requiring contractors to use this stricter model, and over the next decade the waterfall model rapidly became a standard.
Iterative Methods are Formally Described
Meanwhile, the concepts of developing in increments and iterating towards a solution continued to ferment. The 1980s saw the rise of language and definitions which enabled full discussion of these ideas, such as Barry Boehm’s definition of the spiral model in 1986, and Tom Gilb’s 1988 book on Evolutionary Delivery.
In the 1990s, an active international group of thinkers tried a wide variety of incremental and/or iterative methods. Evolutionary Delivery was joined by DSDM, the Rational Unified Process, Staged Delivery, Feature-Driven Development, Crystal, Scrum, XP, and others. Projects using these methods very frequently ran much faster and more successfully than projects using waterfall. Clearly something important was going on.
The Agile Manifesto
In 2001, 17 thought leaders gathered to discuss this variety of methods and why they worked. These leaders hammered out the Agile Manifesto and the 12 Principles of Agile, which were soon being cited as the definitive description of agile. They were a good description of agile as the authors of the Manifesto understood it at the time. Some of the familiar practices we know today, such as a regular cadence, didn’t appear reliably until a few years after the Agile Manifesto, but most of the key features were there. When applied, agile methods began producing excellent, sometimes even spectacular, results. Agile projects could predict their end dates, adapt rapidly to changing requirements, and produce reliable products.
Agile Craftsmanship – Context Matters
By 2005, Agile had become the new silver bullet of software engineering. However, many attempts to adopt Agile were unsuccessful. Often this was because the teams didn’t really understand why Agile worked. The 12 Principles had blossomed into a craft, with very specific instructions: do it exactly this way and it will work.
The instructions worked for contexts which were very similar to that of the original authors. Specifically, for single-team projects in fields where it was easy to release new software and the customer was readily accessible. That’s a good description of in-house IT development (the original context for XP) and for much of the early web development. But it’s not necessarily a good description of all software everywhere. For instance, in-house embedded software development – do you think of the end user as the customer, or is the customer really the hardware which the software drives? How do you collaborate with hardware?
Agile as Engineering – Why Agile Works
In the second half of the 2000s, people such as Mary Poppendieck, Dean Leffingwell, and Alan Shalloway realized that there is a scientific way to look at Agile, which can help adapt Agile to different contexts. Essentially, Agile splits up the development into small, relatively independent batches of work. These batches and their behavior can be managed using standard methods from “queueing theory”, which is a body of knowledge nearly 100 years old on how to optimize a workflow made of batches. For instance, queueing theory tells us that schedule predictability depends on each batch being truly complete before being considered “done”. This explains one of the more common mistakes in Agile adoptions: declaring stories “done” when the team submits their sprint results to system test, and blithely continuing with a new sprint as if system test would not find any defects at all. The team’s velocity will be inaccurate because the batch isn’t truly done. See more (and what to do about it) in this article: Fix Your Agile Project by Taking A Systems View and What Makes Agile Work.
Moving into the Future
Understanding queueing theory enables us to adapt Agile to more challenging contexts, such as large multi-team organizations. This also enables us to think about how to achieve agility in other business contexts. Applying Agile software methods directly often doesn’t work, but thinking about the batch structure and applying the same concepts to the new field can produce very positive results.
For more on queueing theory and how it applies to Agile, see What is Queueing Theory and Why Should I Care. Or read Lean-Agile Software Development by Shalloway, Beaver, and Trott or Scaling Software Agility or any subsequent book by Dean Leffingwell.
References on the early history: “Iterative and Incremental Development: A Brief History” by Craig Larman and Victor Basili in IEEE Computer; June 2003. Managing Software Quality and Business Risk; by Martyn Ould, 1999; The Complete Guide to Software Testing by Bill Hetzel, 1988.