Why Traditional Software Planning Fails

Mike Cohn, founding member of the Agile Alliance, eloquently describes the main reasons that software projects have failed (Cohn, 2006). Here are a few of the main ones:

Lateness is passed down the schedule – traditional software planning processes focus on the dependencies between activities. As a result, best case is that everything finishes on time, but because of dependencies, a slip in any one activity is sure to cause a slip in the entire schedule. What is worse, such slippages are not simply additive. Lateness on one task may ripple over to many other dependent tasks, making all of them late, multiplying the lateness by a factor of the number of dependencies in the system.

We ignore uncertainty – traditional software planning fails because it assumes initial requirements can lead to a perfect and complete specification, and that all unknowns can become knowable at the start of a project. It fails to take into account that customers may change their minds, come up with new requirements, or change requirements during the course of a project. The old school assumes that manufacturing methods can be applied to software development; but studies by the Standish Group (1995, 1998) and others conclusively show these methods don’t work, no matter how rigorously they are applied. Usually we are implementing a new idea or design, which has unknowns and risks, but we need to achieve exacting quality. It’s common sense: we don’t have perfect information, so expecting the perfect estimate or the perfect schedule is silly (I am quoting Eric Brechner, author of “Hard Code,” and Development Manager of Microsoft Office). It is an ill plan that cannot be changed.

Estimations become commitments – when we give an estimate for the completion of any task, implicit in that estimate is a probability of completing that estimate in that time. Armour (2002) points out that every estimate is a probability; but you cannot make commitments to probabilities. You make commitments to dates. Before making a commitment to a date, software teams must be given the chance to use solid estimation techniques such as wide band Delphi estimation to uncover assumptions and risks associated with a project, and develop risk management plans to define how the teams will handle change when (not if) it occurs in their projects (PMI, 2004). These risk management plans must be approved by senior management; otherwise teams are just signing up for failure.

Plans and Planning

Planning is a critical step in any undertaking. We want the problem to be understood as well as conditions allow, and to uncover as many assumptions as early as possible. Why do estimates often fail us for engineering projects, particularly software development? They fail because software engineering is a design activity, not a manufacturing one. There is a common sense understanding of the difference between plans and planning:

A good plan, violently executed now, is better than a perfect plan next week. - Patton
In preparing for battle I have always found that plans are useless, but planning is indispensable. – Eisenhower

No battle plan survives contact with the enemy – Colin Powell

Did you notice the quotes above are from military men? Software teams are equally at war, but it is a mistake to think they are at war with customers, the operational staff, or executive management. They’re not. They (and the customers, operational staff and executive management) are together at war with uncertainty. Agile project planning is all about recognizing that uncertainty exists, minimizing that uncertainty as much as possible, and installing tools and processes that manage that uncertainty and risk within the framework of the business goals and objectives.

Doing it better

In environments that change very slowly, traditional project management methods focus on a full understanding of the product through careful analysis and planning. Before any significant investment is made, the uncertainty is therefore reduced to a level where risk can be comfortably managed.

However, when business and technical requirements change rapidly, often because the business is continually redefined, this full understanding is impossible to achieve. Planning is critical, yet we must be cautious about what we do with the results of our planning phase, and be careful not to fall into the trap of believing we can have perfect estimation and perfect scheduling at the beginning of a project. There will always be a lot of variability in project scope at the beginning of a project, and there will always be change throughout the life cycle of the product. Turing award winners Barry Boehm, Kristen Nygaard, Ole-Johan Dahl, and Peter Naur have all espoused and developed these agile development ideas. We ignore their research at our peril.

Software is not a unique industry in having high uncertainty at the beginning; what other industries do is constantly reiterate and re-plan. They don’t make a big up-front plan and treat it as inviolate. Estimation is a tough but important problem, and we need to be thoughtful about doing it better, as long as we don’t confuse valuable planning with a rigid and brittle plan, or estimates with commitments.

Armour, Phillip. (2002). Ten Unmyths of Project Estimation. Communications of the ACM 45, no. 11155-18.

Brechner, Eric. (2008). Hard Code. Redmond: Microsoft Press.

Cohn, Mike. (2006). Agile Estimating and Planning. Upper Saddle River, NJ: Pearson Education, Inc.

PMI. (2004). A Guide to the Project Management Body of Knowledge, 3rd Ed. Newtown Square: Project Management Institute.

Standish Group International. (1994). The Chaos Report. The Standish Group.