May 10, 2006

Over the Waterfall

If you've ever been a developer on a significant PeopleSoft project—or just about any large development effort—you have probably worked within the Waterfall model.

What is Waterfall? It's the best-known, most widely used software development methodology. Under this model, the project proceeds in an orderly fashion from requirements analysis through design, coding, testing and deployment. As each phase is completed, the work flows down to the next phase, as in a waterfall. Key to this approach is the idea that a phase should be completed and documented before the next phase begins. For example, all design should be complete before one line of code is written.

If your company has done the popular thing and established a "Project Management Office," which in turn produced a large binder describing a development methodology named with a clever acronym, look at the binder and you'll see a version of Waterfall.

The Waterfall model was described in a 1970 paper by W.W. Royce (available here). Ironically, after introducing the basic Waterfall model, Royce wrote that "I believe in this concept, but the implementation described above is risky and invites failure." He then went on to propose fixes, including a more iterative approach (good) and more documentation (not necessarily so good). Unfortunately, his analysis of the model was forgotten and only the part that is "risky and invites failure" was adopted by the industry.

I'm not a fan. There are many practical problems with the Waterfall approach. Waiting for each phase to be fully completed and documented before proceeding means that the project periodically grinds to a halt. Relying on detailed documentation for communication between steps means that errors are copied and magnified as time goes on. Waterfall separates the teams, reducing communication. Scheduling is more difficult because a team can't begin its work until the previous phase is completed. Errors are not caught until late in the process, and there is no clear way of looping back to correct them. (Obviously, this is possible, but it violates the whole idea of Waterfall.)

Waterfall is probably the best method if all of the following are true:

• Your users know exactly what they need at the beginning of the project, and they won't change their minds later;
• The requirements analysis team translates these needs perfectly into a document;
• The design team translates the requirements perfectly into a design document;
• The programmers translate the design perfectly into code;
• All teams are available just when they are needed; and
• Your project is scheduled to start no later than, say, 1995.

Proponents of Waterfall point out that it's important to plan ahead, because an error caught late in the process is much more expensive to fix than one eliminated early by good analysis. That's not as important as it once was, but it's still a consideration. Therefore, time spent up front is a good investment. But the law of diminishing returns is seldom considered. Depending on your project manager, you may be faced with a proliferation of "visioning," scoping, analysis, design, specification, documentation and review steps that tend to drag on, squeezing the construction and testing phases into smaller and smaller windows. This is very dangerous.

There are limits to how good your design can be, no matter how long you spend on it. As I mentioned last week, it is often difficult to know exactly how to do something until you try to do it. It's common for me to begin a project, knowing exactly how I plan to do it, only to find during construction that my assumptions were wrong. It's like solving a maze but being asked to specify the correct path in advance.

So why is Waterfall so popular? Here are a few guesses. Budding project managers learn from experienced project managers, and not necessarily from previous projects or development experience. Old-timers (such as myself) who started out in the mainframe era—when it was difficult to experiment, and up-front design was critical—transfer their experience to current projects. Consulting firms "sell" Waterfall as a way to keep control over projects (and, incidentally, to sell lots of hours). Project managers feel overwhelmed by the idea of "cowboy coders" running off in all directions, and want a more manageable, predictable process. Finally, Waterfall provides an acceptable way of failing. "I used the industry standard methodology, Boss, so that's not the reason the project crashed."

The very good "Ward's Wiki" at c2.com says of Waterfall that it is "largely discredited, but still widely used." That's probably the bottom line. So how do we, as developers, work effectively within Waterfall? How do we reduce the risk inherent in the model? How do we manage the abrupt, inefficient stops and starts in the schedule? How do we get our work done? I'll suggest a couple of ideas next week.

Until next time...