Agile vs. Waterfall: Choosing the Right Methodology

Project management has existed in various forms throughout human history. Everything from organizing more efficient hunting and gathering techniques to building a “smart city” requires implementing methodologies through a specific process (or set of processes).

Although the tools are more sophisticated now, and we’re continuing to improve on our ancestral successes, the failure rate for IT projects still hovers around the 50% mark, and the “lack of clearly defined objectives and milestones to measure progress” is the number one reason cited for the failure.

Given the pervasiveness of technology, all companies are essentially tech companies in some way, shape or form. For this reason, enterprises are inevitably tasked with undertaking IT projects. These can be as simple as building an app or as massive as redesigning their database infrastructure (e.g., moving away from a legacy system and to cloud-based services).

Within the software development world, Agile is the dominant methodology. Meanwhile, it’s predecessor, Waterfall, is still present, but lurking in Agile’s shadow. The truth is, both have their place within project management. Indeed, depending on the size and scope of the project, they may be used in tandem.

Are there projects where Agile makes better sense to use? Yes. The same can be said for Waterfall. But, it may not always be easy to determine when to use which. The guide below will give you a brief set of heuristics for deciding when to use one versus the other (or both).

Waterfall: Over 60 Years Later

The Waterfall method was conceived by Herbert D. Benington over 60 years ago and was the primary method of software engineering prior to the advent of Agile. In short, there are 6 phases, each requiring completion before the next phase can begin:

  1. Preliminary Design: requirements are documented as to what the software should “do” by the time it becomes a finalized deliverable.
  2. Detailed Design: system architecture, including hardware and software specifications, is developed.
  3. Coding: the requirements and design are codified into units (small programs), which are then incorporated into the next phase.
  4. Unit Testing: each unit is tested for bugs, low performance, etc., and once optimized they are integrated into the more extensive production environment; once the unit testing process is complete, then the entire implementation is tested as well.
  5. Deployment: the software is released to the client or customer.
  6. Maintenance: debugging, updating features, and other maintenance processes are completed throughout the software life cycle.

Different authors may term each of the phases something slightly differently, e.g., using “operations” instead of “maintenance.” Also, there are Waterfall variations; but, the concept is still the same: Waterfall moves through a specific set of linear processes where each phase is completed before moving onto the next activity.

Benefits of Waterfall

Developers and end users agree, in advance, on functional requirements.

Throughout the project, the requirements are kept stable and unchanging. The Waterfall method is beneficial when there aren’t any rapidly shifting functional requirements present. This method is handy and helpful when the system’s requirements need to be designed and analyzed only once, correctly, and will stay the same for an extended period of time.

The application can be developed with upfront knowledge of all requirements and system variables.

This aspect, in many cases, will help development teams avoid the core restructuring of applications as the project progresses. In particular, this is of benefit for smaller projects, where requirements can be easily understood by all stakeholders in advance of development.

Measurable progress helps contain the Iron Triangle: Cost, Scope, Time.

As stated in the opening paragraph, measuring progress (or the lack of performing this function) is a prime reason as to why projects fail. Given the frontloading of detailed requirements as agreed upon between the developers and end users, the Waterfall method creates a comprehensive scope of work at the beginning of the project.

Milestones are clearly defined. Since it’s expected that these requirements are unchanging, progress in relation to cost, scope, and time is more easily tracked and measured throughout the project life cycle.

Disadvantages of Waterfall

Clients or customers may not have a clear selection of requirements.

Waterfall works well when the clients or customers have a clear set of requirements. Certainly, these can be hammered out through a stakeholder meeting during the initiation phase of project management. But, consumer demand changes quickly and Waterfall has a “no turning back from here” lock-in for its processes. So, client requirements can neither be vague nor lacking in detail. There’s minimal opportunity for revisions once you arrive at the 3rd step (Coding).

Feedback can’t be solicited until later in development.

Due to the sequential nature of Waterfall, once the software is developed and in the testing stage, iterations based on user feedback are difficult (if not impossible) to incorporate into the end-product.

Aspects of the product that weren’t thoroughly thought out, or for which requirements have shifted during production, will be more difficult to fix if user feedback is negative. This increases product risk as end-users may reject all or part of the finished software.

It’s not suitable for long term projects.

Due to Waterfall’s emphasis on detailed planning before development, the planning stage can take an exceptionally long time. This aspect of Waterfall makes it a poor fit for long-term projects where there is a risk that requirements will change during development.

Extensive planning for a software that doesn’t meet end-user needs can be an expensive mistake if assumptions about the product or market are misplaced.

When to Use Waterfall

The Waterfall method is optimal for projects that

  1. Have clear-cut requirements.
  2. Are shorter in duration.
  3. Have a low risk of new technology which can negatively impact the original requirements.

Agile: It Started at a Ski Lodge

Agile’s core tenants, as expressed in The Manifesto for Agile Software Development, came into being on February 13th, 2001. Thought-leaders from various process-focused firms convened at The Lodge at Snowbird. The participants included representatives from SCRUM, DSDM, Feature-Driven Development, Crystal, and Pragmatic Programming (among others). The Manifesto for Agile Software Development includes several fundamental principles:

  1. Agile embodies a set of principles through which cross-functional teams produce software requirements.
  2. Adaptive planning and evolutionary phases of application development are foundational processes.
  3. Flexible responses to changing requirements and the rapid delivery of systems are key measurements of success.

The above is frequently translated into the following:

  1. Collaborate with end users in short sprints and iterative cycles to develop applications.
  2. Respond to functional specification changes in real-time.
  3. Quickly deploy iterative releases of the system out the door (in weeks rather than months or years).

Agile development has made application development far simpler in the rapidly evolving world of functional requirements. However, it’s not a magic bullet. Notably, rapid-cycle iterative development is not always the best fit for every project.

Benefits of Agile

Feedback can be incorporated into the product faster.

Agile’s regular release cycle allows for continual feedback from the actual end-users. By providing working demos, the Agile method can reveal unexpected issues and generate ideas for future iterations. The development team can then use these insights to improve the likelihood of the product’s success. This feedback loop improves product alignment for both business value and end-user value.

Quick determination of actual business value vs. perceived business value.

This benefit, in part, is driven by the Minimum Viable Product (MVP) release. The first iteration of the application must focus on delivering business value that substantiates on-going application development.

Transparency of functional requirements.

The Agile method has embedded characteristics that ensure functional requirements are visible to end users throughout the iterations of application development. There are no “black holes” of requirement specifications.

Predictable delivery time frames.

Agile encourages the application development cycles to be “time-boxed” into Sprints that last from one to four weeks in duration. The length of the Sprint cycle is declared in advance, and end users then have the assurance that application iterations will be delivered within predictable time frames.

Improved application quality.

Agile-based project delivery is divided into units of activity that include cycles of collaboration, development, testing, and deployment. Since these cycles occur on a repeated, iterated basis, applications undergo constant review for improvements and quality assurance.

Disadvantages of Agile

Agile is an approach; it’s not a systemized process.

Certainly, enterprises can (and do) apply their own systemization to the Agile approach, but that’s not what it was designed for. The goal is more flexibility in producing and maintaining the end product. Thus, scope creep, overestimating and underestimating costs, and losing control of the project schedule are problems that can occur.

Clear-cut requirements are still needed.

While speeding towards an MVP through iterative cycles is a better way to ensure that the end-user is receiving an optimal product, strong documentation and end-user specifications are required to decrease the probability of project failure.

With less emphasis on planning and documentation, and having different cohorts produce each unit in a non-sequential manner, using a pure Agile methodology tends to cause project fragmentation. As a result, the project goals and objectives can easily go off the intended path.

Resource allocation issues are more likely to arise.

Without the usual concerted effort focused on detailed planning and documentation, it’s much more difficult to reallocate human resources should developers and engineers leave in the middle of the project. Less experienced developers will be hard pressed to pick up where the others have left off since documentation can be scant.

Additionally, progress metrics are an essential component of determining the who, what, where, when, how, and why of all resources. The “fix and implement things as we go” subverts the resource allocation process which can easily lead to scrambling to meet the “just in time” reach for resources and, in turn, sets the stage for costs spiraling out of control.

When to Use Agile

Agile is most beneficial to use when projects:

  1. Focus on dynamic technological environments where adding new features or constant changes based on end-user feedback are fundamental to the business model (e.g., app development, new product development, etc.).
  2. Do not have strict compliance requirements such as dealing with health-related or financial data whereby navigating through bureaucratic demands are non-negotiable.
  3. Have vague end-user requirements due to the client not having a complete vision of the final product; this is particularly true for new product development.
  4. Do not have overly stringent time and cost constraints.

Agile and Waterfall? The Best of Both?

So, what’s the best option for your project? The “Best of Both” is a common solution for balancing out the disadvantages of solely relying on either. The list below provides examples of application development and deployment scenarios where a mix of Agile and Waterfall techniques will produce the best delivery results.

Is the product well defined?

If the product is not well defined, and further feedback is needed from the market to validate requirements, Agile is the best choice. An Agile approach that incorporates constant input from end users will allow those who will actually use the product to guide requirements, thereby increasing the chance of project success.

Waterfall is to be avoided where requirements are less clearly defined, or likely to change. However, the project can be planned at a high level using Waterfall, and then executed and controlled (as much as is possible for Agile) via the Agile methodology.

What is the budget?

The first two phases of Waterfall (Preliminary Design and Detailed Design) provide the necessary steps to estimate overall project scope. For projects that are competing with others for a limited company budget, IT teams often want to get their budget established early and upfront. Agile then becomes an excellent method to iterate versions of the larger system once the overall budgetary requirements are estimated to a reasonably accurate extent.

How long is the project?

Waterfall can be a good fit for shorter projects, where the project timeframe limits the risk that requirements will change. For longer projects, the length of planning required and the possibility of underestimating or overestimating feature importance can add unnecessary cost to a development project. Again, Waterfall can be implemented during the initiation and planning stages with the caveat that product development will deploy Agile.

What is being developed?

SAP, Oracle, and other enterprise-class platforms are not commonly deployed through iterative development. Waterfall is the front-runner in this scenario. However, Agile is an excellent method for the development of adjunct applications such as business intelligence and dashboard systems.

Extract, Transform, and Load Data Systems

As is the case with enterprise-wide applications, ETL-driven projects tend toward deployment in “one-big-leap.” Loading data into a repository still in development will often produce no useful system capabilities. Waterfall’s 6 step progression provides a solid path toward getting the data in one place and with the right structure. Yet, Agile techniques can be the key to developing responsive end-user systems that meet the immediate needs of data visibility.

Large-scale E-commerce Applications

The deployment of large or intricate E-commerce stores can be compared to that of their large brick-and-mortar counterparts. The store’s interior has to be completely built-out, and the POS systems must be operating correctly. Yet, the actual “storefront” requirements can shift quickly.

As an example, there may be new products to feature or cross-selling system features to adapt. Agile is a perfect fit for the development of customized user interfaces that must undergo a swift change-and-adaptation phase, while the larger and less flexible e-commerce ecosystem is developed using Waterfall.

Takeaway Facts

Research from the Project Management Institute (PMI) suggests that organizations embracing Agile methodology are more likely to succeed with IT projects. But this isn’t to say that Waterfall is dead.

Waterfall is not “dead.” It may be more than sixty years old, but it still fits great in specific scenarios.

As a matter of fact, it’s still in use for the systems previously discussed. For example, the Waterfall method is robust when applied to enterprise-wide application deployments and with multiple organizations involved.

Agile is the perfect fit for circumstances where requirements are dynamic and iterated versions of the application are the best fit.

The application of this method can result in better team cohesion and customer relationship building.

Application delivery and deployment needs should drive the method selection.

One approach to client service is to bring rigid perspectives on Agile vs. Waterfall then pitch one or the other as the only methodology to consider; this is a sign of a lack of understanding with regard to the nuances of project management. Indeed, an experienced project manager will be able to recommend ways to either merge Agile and Waterfall (the Hybrid approach) or offer alternatives, e.g., Scrum, Critical Path Method (CPM), Extreme Programming (EX), etc.

Depending on the project, there are many different reasons and use cases for utilizing either methodology, a mix of both or a combination of other approaches. Summarily, a plethora of alternative choices exist for project management, and the options should be weighed carefully to ensure the project’s objectives are successful.