September 23, 2019

The (false?) diversity of software development process

agile processesagile software developmentdevelopment methodologydevelopment teamsoftware development life cyclesoftware engineeringsoftware processwaterfall model

Ever wanted to annoy two technical managers? Bring in a Scrum Master and start talking about hybrid Waterfall-Agile approaches, throwing in some business stats to heat up the holy war. Keep shaken not stirred. The polemics about how best to arrange the software development process have been as hot as the arguments about what the best JavaScript framework is. In reality, although there are numerous buzzwords, and numerous subjective ways to measure just how iterative or incremental someone’s approach is… You guessed it. There is no pure black nor white.Here is a very brief overview of what the talk is roughly about: without the “ideological values” parts, but with sympathy to pragmatic questions like “what software development methodology to choose”.

Waterfall: clear, classic, industrial software engineering

The “waterfall model”, introduced by Winston Royce in 1970, is what one imagines instinctively when faced with a project. Clear stages of content development life cycle “flow” into one another, the next one starting after the previous one has finished. In other terms:

conduct exact requirements analysis and create documentation (and only then…)
create system design and architecture (and only then…)
actual development (and then)
test the code (checked?)
Proceed to deployment (delivery).

For some people, this sounds like an echo of the Cold War era, with allusions to a government-financed project. Crystallized requirements. Full control and predictability. Desirably no business risks in the picture.In reality, once this hits the rough terrain of the real business world, where everything changes, the risks become obvious. The development team may find one day that the customer wants to introduce change later on in the process, but this is… well, cumbersome to the development and hence costly to the client. Imagine someone telling you while watching a $1,000,000 movie they’d like the main character to be the opposite gender – just one single change, isn’t it? Thus, risk analysis needs to be thorough.

This is what the haters say (and we’ll get to them in a moment). For about ½ of teams, waterfall is the norm, since everyone likes clear expectations, predictability and easy planning. Especially when it comes to the bills.

Enter Agile

The “eternal adversary” of the plain waterfall is what we all call Agile. However, elements of Agile software development processes came on the scene much earlier than people coined the name. In essence, “agile” is an umbrella term for all the “iterative and incremental development” approaches.In a nutshell:Iterative = repeat until good. “We made it and then remade and built upon it, and then we remade it once again and that was what they wanted.”Incremental = make something one part at a time. “We had to build an online shop so we created the order service, then the paying system, then the review service, and so on.”Agile refers to the approaches that are both. The term appeared in 2001 with the Agile Manifesto, but the underlying principles have been around for a decade in the development community. If you take away the “values and benefits” for a moment, at the heart is the modified “spiral” model. At the beginning, the team produces the “core”, very basic version of the software, going through the normal phases. At the next cycle (iteration), the same steps are followed to produce something bigger based on what’s already done. The cycle expands over and over.

The main difference between simply “spiral” and Agile is what’s at that core. In the former case, it is the prototype, while with Agile, this is the MVP – the minimum viable product that can be released and used already. This is especially useful in domains like mobile app development.This allows to minimize risks for both the team and the customer. Even if drastic changes need to be done at some point, there’s not as much at stake, hence the financial risk is not as large. Agile processes can happen simultaneously, as well. Besides, with constant review at each iteration and feedback flow from the customer, such situations are less likely to be unexpected.

Agile vs Waterfall – what to choose, “traditional” account

On the one hand, Agile has become fashionable and really yields good results. It is openly used by giants like IBM, Cisco and Microsoft, while Apple are very close. According to the recent State of Agile survey,

69% of those who use it report better management of changing priorities
63% note faster time-to-market.

However, there is the burden of commitment – with Agile, the customer needs to be more involved. There are also other factors:

Waterfall is hands-off and easily managed for the customer once the project goals are defined, but the initial requirements need to be crystal clear.Waterfall is perfect for small projects where you can think of all the possible business goals initially, but not so good for larger ones.With Waterfall, the result is predictable – the customer knows exactly what they will get for their investment, the team knows what they are paid for.Waterfall: the team (especially developers and QAs) can focus on their actual work, not meetings and reviewsWaterfall has clear documentation at the very beginning and is better for thoroughly reviewed and regulated projects.

Agile offers more flexibility with adaptive planning, but is involving and demands for everyone to be there for feedback quite often.Agile allows to start with a usable MVP to throw into the market, and improve / change continually whenever the wind blows.With Agile, there is no such thing as precalculated “final total” cost – but there’s less inherent risk when remaking things over.Agile: the team is constantly learning and more creative, but also more individually responsible, making development active and inspired.Agile has no documentation upfront, being trust-based and encouraging responsibility, but allows faster time-to-market.

Buzzword glossary: how everything is put to practice

The terms that come next pop up in discussions frequently. Communities in and around development like to oppose them (think Scrum vs. Kanban, DevOps vs. Agile), but everyone seems to use them in various combinations to enforce some aspects of their work.

RAD (Rapid Application Development)

All about software time-to-market. Predesigned (modularized, “canned”) blocks of code bearing functionalities and entities are used in the so-called “low-code” or “minimal code” approach. The team assembles the application from these blocks extremely fast, in an iterative approach. At each iteration, first, business modeling, then data models are defined (what “objects” or data structures will be needed), then software process models, app generation and finally, testing and delivery.

Scrum & Kanban

“Organizational” techniques to make the team more efficient. Scrum is there to implement Agile. The team is small, with developers and tester, as well as Product Owner and Scrum Master, closely collaborating in short “sprints”. The bulk of what could be done for, and with, the product, is the product backlog, reprioritized in periodic meetings. Then there is a sprint backlog and the product increment (what has been done).Kanban means billboard in Japanese (practice stemming from Toyota), and the Kanban board is minimally 3 columns: To Do, In Progress, Done – with many possible extensions. Allows to streamline and coordinate processes in the team.

Continuous Integration and Extreme Programming

It happens, more often than not, that the separate components and parts of the system function well on their own but fail to integrate properly. To eliminate this risk, the CI system suggests all the developers working on the project submit their code to a shared repository once in a while. In this way, automated builds and tests can be run.“Extreme programming” is the same plus practices like pair programming, collective code ownership, etc.

DevOps

Described as “next-gen Agile”, DevOps extends the latter’s practices beyond development & testing and into operations, system administration, deployment, etc. The short formula often used is code as an infrastructure – important for all software systems. Advocated by no less a figure than AWS, DevOps offers better workflows and possibilities of automation.

Hybrid methodologies all around

Google who still uses the waterfall – no concrete names. This does not really mean it’s “dead”, only that it’s still within the “default, nothing special to speak of” field. On the other hand, the benefits of Agile have inspired many to take on hybrid approaches.In this sense, the waterfall that remains now is already with iterative elements. Many organizations are now “moving towards” Agile practices, adopting the techniques above. Today, you can even meet a Scrum Master at a generally waterfall team. A common approach is to have the project management work predictively, but the middle of the process (i.e. development and testing) are done incrementally. The models for mixing and blending are numerous, both hated and loved. About 67% of organizations use hybrid approaches at least sometimes.The reality is, any method has its strengths and weaknesses. The choice of a specific process ultimately depends on who the customer is. Some teams specialize in such fields that encourage waterfall, while some live in an environment where Agile is the norm. This is where the dogmatic views come from. It takes a view of a professional working not within but across different teams to define what approach will suit your business better. The practices are a continuum that’s constantly morphing and improving itself.