Delivery Patterns for Rapid & Reliable Software Releases

Imagine going to a new supermarket. You don’t know where things are, so you spend extra time finding what you need. You might be down to the last item on your list - the one you really need to make dinner tonight - and it is back on the other side of the store, where you have already been three times. After a few visits, you learn where things are. The visit to the supermarket becomes routine, or even boring.

The real excitement and satisfaction comes from what you are cooking, not the routine of getting the ingredients.

This is the illustration that Manuel Pais (@manupaisable) gave in his All Day DevOps conference talk, Delivery Patterns for Rapid and Reliable Software Releases. Manuel is a developer, a Continuous Delivery and DevOps consultant, and co-author of the upcoming book, Team Guide to Software Releasability.

Manuel’s talk is centered on the idea that software releases are the boring part of software development if you are doing them right. He quotes Dave Farley and Jez Humble, authors of a book on Continuous Delivery, on what Continuous Delivery is about: the “ability to get changes of all types into production, or into the hands of users, safely and quickly in a sustainable way.”

When you do this well, you can focus on the fun part - developing your application. To get to releases, you need to focus on three things: safer releases; faster releases; and, sustainable delivery.

Safer Releases

Manuel talked through three steps to achieve safer releases:

• Map the value stream in the pipeline, including all activities that are required. Make sure to look to the left of code changes to see where the bottlenecks are in your delivery because they might be sooner than coding.

• Measure key metrics. Make sure you don’t have too many metrics, as they can become overwhelming and none will be used, or you will use all of your time and energy measuring metrics. Three metrics to focus on are:
• Speed - the cycle time
• Quality - the defect rate
• Operability - the mean time to repair (MTTR)

Also make sure you know how your metrics interact. For instance, a shorter MTTR might cause a higher defect rate. Finally, keep an eye on flow efficiency. This is the amount of time spent on a task vs. the amount of time waiting for someone to work on a task. Studies show that the average organization has a 10-15% flow efficiency, meaning 85-90% of time is waiting. The issue is not how long it takes to work, but how long to wait.

Put in the ground work. Manuel offered a few examples:

• Automated builds in clean environments
• Ephemeral prod-like test environment
• Health checks and fast smoke tests
• Automated acceptance tests
• Everything as code
• Single path to production

Faster Releases

Previously, the pipeline was Build-Test-Deploy. Now it is much more involved:

To realize faster releases, Manuel walks through three steps:

1. Identify and remove bottlenecks. Make sure to see where multiple projects might be creating a bottleneck. For instance, if there is one testing team for all applications, there might be a bottleneck that isn’t immediately evident in the pipeline for one application.

1. Shorten the path to production. There might be opportunities to shorten the paths in the same pipeline. For instance, a small graphic change might not need to go through everything, but in other cases a change approval board review might be warranted.
   
   
2. Continuous pruning and fast feedback. Software keeps growing and the number of tests keep growing. This isn’t necessarily sustainable. It can be more efficient to run the test that failed last time first, then look at the new tests for the new features. It is based on a concept from the book, Fifty Quick Ideas to Improve Your Tests, “There are just two moments when an automated test provides useful information: the first time it passes and when it subsequently fails.”

Sustainable Delivery

This is a principle that Manuel notes often gets overlooked or misunderstood. In today’s environments, CI/CD pipelines are mission-critical systems to get changes into productions at the pace we require. Without the right tools and processes that are reliable and sustainable, the delivery system becomes the bottleneck.

Delivery becomes unstable and we spend way too much time fixing it.

As a system, we need to think about system operability:

• Available - no downtime
• Scalable - no pipelines waiting to run
• Monitored - alerts on resources and errors
• Recoverable - focus on diagnosis and MTTR

To meet these goals, Manuel recommends treating your pipeline as a product, using these practices:

• Immutable infrastructure
• Blue-green deployments
• Monitoring and alerts
• Aggregated logging
• Pipelines-as-code
• Auto-scaling

Our delivery systems are becoming ever more critical to meeting our business goals, from speedy updates to making our systems more secure, they are the backbone of our software delivery lifecycle. When we can get these systems fast, reliable, and sustainable, we can focus our creative energies on developing our applications.

You can hear more from Manuel in his talk below.

Register now for the next All Day DevOps, November 6, 2019. It will be a day to discuss security, CI/CD, cloud native infrastructure, cultural transformation, site reliability engineering, and other interesting topics.

photo: Tom Fisk