Continuous Delivery For All

Jez Humble’s (@jezhumble) career has spanned roles through coding, infrastructure, and product development across three continents and organizations of varying sizes. To say he knows a lot about continuous delivery is a total understatement. In 2010, he and Dave Farley literally wrote the book on Continuous Delivery — and if you have not yet read it, I would suggest you pick up a copy. It’s a fascinating read.

I’ve heard Jez speak over the years at a number of DevOps conferences. I was excited when he accepted our invitation to speak at All Day DevOps. At the conference, Jez presented “Continuous Delivery Sounds Great But It Won’t Work Here”. He addressed the four reasons he consistently hears from organizations on why CD won’t work in their organization:

We’re regulated
We’re not building websites
Too much legacy
Our people are too stupid

Jez also outlined the real reasons behind the stated reasons: their culture sucks and/or their architecture sucks.

Jez spent his talk addressing each objection.

We’re Regulated

Jez noted it took Amazon four years to go from monolithic to service-oriented architecture (2001–2005) but they did it. Do you think they are the pillar of free enterprise — free of regulation? Not so. Being a publicly-traded company, they are heavily regulated; they are subject to Sarbanes-Oxley regulations; they handle personal and corporate financial and proprietary data; they provide private cloud services to the U.S. government, including highly sensitive data with the Department of Defense.

Speaking of Amazon Web Services’ GovCloud, they provide the cloud platform for cloud.gov, the 18F project Jez worked on to address the authority to operate process within the government.

The old process had screened every software application before it was allowed on a federal IT system. It stifled innovation because it duplicated efforts multiple times over. There was no cross-agency assumption that if one agency approved a software package, it was okay at another agency. Cloud.gov moved controls from operations into infrastructure as a service. Now, agencies can receive software through cloud.gov to integrate into their applications.

The bottom line is CD is excellent for heavily regulated environments because it gives assurance to regulators by removing opportunities for human error.

We Aren’t Building Websites

Jez’s favorite example is HP LaserJet firmware. They had a 400 person team in three continents. Obviously, it was critical for new printers, but the team was moving slowly. Only 5% of their time was spent on innovation. They tried hiring, firing, insourcing, and outsourcing. With business-school tried-and-true solutions exhausted, and knowing they have to eliminate non-value adding work to increase innovation, they looked to CD with two business goals in mind:

1) Get firmware off the critical path;

2) Achieve 10X increase in development productivity, measured by % spent on innovation.

They had other goals:

Reduce hardware variation
Create a single package
Implement continuous integration
Implement comprehensive test automation
Create a simulator

Note: the chart above doesn’t show the 23% of time spent managing the automation tests. Time well spent.

In the end, between 2008 and 2011:

Overall development costs were reduced by ~40%
Programs under development increased by ~140%
Development costs per program decreased 78%
Resources now driving innovation increased by 8X

Too Much Legacy

Jez’s example here is Suncorp, one of Australia’s largest insurers (note: also falls under the “We’re heavily regulated” excuse). Suncorp was burdened with loads of mainframes and legacy systems, but they built a set of test suites with open source tools to move into Continuous Delivery.

Our People are Too Stupid

Well...jokes aside: nope.

Jez gave an example outside of the software development industry. In fact, the example is in an old-line manufacturer — automotives. GM’s worst performing plant had been closed down. GM fired everyone from the plant. The workers were notorious for problems from sabotaging cars to drinking to gambling on the job.

Remarkably, GM’s union leaders convinced Toyota to hire the old GM employees to learn the Toyota processes. Toyota hired the GM employees to run their new NUMMI plant. Toyota and the old GM employees showed it isn’t the people that are the problem; it is the leadership, the management system, and the use of obstacles.

At the GM plant, you had a job and a certain amount of time to do it. If it didn’t get done, the assembly line kept running. At the Toyota NUMMI plant, employees were empowered to ask managers for help — without retribution. The car had to be in a good state before proceeded to the next stage of the production line.

In software development, this is Continuous Integration. Toyota’s NUMMI is the inspiration for Lean in the U.S.