Organically DevOps: Building Quality and Security into the Software Supply Chain at Liberty Mutual

Some people are directors, managers, and administrators. Others are disrupters.

Eddie Webb (@edwardawebb) is an IT Disrupter for Software Development Platforms at Liberty Mutual and was a presenter at the 2016 All Day DevOps conference. His talk, Organically DevOps: Building Quality and Security into the Software Supply Chain at Liberty Mutual, looked at Liberty Mutual’s transformation to Continuous Integration, Continuous Delivery, and DevOps. For a large, heavily regulated industry, this task can not only be daunting, but viewed by many as impossible.

Eddie’s talk looks at the origins of DevOps at Liberty Mutual, the challenges of centralizing DevOps, and how to make it easy for developers and operations to do the right thing. He noted that DevOps is not their goal; Continuous Delivery is their goal, and DevOps is the enabler.

It all started at Liberty Mutual in 2004 with a vision: “Black-box reusable services that benefit the business.” They also had a concept that “developers own the quality of the code; we have no QA team.” Employees were empowered with this “concrete vision but malleable strategy,” as Eddie said. They gave them time and space to explore solutions, because ownership increases adoption.

The team started with fixing what frustrated them, such as: CI/CD pipelines, artifact and code repositories, elastic cloud-native runtime, and Agile planning and delivery.

But what about centralizing DevOps?

The reality is that tools can be centralized, but culture can not.

Eddie laid out an example of a large organization’s IT structur, saying that this is how things work at Liberty Mutual. Eddie’s team is the Central IT, and their customers are the Market IT team, not Liberty Mutual’s end customers. As Eddie pointed out, the more layers you create, the more friction that enters the system. Often, organizations try to reduce the friction through micro-fixes, but Eddie’s team asked how to change the culture to reduce the friction.

This required distributing innovation and centralizing the tools that work. As an example, if you build a manufacturing plant, you wouldn’t build your own power plant. If you are in IT, why build your own cloud?

This is the Netflix way. At Netflix, teams are independent, but responsible. Engineers can choose the best tools for the job. No one single team is responsible for innovation. However, the Engineering Tools team helps direct other teams’ experimentation toward new products without getting in the way of innovation.

Eddie quoted Drive when looking at how to motivate innovation in complex work environments. The author says if you want to incentivize or motivate cognitive tasks, you need three things: autonomy, mastery, and purpose.

When you centralize non-value added platforms, it enables teams to:

• Master their specific domains of business
• See clear purpose in everything they build
• Maintain autonomy of choice of frameworks, patterns, and approach

When you decentralize, it raises the question - what about security? Developers are humans, and humans inherently try to avoid tasks they see as unnecessary or overly burdensome, and this becomes easier when the compliance person isn’t there (think of how we handle speed limits). Traditional security measures erode profit and market share, and, from a stat Eddie quoted from Gartner, 77% of IT security professionals agree that security slows down IT.

Whether you are centralized or decentralized, security (and other compliance-oriented tasks) can increase adoption rates by making it easy to do the right thing. How do you do this? Eddie proposes governing with tools (think of speed governors in fleet cars).

You also need to have immutable environments, because “We can’t trust humans to manipulate production.” So, again, you have the make the path you want them to take, and make this the easiest path. As Eddie notes, “You can’t put too much friction into the pipeline, because developers are good at bypassing friction.” You have to make your CI/CD pipelines the easiest way to get ideas to production by. You also have to abstract complexity and automate policies. You have to have an “opinionated platform, and observe your customer so you can address their pain."

Eddie concluded with the following final points:

• Don’t mandate DevOps. Give employees the chance to master their discipline with examples to set and follow.
• Favor deep end-to-end accomplishments over broad but incremental steps forward. Focus on taking the right teams far before encouraging broad adoption.
• Centralize the platforms and tools that your teams shouldn’t be thinking about. Provide foundational services/commodities and let teams stay on purpose.
• Incorporate contributions from everyone; don’t stifle autonomy. Stay open to new ways of working.
• Challenge security policies, but respect intentions. Find new ways to enforce concerns without abandoning precaution.

You can listen to Eddie’s entire talk online here. You might learn an unexpected lesson.

If you missed any of the other 30-minute long presentations from 2016 All Day DevOps, they are easy to find and available free-of-charge here.  Finally, be sure to register you and the rest of your team for the 2017 All Day DevOps conference here.  This year’s event will offer 100 practitioner-led sessions (no vendor pitches allowed).  It’s all free and online on October 24th.