A lot has been written in the past about CI/CD for software, but it's really hard to find examples of how people apply these principles to the whole product.
So let's raise the bar and talk about what it takes to do application CD and to deploy multiple times per day. Specifically, let's talk about the approach Autodesk took to solving this problems, and what lessons were learned along the day.
A few years ago, Autodesk was making a number of changes:
- Moving product model to perpetual licensing model.
- Going from being a desktop company to a cloud-platform company, meaning lots of web applications and cloud-enabling the desktop ones.
- Becoming a whole lot more collaborative internally.
- Driving toward a unified solution in order to become more efficient.
As part of this, there was a need to define metrics for success. This included the four DORA Metrics:
- Deployment frequency
- Change lead time
- Time to restore service
- Change failure rate
Autodesk also sought to become significantly more social in a variety of ways. This included tools like Slack and more collaboration, but it also meant letting anyone internally see anyone's code and "inner source" their software. And of course, there was also a push to contribute more to open source.
All of this sounds great, but there's something of a catch. These types of policies and philosophies are great for startups. But it's a significant challenge for a company with a lot of people and a ton of existing infrastructure and applications, including 200 desktop, web, and mobile applications.
Some of the friction points for this included those existing applications, the historical tendency to do a yearly, global launch, lots of functionality duplication arising from organizational silos, and last-generation tools.
Does this sound familiar to anyone? It's a common tale in the enterprise. But it's not insurmountable.
What if you could change something like a GUI message, change it, translate it to many different languages, and deploy that multiple times per day? Let's look at Autodesk's three-year journey to getting there.
This includes, broadly speaking:
- Executive buy-in
- Software CI/CD pipeline
- Inner-sourcing components
- Documentation pipeline
- Localization pipeline
Step 1: Management support
The first step to any significant organizational change comes from leadership. This requires a multi-level need for buy-in, including executives, management, and engineers.
To make this happen, leadership needs to understand the business case for CI/CD. Once that happens, you've cleared the first significant hurdle to making a change like this.
Step 2: New development platform
With management support in place, next up is going to a new development platform, using a lot of new tool to meet the demands of:
- 4000 users
- 16,000 Repos
- 18M Git commands/day
- 1.5M Artifacts
- 1.8M Downloads/day
Starting with this platform and using these tools as the base, Autodesk built a comprehensive CI/CD platform that includes CI/CD basics, but also sophisticated concerns like static analysis and test coverage measurements. All of this contributes to an implementation of DORA best practices.
All of this came with some important lessons learned.
- Transformational leadership is required
- Tiered adoption is important, rather than trying to do everything at once.
- White-glove versus self serve: initially, they held teams' hands, but with an eye toward making it sustainable by making it self-serve.
- They set a goal of a 70% success rate, and fixing that goal in mind was critical to achieving what they have.
- Broad adoption isn't easy, and getting to their goal has required evangelism, marketing, and even nagging.
- There are high support costs involved with a significant transformation. For instance, a lot of people within the organization have lots of questions that need answers.
Step 3: Component Governance
In the beginning, Autodesk started this transformation with lots of manual approval and validation of components. But as they migrated binaries from Perforce into Artifactory, they started seeing lots of duplication and copies. And to make matters worse, some binaries and their attendant source code varied only slightly.
This led to a desire to automate and validate binaries and ingestion thereof.
They started with JIRA and put together a workflow. Anyone requesting a new component uses JIRA, which routes the request to the appropriate groups for action (e.g., legal). If a component is approved, they would fork that component into a new repo, where all subsequent changes would reside.
Once they had this repo, they'd wrap "inner-source best practices" around that repo. They'd add a CI/CD pipeline around it to manage changes. This helped them organize the components in such a way to eliminate duplication.
Unifying components came with its own set of lessons learned.
- This does come with some local cost, but it comes with a lot of global gain at the organization level.
- Finding an owner for individual components is hard. It requires finding volunteers (or sometimes "volunteering" them)
- Unification is hard, because it requires breaking dependencies at scale and course-correcting from there.
- While unification is hard, it's enormously valuable. As a critical example, consider heartbleed. If Autodesk had to fix that again, it would be a simple matter of addressing the SSL component, rather than scrambling across dozens of applications.
Step 4: Application Documentation
Autodesk has an enormous documentation footprint:
- API documentation
- Application content, including online help, "what's new" and update notifications.
- Translations into 12 languages of the above.
They started with this as their solution to documentation.
The biggest change that they made was to move to using markdown as the source of truth for their documentation. This allowed them to version control it alongside the code and to then work it into their CI/CD pipeline to make sure it went where it was needed.
This approach to documentation allows for some cool capabilities. That includes the "magic green button," which creates a drop-in widget that allows Autodesk to crowdsource translations of their documentation via pull requests.
As with the other steps, the documentation journey came with lessons learned.
- Storing documents alongside software let them move away from "waterfall documentation" and to bring it in line with their software development methodology.
- This approach allows applying DORA to documentation.
- A scheme that allows anyone to edit documentation is a huge efficiency boost.
- Since documentation looks like code and is managed with pull requests, developers participate much more readily.
- Documentation writers are actually really keen to move to this approach.
Step 5: Localization
This is one of the most important and challenging steps. Autodesk is in 12 languages, which means a translation of 1.4 billion words per year being translated.
Existing solutions to this aren't great. You can ignore the problem and just go with English. Or you can hire Localization Service Providers to do the service for you in bulk.
But think about something. Isn't this kind of an anti-pattern to a lean/CI/CD approach? You build everything in tight feedback loops, but then turn around and ship the translations to a service that does it over the course of months?
Autodesk addressed this gap in the following way, using an internal workflow.
This allowed them to create a localization pipeline that existed alongside the software pipeline.
- Localization can be continuous. It's really hard, but it's possible.
- It's worth making a large investment in auto-translation.
- Neural translation technology looks promising, and it's worth investing in the tuning and training.
Wrapping Up with New Bliss
In the end, here is what the entire application pipeline looks like. And it represents a much happier life for Autodesk.
They have succeeded in getting to a company-wide CI/CD approach to the product. Teams are actively onboarding to the setup with these practices, and those who onboard really like it. Even the desktop folks are interested in adopting this approach, in spite of the obvious challenges with CD and a desktop app.
Looking at things holistically, though, there's been a night-and-day culture shift. The engineering mentality toward collaboration has shifted completely, to the point where someone standing up in a meeting and proposing to do things the old way would be met with disbelief.
In the end, it's worth thinking about what ramifications adopting CI/CD at your application would have at your company. How much better might things work and how much happier might folks be? So chart your North Star, and start moving in that direction.
Missed George Swan’s session, or want to see some other great presentations from October 17? Head over to https://www.alldaydevops.com/live and make sure you’re registered. Then, catch up on what you missed (or re-watch your favorites)!
About the author, Erik Dietrich
Erik Dietrich is a veteran of the software world and has occupied just about every position in it: developer, architect, manager, CIO, and, eventually, independent management and strategy consultant. He’s the cofounder of Hit Subscribe and writes at daedtech.com. Connect with him @daedtech.