ABN AMRO Embraced CI/CD to Accelerate Innovation and Improve Security

ABN AMRO is one of the largest banks in the Netherlands. It is a large enterprise that is heavily regulated. They employ 22,000 employees and 5,000 of them work in IT. After a major transformation journey from waterfall to Agile, they now have over 300 Agile teams.

Many organizations would have shied away from the transformation, but ABN AMRO saw FinTech companies nipping at their heels. Transformation was imperative to survival. They couldn’t be the technological equivalent of the stereotypical fat cat, cigar smoking, short-work-day bankers who refuse to adapt.

Stefan Simenon was in the middle of the transformation when it began three years ago, and he recently shared his experience about the journey. His talk, ABN AMRO Transforms with CI/CD to Accelerate Software Delivery and Improve Security, focuses on explaining how the bank implemented in CI/CD pipelines to accelerate innovation while maintaining strong governance and security standards.

You don’t implement - or even think about implementing - a cultural shift like this in an organization this size because it is a latest trend or you watched some inspiring talks during a recent conference. You have to feel the burden of the status quo. For ABN AMRO, several challenges were staring them in the face:

• Long lead times for software delivery
• Software quality issues found at a late stage
• Many manual handovers and approvals
• Code merges happening late in the dev lifecycle
• Inefficient cooperation between Dev and Ops
• Big non-frequent releases to production

Admitting you have a problem is the first step, but there are many more. As they agreed to move forward with CI/CD, they recognized that CI/CD is about changing the mindset, behaviors, processes, and the “Way of Work” first.  The right tool choices would come later.

To proceed, they setup the project organization into a cluster with central and decentralized orientations. The centralized part paved the way by setting up the conditions for the teams to get working. The decentralized parts moved forward by implementing CI/CD within the teams.

Once the teams were in place, they determined they would start with the technologies they had and wait for other tools. They also ensured there was strong alignment between Development, Operations, and Security.

Recognizing that other large organizations often take 3-8 years to implement this level of change and change course along the way, they plan for small milestones at three month intervals while keeping the overall transformation journey in mind. This allowed them to learn and improve as they progress.

One interesting approach they have taken is called “build breakers”. That is, once a developer triggers a build and the unit testing is complete, three separate scans are run: a code quality scan with SonarQube; a secure source coding scan with Fortify; and, an open source dependency scan with Nexus Lifecycle. A break in any one of these will send the build back to the developer to be fixed.

They also setup an IT for IT organization (IT4IT) to enable CI/CD implementation. The IT4IT organization:

• Implements tooling upgrades
• Implements new tools
• Enhances and improve CI/CD pipelines
• Implements new CI/CD pipelines
• Handles user management
• Supports Agile teams
• Conducts incident and problem management

A lot has happened since they began three years ago. Here are just some of the benefits have they seen so far:

• Test environment uptime improved
• Improved code quality and secure coding
• Improved cooperation across stakeholders
• Improved time to market
• Improved development processes

There is still more to do. As they move forward, they want to further transform to DevOps by improving collaboration between Dev and Ops. They also want to automate and improve tooling pipelines, enhance the IT4IT landscape, implement a hybrid cloud strategy with a mix of internal and AWS clouds, and move toward a service oriented architecture. They also realize that improving the Way of Working, mindsets, and behaviors has to stay top of mind throughout their journey -- it is the foundation all of this is built upon.

At the conclusion of his talk, Stefan offered some takeaways:

• Ensure you have senior management and involvement
• Invest in reducing technical debt
• Create a safe environment so people know that failing is okay
• Do not focus just on tooling
• Do not underestimate the journey and complexity
• Do no focus on the long term but rather on small improvements

Stefan’s full talk, available for free here, digs deeper into the specifics and the results to date. You can watch any of the 100 All Day DevOps practitioner-led sessions free-of-charge here.