Cancer sucks. But with folks like Sarah, DevOps is helping make a difference in the race to a cure.
Sarah Elkins is not curing cancer herself, but she is employing DevOps practices to help those who are. Sarah supports the technology infrastructure for those who are trying to cure cancer at the National Institute of Health (NIH).
Sarah Elkins (@configures) configures technology solutions at the National Cancer Institute (NCI), where she has worked for 9 years. NCI is a federal agency - part of the National Institutes of Health. They support over 700 websites, from basic HTML to content management systems to complex bioinformatics systems which have multiple tiers and thousands of servers. They use multiple operating systems, containers, and physical and virtual machines (both on-premise and some AWS). Developers range from lone scientists to large teams.
Sarah and I have been following one another on Twitter for several years, but I recently caught the session she delivered in the “DevOps in Government” track at All Day DevOps. Sarah led a discussion about how the National Cancer Institute is automating its builds and deployments - showing it is possible even in a large bureaucracy.
Her talk covered the processes and technologies which enable software to move from source code repositories all the way to production servers at nci.nih.gov, including the use of GitHub, Jenkins, Nexus, and more technologies, with a variety of teams involved.
We need to talk
Sarah’s cure begins with engagement - development, infrastructure, and security all working together -- just as teams of specialists work with patients to help them beat cancer.
To help speed development and approvals, a necessary evil that can grind development to a halt if not managed well, they agree that applications drawing on an existing technology catalog are approved as operational and security teams provide guidance to development on what is required for an Authority to Operate (ATO). In other words, they are trying to pre-approve as much as possible and communicate earlier in the process.
DevOps practices are maturing
Source code is primarily kept on GitHub, including a public repository for non-proprietary source code, and they primarily use Maven or Apache Ant for build scripts. Infrastructure teams provide XML templates to provide consistency.
Most NCI software relies on build dependencies on open source software components. They use artifacts during builds and utilize Sonatype's Nexus as their repository manager.
All of this supports the automation of builds and deployments at the NCI. For builds, most active projects are either on, or migrating to, Jenkins. Build artifacts may be .zip, .war/.ear, or Docker images.
For application deployments, they use development, quality assurance, stage, and production stages. Most teams use some form of automation, from simple (copy content, stop/start container) to more complex scripts. They allow developers to perform deployments for robust applications, and some manual orchestration is required, for instance for database timing or related applications.
As an organization, they are moving towards Continuous Integration, with varying progress among teams.
Building security in
The have also embraced involving security early, often, and automatically. They have deployed self-serve/on-demand Nessus scans, which allow developers to see, on their own, how they are doing as soon as the application is stable. Security teams run AppScan and Twistlock for Docker image scanning during Jenkins builds, just before deploying, and frequently scanning the repository in between. For issues found, development and infrastructure teams work together to remediate security concerns.
At the end of her talk, Sarah offered up three takeaways:
DevOps at NCI is a work in progress (isn’t it everywhere?)
The wide-ranging needs at NCI require flexibility, communication, and teamwork
There is no shortage of work
Since it is a work in progress, what opportunities are ahead at NCI?
More automation for individual applications with Jenkins
Developers can perform container restarts on lower tiers
Database updates (some projects are using Liquibase already)
Some applications are still manual/batch scripted
More Docker and improving Jenkins / Docker instances
More orchestration automation and Puppet/Ansible integration
Security improvements through
Scan dependency artifacts before building
More integration and automation
Sarah dives into more details in her talk, which you can watch here. If you missed any of the other 30-minute long presentations from All Day DevOps, they are easy to find and available free-of-charge here. DevOps in Government is one of the 5 tracks.