Acing Continuous Testing in DevOps

How do we ace our continuous tests? What does that look like? Eran Kinsbruner, who works for Perfecto, has some ideas.

First, we should understand what continuous testing is: finding the key, high-value scenarios for your software and automating them. This will give you feedback on the business risk of releases on every change. 

Continuous testing is about more than opening the door to escape defects. We also don’t continuous test every aspect of our system, only the ones that give us the greatest value. 

Shifting Left

In order to be continuous, we need to be automated. We don’t just test when the developer writes the code, or after. We test after we release the code, and we test when we deploy the code. We test at all points we can. 

There’s the idea of “shifting left” for testing, but people tend to misuse this term. Instead, we should look at the benefits of shifting left where it’s most appropriate. 

Some of the benefits of shifting left include

• It’s cost effective to reveal defects at the earliest possible moment.
• It supports the goal of releasing faster.
• It’s easier to maintain code and quality.
• We can cover more functional and non-functional areas early, like security and performance.

Many teams only shift left with the things that are easy, like functional, unit, and UI tests. They leave out things like user acceptance testing or security. 

Making all your tests continuous is hard.

It takes time to analyze failures. Many times, a team is lacking the training and skill sets needed to understand certain types of tests. An organization's test environment may not be configured in a way that makes it easy to set up, tear down and run tests over and over again. 

Matching Technology With Personas and Objectives

Beyond these challenges, it’s key to match your technology with personas and your business objectives. Different technologies will match different objectives. Something like Cypres.io will help you accelerate feedback, whereas other tools may promote higher coverage. 

We don’t want to automate everything—only what’s key to our objectives. 

Set up a strategy, review your capabilities, and figure out which ones are most critical. Also, review what aspects they’re most critical in. Bill pay may not be the most important user experience, but it’s critical to ensure strong data security. You should manage the scope of testing for each feature. 

You’ll want to ask yourself

• How much time do you need to schedule the testing?
• What environments can you use to automate tests?
• What tools do you have available to automate them? Will it be expensive to procure the right ones?

Case Study: Mobile Testing Requirements

In one mobile banking application, there was a selection of UI testing and cross-platform coverage to cover app functionality. The team had no functional testing around security and accessibility or tests for various network conditions. And they would test interruptions around calls and alerts. 

But all this is only good for a few weeks. You have to have all this as a continuous part of your CI/CD activities because things change. In a week, the next version of iOS could be released, invalidating your current tests. 

You can get fast feedback on your tests by parallelizing your test automation. Lots of tests means that running them all one after the other can be slow—sometimes more than a few hours. Running them in parallel will give you confidence and feedback quickly, making you more willing to add additional types of tests. 

Create an Open, Integrated Ecosystem

A key to continuous testing is to choose tooling and share knowledge that crosses languages. Your business folks may be used to tests written in Cucumber. This works great with Java, but your mobile devs may be working in Objective C. Ensure that you use tooling that you can integrate into your wider environment, having a heterogenous pipeline of tests. 

Keep a Solid Automated Pipeline

No one type of test will give you the confidence you need that your key scenarios are working. You need various tests that will run at different times to give varying speeds of feedback. You may have smoke tests running on every pull request. Maybe once a day at noon you have a regression test suite that runs more thoroughly. Ensure these tests are not flaky and can run consistently. 

Use a phased coverage approach. You want some fast tests to run multiple times a day, and you want other, slower tests, like UI tests, to run only a few times a day. 

Keep your pipeline clean and green. If you see a pipeline going red, fix it immediately. Figure out ways to improve your process to ensure your tests will be green most of the time. If a test failure lingers and you ignore it, then the test has no value. It is outdated and broken.

Or Else: The Money Quickly Adds Up

The needs of software changes, and the only way to keep up with change is with automated, continuous tests. If you don’t implement these, it will cost your team and your organization. Developers lose productivity from unstable builds. But they will go fast with confidence with a solid, varied, continuous testing pipeline. 

This session was summarized by Mark Henke. Mark has spent over 10 years architecting systems that talk to other systems, doing DevOps before it was cool, and matching software to its business function. Every developer is a leader of something on their team, and he wants to help them see that.