The third and last type of DevOps testing is Triggered Testing. In previous posts, I have covered how we need testing jobs to be Scheduled and why some of these should be able to executed On Demand. Now let’s see how these being triggered help in fast paced delivery with Quality.
One view of the DevOps is that it is the extension of Agile by taking it beyond just the Development phase and apply it to all phases. That would mean that feedback on usage of features will keep flowing continuously requiring the Development team to make changes continuously.
As the changes come in at any time, a Scheduled job can only run tests for changes since the last run. Depending upon the schedule, there can be a delay of 1 hour to 24 hours. To plug this gap, we can trigger testing for all changes.
You guessed it right that it is called Continuous Integration builds that are triggered with every push a Developer makes to the ‘master’ branch. A typical CI build is build code, build tests, run tests hence serves as Triggered Testing for the given change. All modern CI tools have this capability to enable it e.g. in Microsoft TFS you can set ‘Continuous Integration’ trigger and it will run automatically for any push.
(picture taken from here)
Any commercial solution is complex enough that running all tests for every change can slow down the release process. Though testing is mean to slow down the process to gauge Quality but it should at least synch with release frequency like daily deployments or even more frequent. To curb this, you can either define a subset of tests that will run with CI job or as I covered earlier that you can define “Test Impact Analysis” Maps that store info or source code to test code mapping. Thus you have the ability to only run tests that corresponds to a given change to save time.
Another area where Triggered testing comes handy is that some types of tests are sensitive to certain areas and should only be executed if that piece of code changes. For example, we have a module which converts data from one type to another, and if it changes, we need to run all our Data Conversion tests. You are now thinking in DevOps way, when you thought “Oh, that’s right but we should be able to run them automatically’. That automatically is triggering these tests if a condition is true. There are many ways you can achieve setting these conditions with most involve some scripting to allow this.
The other part of being Triggered testing comes with Deployments. Testing happens at each stage of DevOps as shown in this diagram and Tests execute after we Deploy. We have the typical Dev, QA and Prod environments for our deployments where changes first go to Dev, then tests are executed and if they pass, it is moved to QA. Currently we are not much confident about all our processes, so QA to Prod requires manual intervention but we are on the way to take this away as well.
Those Triggered tests that run in Deployed version are enabled through CD builds or as TFS calls them “Release Definitions” where you can define what tests run after it is deployed.
Before I sum up, it is important to note that these three types of testing or rather attributes of testing are played in harmony. Your strategy defines what tests are Triggered, Scheduled and On-Demand to come up with best plan for your needs. And if you can make each type of test having these attributes like Performance Testing is Scheduled but it can be Triggered or can be executed On-Demand, you are on your way to fast paced delivery with much more confidence.
In what circumstances you use Triggered Testing? What tools/techniques you use to enable it?