The third and last type of DevOps testing is Triggered Testing. In previous posts, I have covered how we need testing jobs to be Scheduled and why some of these should be able to executed On Demand. Now let’s see how these being triggered help in fast paced delivery with Quality.
One view of the DevOps is that it is the extension of Agile by taking it beyond just the Development phase and apply it to all phases. That would mean that feedback on usage of features will keep flowing continuously requiring the Development team to make changes continuously.
As the changes come in at any time, a Scheduled job can only run tests for changes since the last run. Depending upon the schedule, there can be a delay of 1 hour to 24 hours. To plug this gap, we can trigger testing for all changes.
You guessed it right that it is called Continuous Integration builds that are triggered with every push a Developer makes to the ‘master’ branch. A typical CI build is build code, build tests, run tests hence serves as Triggered Testing for the given change. All modern CI tools have this capability to enable it e.g. in Microsoft TFS you can set ‘Continuous Integration’ trigger and it will run automatically for any push.
(picture taken from here)
Any commercial solution is complex enough that running all tests for every change can slow down the release process. Though testing is mean to slow down the process to gauge Quality but it should at least synch with release frequency like daily deployments or even more frequent. To curb this, you can either define a subset of tests that will run with CI job or as I covered earlier that you can define “Test Impact Analysis” Maps that store info or source code to test code mapping. Thus you have the ability to only run tests that corresponds to a given change to save time.
Another area where Triggered testing comes handy is that some types of tests are sensitive to certain areas and should only be executed if that piece of code changes. For example, we have a module which converts data from one type to another, and if it changes, we need to run all our Data Conversion tests. You are now thinking in DevOps way, when you thought “Oh, that’s right but we should be able to run them automatically’. That automatically is triggering these tests if a condition is true. There are many ways you can achieve setting these conditions with most involve some scripting to allow this.
The other part of being Triggered testing comes with Deployments. Testing happens at each stage of DevOps as shown in this diagram and Tests execute after we Deploy. We have the typical Dev, QA and Prod environments for our deployments where changes first go to Dev, then tests are executed and if they pass, it is moved to QA. Currently we are not much confident about all our processes, so QA to Prod requires manual intervention but we are on the way to take this away as well.
Those Triggered tests that run in Deployed version are enabled through CD builds or as TFS calls them “Release Definitions” where you can define what tests run after it is deployed.
Before I sum up, it is important to note that these three types of testing or rather attributes of testing are played in harmony. Your strategy defines what tests are Triggered, Scheduled and On-Demand to come up with best plan for your needs. And if you can make each type of test having these attributes like Performance Testing is Scheduled but it can be Triggered or can be executed On-Demand, you are on your way to fast paced delivery with much more confidence.
In what circumstances you use Triggered Testing? What tools/techniques you use to enable it?
In an earlier post, it was explained that how a DevOps testing consists of three main types: Scheduled Testing, On-Demand Testing and Triggered Testing. I covered how we scheduled different types of testing in the same article and now let me dig into details of On Demand Testing.
Keeping the notion that “testing is an activity” that is performed more often than a “phase” towards the end, it is necessary to configure Continuous Testing. That requires to setup automated testing jobs in a way that they can be called when needed. This is what we call On Demand Testing. This is contrary to common notion of having testers ready to do testing “On Demand” as features get ready to test.
(the original photo is here)
For example, we have a set of tests that we call “Data Conversion tests” as they convert data from our old file format to new file format. As you can imagine, lot of defects uncovered by such tests are data dependent and a typical bug looks like “Data conversion fails for this particular file due to this type of data”. Now as a Developer takes upon that defect and fixes this particular case, she’d like to be sure that this change hasn’t affected the other dataset in the conversion suite of tests. The conversion testing job is setup in a way that a Developer can run it at any given time.
I shared that we are using Jenkins for the scheduled testing. So one way for an individual team member to run any of the jobs on-demand is to push set of changes, log into the Jenkins system and start a build of automated testing job say Conversion Testing for the above example. This is good, but this might be too late as changes have been pushed. Secondly the Jenkins VMs are setup in a separate location than where the Developer is sitting and feedback time can be anything between 2-3 hours.
Remember, tightening the feedback loop is a main goal of DevOps. The quicker we can let a Developer know that the changes work (or don’t work), the better we are positioned to release more often with confidence.
So in this case, we exploited our existing build scripts which are written in Python but are basically a set of XML files that define parts that can be executed by any team member. We added a shortened dataset that has enough diversity and yet is small, which runs within 15-20 minutes. Then we added a part in the XML file that can do this conversion job on any box at any given time by anyone in the team.
Coming back to the same Developer fixing a Conversion defect, the Developer after fixing the bug now can run the above part on her system. Within half an hour, she’ll have results and if they look good, she’d push changes with a confidence that next round of Scheduled Testing with larger dataset would also pass.
Please note that we have made most of our testing jobs as On-Demand but we are having hard time at few. One of them being Performance Testing because that is done on a dedicated machine in a controlled environment for consistency of results. Let’s see what we can do about it.
Do you have automated testing that can be executed On-Demand? How did you implement it?
To have constant heartbeat of release, testing has to take the central stage. It can no more be an activity that is performed at the end of release cycle rather it has to happen during all the phases of a release. And mind you release cycle has shrunk from months to days.
In “Effective DevOps” book, the authors lay out many plans for making an effective plan to move towards a DevOps culture. On automation, it suggests that automation tools belong to either of the following three types:
- Scheduled Automation: This tool runs on a predefined schedule.
- Triggered Automation: This tool runs when a specific event happens…
- On-Demand Automation: This tool is run by the user, often on the command line…
(Page 185 under Test and Build Automation section)
The way, we took upon this advice to ramp up our efforts for Continuous Testing is that each Testing that we perform should be available in all three forms:
- Scheduled Testing: Daily or hourly tests to ensure that changes during that time were merged successfully. There are no disruptions by the recent changes.
- Triggered Testing: Testing that gets triggered on some action. For example a CI job that runs tests which was executed due to push by a Developer.
- On-Demand Testing: Testing that is executed on a needs basis. A quick run of tests to find out how things are on a certain front.
Take Performance testing for example. It should be scheduled to find issues on daily or weekly basis, but it could also be triggered as part of release criteria or it could be run On-Demand by an individual Developer on her box.
In order to achieve this, we re-defined our testing jobs to allow all three options at once. As the idea was to use Tools in this way, we picked upon Jenkins.
There are other options too like GoCD and Microsoft Team Foundation Server (TFS) is also catching up but Jenkins has the largest set of available plugins to do a variety of tasks. Also our prime use case was to use Jenkins as Automation Server and we have yet to define delivery pipelines.
(the original icon is at: http://www.iconarchive.com/show/plump-icons-by-zerode/Document-scheduled-tasks-icon.html )
I’ll write separately on Triggered and On-Demand testing soon and now getting into some details on how we accomplished Scheduled Testing below.
We had few physical and virtual machines, on which we were using Windows Task Scheduler to run tasks. That task will kick off on a given day and time, and would trigger Python script. The Python scripts were in a local Mercurial repository based in one of these boxes.
The testing jobs were Scheduled perfectly but the schedule and outcome of these jobs were not known to the rest of the team. Only testing knew when these jobs run and whether last job was successful or not.
We made on of the boxes as Jenkins Master and others as slaves. We configured Jenkins jobs and defined the schedules there. We also moved all our Python scripts to a shared Mercurial repository on a server that anyone could get. We also created custom parts into our home grown Build system that allows running pieces in series or parallel.
Given that Jenkins gives you a web view which can be accessed by all, the testing schedule became public to everyone. Though we had a “Testing Dashboard” but it was an effort to keep it up-to-date. Also anyone in the team could see how was the last few jobs of say Performance testing and what were the results.
Moving to Jenkins and making our scripts public also helped us make same set of tests Triggered and On-Demand. More details on coming posts so as how.
I wish I could show a “Before” and “After” pictures that many marketing campaigns do to show how beautiful it now looks like.
Do you have Scheduled testing in place? What tools you use and what policies you apply?
DevOps is a big thing these days and it is hot. Lot of people are talking about it and as we are going through this journey of adapting DevOps culture, we are learning it.
In recent month or so, I have given some introduction to fellow testers at my organization and shared thoughts at couple of Industry sessions. So I thought it would be good idea to share my slides here.
For further study, I’d recommend reading following content:
- “Effective DevOps” book by Jennifer Davis & Katherine Daniels
- Microsoft’s guide to DevOps
- Continuous Integration blog on Nutcache
- “A Practical guide to Testing in DevOps” book by Katrina Clokie
- Continuous Testing by Dan Ashby
- Moving to weekly releases e-book by Rob Lambert
All the best with your DevOps journey!