Only test changes

We are living in ‘survival of the fastest’ era. We don’t have time for anything. We prefer reading blogs instead of books and we look for tweets rather than lengthy press releases. So when it comes to testing a release that has only a few changes, we don’t have time to run all the tests.

The question but is: which subset of tests we should be running?

I have touched this subject in Test small vs. all, but looking at build change logs and picking up tests to run is a task that requires decision making. What if we can know the changes automatically and run tests based upon that?

That is possible through TIAMaps. No this term is not mine but part of it is. It originates from Microsoft’s concept of ‘Test Impact Analysis’ which I got to know from Martin Fowler’s this blog post. I’d recommend to read it first.

If you are lazier than me and couldn’t finish the whole blog, below is a summary along with a picture copied from there:

First you determine which pieces of your source code are touched upon by your tests and you store this information is some sort of maps. Then when your source code changes, you get the tests to run from the map and then just run those tests.

Below is a summary of TIAMap implementation in our project.

Why we needed it:

We didn’t do it for fun or due to “let’s do something shiny and new”. We are running out of time. Our unit tests suite has around six thousand tests and a complete run (yes, they run in parallel) takes about 20 minutes. Hmmm… a little change that needs to go has to go through 20 minutes of Unit test execution, that’s bad. Let’s see what others are doing. Oh yeah, Test Impact Analysis is the solution.

Generating TIA Maps

Code coverage comes to the rescue. If we already have a tool that finds out which lines of code are touched by all tests, can’t we have a list of source files that are touched by a single test?

So we configured a job that would run for tests and saves this simple map: test name -> source file names. There were two lessons that we learned:

  1. Initially, we had a job that would run for all 6,000 thousands and it was taking days. We became smarter and after generating first TIA Map for all tests, we only update maps for the tests that changed. We don’t have a way to find the test names that changed, but our job is based upon timestamp of files that have test code.
  2. We were storing the Map in a SQLite Db. As the Db had to pushed to our repository again and again, it was difficult to find deltas of change. We switched to simple text file to store the Map. Changes can be seen in our source control tools and anyone can look at those text files for any inspections.

Running Tests

As you can imagine that the hard part is to get those TIAMaps. Once we have them, we now do the following:

  • When there is a need to run tests, we determine which source files have changed since the last run.
  • We have a Python script that does the magic of consulting the maps and getting a list of tests to be executed.
  • We feed that list of tests to our existing test execution program.

How is it going?

It is early to say that as we have rolled this as pilot and I may have more insights into the results in few months. But the initial feedback is indicative of us being on the right path. Time is being saved big time and we are looking for any issues that may arise due to faulty maps or execution logic.

Have you ever tried anything similar? Or would you like to try it out?


Tags: ,

5 responses to “Only test changes”

  1. Qambar says :

    Very though provoking blog Majd, but please tell me what is difference between Test impact analysis and usuall impact analysis which we perform during requirement gathering phase or during sprints to identify scope of dev/testing, impact of change/user story , is this same thing.


    • majd says :

      Thanks Qambar. The impact analysis that you mentioned is Requirements mapped to features/modules of code. Test Impact Analysis is Tests mapped to code such that when code changes, you can find out which tests to run automatically. Hope this helps.


  2. Qamar Tarar says :

    When we talk about such Things like 100% Automation or Tools like TIA, we Forget the practical effects or Problems. Its nice to talk as ” We Need 100% autoamation” or ” we Need 100% unit Tests coverage” but in practice, it is Happening a very small fraction of it. As I understood from all These Blogs about TIA, you Need some obligatory Things to have this method work:
    – You Need every where in your whole Company the same Microsoft development Environment
    – You Need the same Mentality from all developers to write unit Tests and perform pre-integration Tests
    – you Need almost 100% coverage or Instrumentation to perform such Testing
    – you Need experience both with Testing as well as with Platform / Technique like TIA

    For big guns like Microsoft, Google, Facebook or any other big Name, may be you can have such platform but almost every other Company having their own particular development culture and Environment with different Kind of Knowledge base for its developers, its difficult to use such methods or techniques.

    From my personal experience as tester and my understanding, its all your experience that makes you able to perform such Testing. If you have good product Knowledge, you will find some way to perform such Regression Testing and integration Testing for new Releases.

    Also its very difficult for all developers to write unit Tests as time and experience matters.


    • majd says :

      Thanks Qamar for your thoughts. The intention of doing this exercise is not to ‘only’ rely on automated unit tests rather the purpose is to make them more useful and robust.

      I can understand that it is tough to take the path of having your Development team write tests in small setups but that is the path that will help survive as the team/product grows.


Trackbacks / Pingbacks

  1. Triggered Testing | Knowledge Tester - June 12, 2018

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.