Advertisements

Build Pass when Tests Fail

It happens to all of us. We are used to doing a process in a particular way for years and never think of other ways of doing it. Then someday someone says something. That serves as an eye opener and we start seeing other ways of doing the same thing.

This happened to us for our rule: “Break a build if a single unit test fails”

Sounds very simple and rational. We have this rule for may be 10 years and I have repeated this over and over in all new projects that we took up in these years. Our structure follows the Plan A as shown below i.e. to run tests as part of the build process and if a single test fails, build fails.

What changed in year 2017 was a quest to find ways to release faster, you know the DevOps like stuff. So we started to look at the reasons for the build to be broken. We build our source 4-6 times a day, so we had enough data to look into. One of the reasons was always a failing test.

Now we thought, as you must be thinking by now, that it is a good thing. We should not ship a build for which a given test breaks. But our data (and wisdom) suggested that failing tests were in following three categories:

  • The underlying code changed but test code was not changed to reflect this. Thus test fails but code doesn’t fail.
  • The test is flaky (expect a full blog post on what flaky tests are and what we are doing about them). For now, flaky test is that passes and fails with same code on different occasions.
  • The test genuinely fails i.e. the feature is indeed broken.

Now 1 and 2 are important and Developer who wrote the test need to pay attention. But does this stop the build to be used? Of course not.

3 is a serious issue but with the notions of ‘Testing in Production’ combined with the fact that fix is just few hours away, we figured out a new rule as shown in Plan B.

Yes, when a build fails due to a failing test, we actually report bugs for failing tests and declare the build as Pass. Thus the wheel keeps rolling and if it needs to be stopped or rolled back, it can be.

A few weeks into this strategy while all looked good, it happened what was always feared to be happening. A build in which 20% of our tests were failing was declared pass and our bug tracker saw around 100 new bugs that night. Let’s call that spamming.

That raised our understanding to move from binary (fail or pass) to a bit fuzzy implementation. We came with a new rule that if 10 or more tests (where 10 is an arbitrary number) fail, we’ll declare the build as failed. Otherwise we’ll follow the same path. So this is now our current plan called Plan C.

I know what you are thinking that a single important test could be much important than 10 relatively unimportant tests. But we don’t have a way to weigh tests for now. When we have that, we can even improve our plans.

Does this sound familiar to your situation? What is your criteria of a passed build in context of test results?

Advertisements

Tags: ,

4 responses to “Build Pass when Tests Fail”

  1. Saher Tabbasum says :

    I am not a much expert tester, but after reading the post, I think that it is not imp indeed that only if the failed test count is greater than 10 only then we can declare a build to be failed, but , I think what kind of test failed is also important to be known. The plan will hold more worth if their is an additional check on 10 count failed test to also check what kind of test are failed. That will help to declare a build pass or fail.

    Like

    • majd says :

      Your answer suggests that you are an expert tester. You are absolutely correct that kind of failure is more important than number of failures. As I mentioned, we do not have a way to categorize our tests, so this decision is currently not automated. But we’ll tackle that problem some day.

      Like

  2. Elif says :

    After count>0 check, i further check that this count belongs to products functionality or quality critically or not. If yes, then fail, if no then mention fail tests and pass

    Like

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

w

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.