Pitfalls of Continuous Integration

Continuous Integration (CI) has gotten fire in recent years and has been one of the features of agile teams. Simply put CI is a safety net to enable faster changes to the code. Since we are living in times where rate of change has increased than ever, we like to go for quicker releases and shorter life cycles. Wikipedia defines the concept well by saying “it is the practice of merging all developer working copies with a shared mainline several times a day” and lists all practices like merging very frequently, automating the build, self-running tests, every commit should be built and so on.

My favorite diagram to explain the concept is given below. Imagine that in a team, Ali works from Pakistan, Jim works from US and Rakesh adds value from India. Now whenever is the time of the day, Ali or Jim or Rakesh check in their code to the repository, this is what happens.

CI steps

Thus if a single test fails which was passing earlier, the commit is denied and the respective programmer is notified.

Though it is believed that there are tools like Jenkins, Hudson etc. that enable CI, CI is a concept and you can do this with a simple batch file. The project that I’m working on these days, have build scripts written in Python that do this magic.

Okay, so this was the background and this looks like a very good solution for producing quality at high rates. But CI is not the end of quality measures rather it is one of them. Being at the receiving end of code, as a tester I can tell that teams that implement CI have better quality than others. But I have also observed some dangers lying within.

What tests can’t see, gets escaped. When you have 10,000 tests in project, you get in the trap of numbers. Unless you know the areas that are being covered by those tests, the code which is not touched by the tests will always have issues. One way of getting around this is periodic review of tests by the entire team specially skilled testers to see what else we can add to the suite.

The ignored hell. The design is such that if a single test fails, the build process fails. Thus when tests fail (which they can any time) and you are in need of a build for whatsoever reason, the tendency is to put those tests in ignored list. Tests that take time are also put in ignored list for faster delivery of the builds, so usually all performance tests are in ignores list. With time, these ignored tests get ignored by the team and present a risk. One approach is to keep a backlog item to review ignored tests at the end of each iteration.

Programmers have affiliation with code. The bulk of these tests are unit tests and even if they are scenario level or integration tests, they are mostly written by programmers. Some Project Managers gets so much obsessed with these tests that they start believing that they don’t need any specialized tester in the team and programmers can test all. The problem is that whoever creates something have an affiliation with it. Programmers are no exception and they love the code they have written. So all their tests are focused to prove what their code can do, rather than where the code fails. One trick that works is pairing these programmers with testers to test the code. Even better is to have testers in team that write code to test code. They’ll bring in all stuff that programmers can’t see.

Have you experienced working in a team that works in CI mode? Have you observed any success and cautions that you’d like to share.



11 responses to “Pitfalls of Continuous Integration”

  1. meetzaki says :

    //ranting ahead 🙂

    As a strong advocate of CI and a proponent of full-stack development, I feel compelled to slightly disagree here. I have, in past more than a year, researched and worked with my team to adapt to the agile/hacker way that you have mentioned. The area where I disagree can be better explained by the following two links.

    First, testing/deploying Facebook.

    Second, testing/deploying Quora.

    Third, Github.

    I wouldn’t put the credit of all this on Mark Zuckerburg, but he did, in fact, bring to limelight the hacker way. The Hacker way has changed many of the tenants of traditional software development practices but here are my personal, key takeaways.

    – Hire people you can treat as adults. Adults don’t need supervision and potty training.
    – Create a shame culture around your code base for breaking things.
    – Ship early, ship often.
    – Move fast, break things.

    Also, I see the post to be slightly on the apologetic side from QA. As head of engineering at my organization where I am pushing for full-stack application developers, I also have introduced a notion of full-stack quality engineers. More often than not, our emphasis on manual functional testing is so deep that we forget about the test engineer’s complete stack which includes the following non-exhaustive/generic list:

    – functional testing
    – db testing / integration testing
    – security testing / penetration testing / exploit testing
    – performance testing / load testing / stress testing / availability testing?
    – M&A quality compliance (which encompasses everything from security to legal issues)
    – maintaining quality metrics (A key part of CI is the static analysis of code files which means the code has to pass a certain criteria – there are several linting tools for that)
    – configuration testing (especially if the deploy target varies greatly)
    – infrastructure testing(including compliance to software infrastructure quality metric)

    How much of the above stack is really being served? I’m gambling on this one, but above is my stack and I’m going to make my team live up to it to the fullest. For it to do so, developers have to up their game and act as adults and not “ignore the tests”. (I’ve had several episodes of that in the company – was quiet embarassing for developers). Once we are clear of the tedious things like deployment and unit testing can we only focus on this quality stack.

    While traditional, mission critical, domain specific software and its quality assurance is still pretty relevant and what you point out holds very true, the new vistas that the software is exploring on the order of billions requires much deeper thinking than that, IMHO. Distributed systems like Facebook, Github, Whatsapp, Quora and several systems inside google, plus the plethora of services that third-parties integrate with are at the cutting-edge and are devising new ways of testing on the full-stack. We need to see how our quality initiatives scale proportional to the scale of systems.


    • majd says :

      Thanks Zaki for sharing your thoughts and a ‘blog post size comment’ 🙂 It certainly adds value to this discussion and you brought very good points in support of CI.

      First, I’m glad that as head of Engineering you are considering ways to improve quality. It is much appreciated. Second, the intent of the post is not to criticize CI or suggest to abandon CI, rather move to the next level i.e. once CI is in practice, look for the holes and fix that. I’m sorry if some of the statements look against CI, but let me say clearly that CI certainly adds value.

      Regarding having focused tester to compliment testing efforts from rest of the team, the idea is to reinforce quality built in the code by running exploratory and scenario tests. You can read more about Agile testing quadrants in ‘Agile Testing’ book by Lisa Crispin and Janet Gregory that explains how spreading testing efforts brings the best quality.

      Thanks again for your detailed analysis!


    • H Hamid says :

      Good share


  2. meetzaki says :

    Also, a link worthy of a flamewar 😉


  3. Abdul Aziz says :

    We moved to CI some time ago and there is no looking back. CI may have it’s pitfalls as you mentioned if not used and configured properly. The CI tools are so configurable that you can define specific predefined or custom actions (as you mentioned scripts) on any event. The real power of a CI server can be utilized when used with modern revision control systems (like Git/Bitbucket) and integrated with your project and issue tracking software (like Jira).


    • majd says :

      Thanks Aziz for sharing your experience. It is true that CI certainly helps a lot in producing quality code faster. You said it well that all these pitfalls can be avoided by proper configuration and care.


  4. H Hamid says :

    Good post with very valid points. Test engineering is evolving each day. Also, good points shared by Zaki. Now the discussion about having a shame culture or a test engineer to produce high quality products, really depends on what kind of products you are building and what kind of development process you are following. Facebook has got rid of the test title but not the test activities (they have added more layers to different testing levels). Teams where software engineers are doing both test and development, may have a different way of managing both. It is not possible without a right mix of culture, tools and mindset. If they can make it work, other teams can do the same. The same article also mentions that “This process works for Facebook partly because Facebook does not, by and large, need to produce particularly high-quality software”. So may be we need to consider that aspect of discussion as well. 🙂


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.