Usually when you look at diagrams that explain DevOps, they show few wheels that keep moving. This is what we can call, “code, build, deploy” cycle or sometimes called a CI/CD pipeline (CI: Continuous Integration, CD: Continuous Deployment or Delivery).
This looks simple as below that you need to be making changes to code consistently that will trigger automatic builds or the CI part, which in turn will trigger deployments or the CD part:
If you have seen the infinity shaped DevOps diagrams, you may be asking: “hey, where is Testing?”. My answer to that question is that it is no more a separate phase but is part of everything. The code commits ‘must’ have tests for changes; the build ‘must’ run tests and only passes if tests pass; the deployment is followed by some tests conducted in Production environment.
In addition to the above, these are some points for each of the wheels that I have seen/heard so far. Each of them calls for a separate post but wanted to sum up the whole here for now:
- Branching policy to manage multiple release
- Having a ‘Pull Request’ workflow
- Tests being inherent part of source code
- Peer Code Reviews
- Instant builds like CI
- Follow up builds that run additional tests e.g. Performance testing
- Builds only pass when all tests pass
- Developers own buildology rather than it being a separate team
- Manual vs. automatic deployments
- Testing in production
- Approval process
- Usage logs and monitoring
Many people when embark this journey of CBD cycle, they assume that everything will start working correctly as soon as they define a Build and Release pipeline. They discover soon that it is a bumpy ride and at times they become frustrated with this journey.
What I have experienced is that if you add two more letters to the story, you increase your chances of success significantly. These are A and F to make the picture as:
- Breaking your solution into small pieces that can be managed independently e.g. Microservices architecture
- Testability of system
- Test automation frameworks
- Tightening the feedback loop where Developer knows immediately when something fails
- Traceability on who broke what
- Ways to get feedback on how users are actually using the solution
That makes the plan ready which is “ABCDF” recipe for delivering quality with speed. And if any of the pieces is not ready, it may not work the way you expect it to.
Am I missing some other letter? How does your recipe look like?
The manufacturing world has gone through many revolutions to reduce the ‘lead time’ and Release pipeline is an attempt to do the same in Software world.
I should not make things complex for those of you who are not much familiar with intricacies of manufacturing world. So, let me make your life simple.
‘Lead time’ is the time from when you asked something and when you get it. Let’s take example of restaurant which is Services world but all of us have gone though that experience. You give order to the waiter who is serving your table and the lead time begins. The waiter serves the order and the lead time ends here. Usually either you’d ask, or the waiter would give you an estimated lead time depending upon what you ordered. This time includes waiter conveying it to the Kitchen, Chef picking up pieces which are prepared or not, assembling the order, doing the presentation, waiter collecting it and reaching your table. You must have also experienced that some restaurants maintain same lead time regardless of how many customers they are serving vs. there are some who cannot manage it.
Reducing lead time is one of the main functions of management.
Okay, now back to manufacturing world from where this term originated. It is the time when you place an order and you get it delivered. This typically happens in a supply chain system and you, as a Customer usually don’t experience it. But imagine that you visited a furniture shop, looked at different sets at display, selected one and ordered it. The lead time begins, and it ends when your furniture is delivered to your home after manufacturing/assembly.
The manufacturing world has managed to reduce the lead time by doing lot of inventions and ‘Assembly line’ is one of them. An assembly line starts with nothing and pieces keep joining it until a finished product is reached. You might have seen fancy documentaries on how car manufacturing plants work and I happen to visit air conditioning assembly lines myself to see how this works. Here is one such video on youtube. If you see more machines than humans, beware the same will happen in software world.
Sorry for a long background but I hope I didn’t lose you and you are still reading.
Now same thing is happening in software world. Your customer reports a bug or requests a new feature and the ‘lead time’ begins. The time when fixed bug or enhancement reaches the production, the ‘lead time’ ends.
Remember, reducing lead time is one of the main functions of management.
One of the things have been to introduce a Release pipeline which consists of changing the code, building the code, test execution, deployment to production in stages, testing in production. The main thing in keeping the assembly line moving is to have different pieces ready, the steps to integrate that piece into whole project reproducible, and automated/skilled labor to do it and move it to the next step. In an assembly line, if a particular station is functioning slow, it becomes a bottleneck. Similar effort is required here to first set up the Release pipeline and then remove any manual steps to automation if they are slowing down it.
The picture is from this article.
For example, Microsoft Visual Studio Team Services (VSTS) defines a Release pipeline where artifacts are produced from different ‘Build’ definitions and then a ‘Release’ definition takes them to production. If you like, you can call them pieces being assembled together to build a product. Similarly Jenkins has a ‘Pipeline’ concepts where you can chain together any number of ‘Jobs’ in any combination to achieve the same. There are so many tools selling the same concept.
As you can imagine that manufacturing world has gained lot of benefits by introducing assembly lines and software world is thinking to do the same with release pipelines. Reducing lead time is obviously the biggest gain. So, any step that stops the automated delivery in this pipeline would raise alarms and that has been a prime reason to do more and more automated testing. Such that when one step is complete, self-tests can assure that this integration was complete and without bugs. This looks easy but is hard to implement. I plan to write in some future posts that why this is so tough and some tips on achieving this as we are going through this transformation.
Are you aware if your team has a release pipeline? Do you know what are it’s steps and who manages it? Do you think your team should move towards it?
If you have any contact with Bollywood, chances are that you have heard the song ‘O baby doll main sonay de, yeh dunia pittal de’. The video of this song is beyond the circle of this blog, so I am not including the link here. Those of you who have seen it are smiling and those who have not seen it are gone to see it.
Welcome back all.
Sona (Gold) and pittal/peetal (Brass) are two commonly used metals. The things made of gold are precious and those of brass are cheap. This comparison has been done for centuries and elite people like Kings and Queens would use utensils made of gold and the common masses would use household utensils made of brass. Things of gold last for (almost) ever whereas things of brass need to be redecorated through a phenomenon called ‘Qali’ (alkaline).
(the photo is taken from: http://www.punjabiportal.com/forum/post430353.html )
This process of qali is also a favorite topic of some Punjabi songs. The most famous is perhaps ‘na main sonay de, na chandi de, main pittal bhari parat… way mainu dharti qali kara day main nachan saari raat’. Which means “I am not of gold or silver rather a dish made of Brass. If you would qali the land, I’d dance the whole night”. Again the search is up to the readers 🙂
Then the artisans who knew the art of Qali would move from one town to another and from one street to another singing a song ‘paanday qali kara lo’ which means you can qali your home utensils (made of Brass). One of the most accomplished comedians of Punjabi movies Munawwar Zareef picked this for a song (the video in the link is irrelevant, just listen it). As I was searching for this song on internet, to my pleasant surprise Mehdi Hassan also sang a song on this concept. You must listen it.
Enough digging 🙂 What we can learn from this and do our software need to be Qali’ed? Obviously yes. The inherent nature of software is that they are imaginary and it cannot be made of Gold in first attempt. It is always ‘Software pittal day’ (made of Brass) and require a Qali. What I mean here is the concept of continuous delivery which is getting lot of momentum. People are moving from yearly releases to weekly releases.
The street vendor singing ‘software qali kara lo’ is in your area. How would you Qali your software?