Automation is not automating manual tests
The first thought that comes to a mind that doesn’t understand Testing and Automation is this: “I’ll automate all tests that are being executed manually and it will save time”.
This is what I thought almost a decade ago and added a column in the test plans that I used to execute. That column was named something link “Automated Test Id” and I started assigning Ids to each test that I automated and cross referenced it here. Looks nice, isn’t it.
Soon cracks appeared in my proposed system. There were some tests that I couldn’t automate e.g. the ones that needed visual verification of Engineering components. There were some automated tests that when I wrote, I added additional checks than what the manual test proposed e.g. checking more negative testing scenarios as it was easy to do that in automated tests compared to manual tests. Then there were new tests that I started to think of while running either automated or manual tests. I wasn’t sure if I should first write a test script for them and then add it’s entry in the test plan sheets OR I should first add it to my manual test plan and then automate it. This was one of the problems where I started to seek help from the internet and I concluded that my strategy was flawed. But I learned the following:
Manual tests has no correlation with automated tests. If they appear same sometimes, that is just a coincidence.
I also thought that anyone who has spent any serious effort in test automation would know that. But last week I saw the following in a project being managed in Microsoft’s Team Foundation Server:
It was okay to have this dialog in 2006 but towards the end of 2016, it is a sin. I’m not sure if this is standard one coming from Microsoft or it was done by some geek who manages the project.
Joel Spolsky said around that time in this context of automating all tests:
And so one result of the new emphasis on automated testing was that the Vista release of Windows was extremely inconsistent and unpolished. Lots of obvious problems got through in the final product… none of which was a “bug” by the definition of the automated scripts, but every one of which contributed to the general feeling that Vista was a downgrade from XP. The geeky definition of quality won out over the suit’s definition; I’m sure the automated scripts for Windows Vista are running at 100% success right now at Microsoft, but it doesn’t help when just about every tech reviewer is advising people to stick with XP for as long as humanly possible.
When people have the above-mentioned thought process, they say/ask the following things. I’d suggest to stay away from these:
- Our testing is 75% automated. See this is because they believe that we have this many tests in total and out of which, that many are automated.
- Through automation we have saved 250 hours. Which means that if those tests were executed manually for this many times, such would be the time taken to accomplish this.
- By automating all our testing, we’ll downsize our testing team. Because they will replace the manual tests being executed by those testers.
- Which manual tests I should be automating?
The above statements keep coming because a) the tool vendors create this hype as they want to sell their phakkis and b) the management who runs the show has minimal understanding of what testing is and what automation is.
Have you said or seen above understanding on test automation? Do you believe in them or have any thoughts on this?