A Tester has to juggle between lot of activities: understand the system under test, learn the domain for which the system is designed, design tests, execute tests, find bugs, report bugs, prepare meaningful testing reports, plan testing, prepare test data, prepare test environment, deploy system under test and so on.
If you look at the above list, it can be categorized in two blocks. First being intellectual or human work that includes learning, designing, planning etc. The other block is the dirty work that every job comes with and doesn’t consume much of your brain cells. The things that I mentioned earlier as Test supporting activities. These are the things that you should try to put them on ‘auto’. With the understanding that tools fit into your testing strategies and not define them, believe me Python is the best tool to do that.
To get you an idea of what tasks I’m exactly talking about, in our current project we have written Python scripts to do the following:
- Download build from the ftp server. The task is scheduled and looks every few hours to check for the latest build. The script downloads it, unzips it, places it at desired location and sends email to all the team members.
- Upload the testing results on SharePoint server. When some of our testing tasks finish, we hand over the results to this script which massages data, makes them more ‘management-friendly’ and uploads at a desired location on SharePoint server so that all can see it.
- Regression testing tasks. One of the required regression for our project is to make sure that latest build works well with existing data. So we have ~250 different type of models that we pass to the build and see if it all goes well. This is again done through a Python script.
- … You get the idea.
(the original image is here)
Now your could argue that any scripting language can be used to achieve the above, so why I’m loving/promoting Python. To be honest, in some earlier endeavors I’ve used VBscript, AutoIt and C# projects to accomplish the above. So all are good but since life is short, these things make Python more loveable are:
- Batteries included. Yes it comes with all you need to play with. Test supporting activities have a lot of a variation to be able to work with XML, Excel, SharePoint, Batch files and so on. Python has a solution for all.
- It’s Portable. The same scripts that run on Windows are running on Mac and on Linux for us. The modern world is so more complex and so more wide that portability is gem of a feature.
- It’s easy. If I can learn and use it, any tester should be able to learn it. And it is also backed up by very active development community that would share code samples and answer your questions if you run into trouble.
Do you have a list of support activities? Have you put them on ‘auto’ and how?
Recently a tester friend of mine discussed one of the classic problems of testing. You are about to release your next version and tested all areas on a “Release Candidate (RC)” build and then you find one issue that is big enough to ask for a new build. Now you get another RC build. Should you test it all as if this is the first RC build? Or should you start from where you left the last build? Or should you just test the fixed issue? Should you test all or test small?
As a hardcore tester, everyone is taught to be suspicious of things. We apply “guilty until proven innocent” formula to everything that we test. And given that we know from experience that in software world one change can upset many unwanted things, the first preference seems to be Test all.
But here is the dilemma. The whole set of tests take many days to complete and time is short. And all those people called Project Managers, Release Managers, Product Owners etc. etc. are asking you after like every hour “How are the testing results? Can we release now?”
Let’s see if we have a case for Test small i.e. only test what has changed. I believe yes and here are the things that you need to achieve this:
Know what has changed
The biggest challenge is that until the testing team has a very good insight into what is built in the system and how it is built, they’ll continue to consume all builds including RC as black boxes. See the figure below that if you receive two black boxed builds, you can’t tell what has changed. But if you can see inside those boxes, you can tell what has changed.
How can you look inside the system?
You need to have all the details of how your source control is maintained. Is it a trunk based development or branch release system? Access to changes made to the code so that you exactly know what happened between the two builds. Many build systems can generate a ‘diff’ report between two builds. Work with the programmers in your team to have those reports coming out with every build. There are tools that can compare two installed versions of an application and even you can write your own tools to know what has exactly changed. Take control of this system rather than waiting for the information to arrive from others.
Now that you are sure what has changed, your test strategy for the latest RC will be just around that change and not a “Test all” strategy.
Unit tests are the safety net
Yes, you know that right. If your programmers start writing Unit tests and you befriend them to support this activity as a testing unit, you can see if all tests that passed on last RC are passing on the new RC, things could be alright in those areas. The bigger is the scope of Unit testing effort, the safer are your releases.
History repeats itself
Keeping a track of what happened in the release cycle until the latest RC is always a good indicator of what will go wrong in this build. Say some of your modules have been working fine in all previous runs of testing and the change is not in that module, you can safely ignore such modules for the latest RC. However if some modules created problems here and there in the release cycle, adding them to “Test small” strategy might be interesting.
How do you tackle this problem? Do you do a Test all or you have discovered some form of Test small?
Last week, I was invited by a group of “soon to be graduated” students of BS in Computer Sciences at FAST Islamabad campus. One of their subjects require them to host a talk by someone from Industry to complete the course. From the list of topics, I chose to speak about “Careers in Software Testing”.
Since on one of my previous tours, I had talked about taking a career in Software Testing, this time I focused more on what happens if you take that route. The main points that were laid down to the audience were:
- Software Testing belongs to the Software Development as a whole. So careers in testing are similar to careers in Software Development.
- The first few years of this career is establishing yourself as a technical resource.
- Once you gain that position, you have two options. One is to become a people’s manager and start rising on the management ladders. The other is to remain a technical hub of knowledge and become the person who defines the technical path of the company.
- The technical paths in Software Testing are various. The latest hot ones are Performance testers, Security or Penetration Testers and Automation Engineers.
Picture taken from Haibye.com.
- If the above looks too complicated or too much effort, try starting your own business and you’ll be CEO on the first day.
- Presented Microsoft as a case study which offers jobs in Testing.
- Covered how testing in Pakistan is growing and growing and growing.
The slides are shared here if you want to take a closer look into the above contents.
Did I miss something? What else we should tell graduating students about career paths in Software Testing?
Usually people draw comparisons between testing effort and coding effort (which some also call Development effort whereas Development is the whole process of implementing requirements which includes testing). To use this as a trick question, I tried asking this in one of the recent interview session:
“For a given feature/user story, Programmer estimates that it will take 3 weeks to write code. How much time you think, you’ll need to test this feature?”
I got answers like:
- At least one week of time. This is based on my experience.
- I’ll ask for half the time as it takes at least 50% of the time to test.
- To be on the safe side, I’ll ask for 60% time as my experience says it usually takes 50% time and then we may encounter more problems in a given feature.
Clearly the tester is thinking of testing effort is directly proportional to the coding effort. Even some people also have 10% unit testing tax formula or some suggesting law of diminishing returns on testing.
For me testing effort is not proportional of the coding effort and the answer is “it depends” :). Let me explain with two experiences.
Few years ago, with introduction of Windows 64bit versions, one of the teams I know took the approach to let their applications run as 32bit applications in 64bit environment. The did minimal changes to announce that their software “supports” 64bit and then asked the testing team to confirm each and every features “works” as expected. Easily in this case the testing effort was way more than the coding effort given that the application was a huge one.
In another project, one of the teams was having performance issues in their system. They decided to flip the design scheme to take it from ‘attaching B to A’ to ‘create A and link with B’ and it took them couple of months effort to change all the code to get in line with this new design philosophy. Once done, testing team was asked to do performance benchmarking. As the UI layer never changed, the set of tests were executed without any issues and new numbers were reported in a few days. In this case testing effort was almost not notable as compared to the coding effort.
So when you are asked next time, “how much time it will take to test?” , you should first understand what the problem is you are asked to test and then negotiate a good number.
Do you ever developed a formula for testing efforts? And do you still want to use it?
It was 1881 when the French undertook the project of Panama canal with the ambition to join the Pacific and Atlantic oceans. The idea was there ever since this part of land was discovered by Vasco Núñez de Balboa (against whom name is the Panamian currency called Balboas) and even in 16th century, the Spanish wanted to construct a canal but they did not have the technical expertise needed for this at that time. To take part of this strategic path, a rail track was constructed by the Chinese in 1860s where ships were off loaded on one part of ~70KM land and ported through rail to the other side where another ship would take the load and keep moving.
The French seemed well prepared. Ferdinand de Lesseps who was the chief engineer of the famous Suez canal along with his team brought the most experienced workforce in this domain. The plan was prepared with all the geographic considerations and work started at a real good pace. But in 9 years of construction, the project was still any thing but on target with about 20,000 workers losing their lives. No not in the blasting or other construction related injuries or the venomous snakes hidden in the rainforest, but due to the yellow fever and malaria. Given the loss of workers, the morale of the rest of the workforce was really down. There were other factors also and the project was abandoned and it appeared that such a canal is not possible.
(Photo by me. Panama Canal expansion project. New set of locks on the Atlantic side)
Then the Americans took over this project in 1903 and used some part of the work done by the French earlier. They knew that the project didn’t complete due to the loss of workers rather than any technical grounds, so they first thought to solve that problem. There was a sanitation office setup with this special mission to eradicate yellow fever, malaria and similar diseases in this region. There was investment in extensive sanitation projects, including city water systems, fumigation of buildings, spraying of insect-breeding areas with oil and larvicide, installation of mosquito netting and window screens, and elimination of stagnant water. This did not completely eliminate the malaria like diseases but helped a lot in controlling it which resulted in the canal completed almost on scheduled and starting operating in 1914. It is now 101 years that canal is still operating and now an expansion of canal is underway that will double the passage of ships in a day which is estimated to open in 2016.
The above is a summary of the story of Panama Canal that I heard during my recent visit there. We enjoyed the beautiful beaches and diverse wildlife in the rainforest but the most talking goes around the Panama Canal (including the famous Palindrome “A man, a plan, a canal, Panama”). And using the ‘over-simplification logic’, I concluded that success of building the canal was not coming with a beautifully engineered design or well orchestrated work. Rather it was about supporting and protecting the workforce from diseases like Malaria so that they can focus on the work.
And then I looked around the modern day workplaces and the diseases that make them toxic. There are Malaria like attacks from distrust, dishonesty, lack of communication and not appreciating the lives of talented workers.
The workplaces that work on removing such diseases first rather than solving the technical problems, produces wonders like Panama Canal. Are you willing to do that?
Every now and then, I’m been involved in discussion with fellow testers on the topic of Performance Testing. I realized that I never covered this on the blog. So here you go.
Note: There are many types of Performance Testing and below notes are related to checking time based performance of systems. Though these apply to other types also, but it’s always good to have a context.
Performance Tests don’t pass or fail
In a way that the result of a test is always a binary: either they pass or fail. Whereas Performance tests give you a number and only you have to decide if that number is reasonable enough to call that a Passing test. For example, you observe that the load time of your website’s home page on IE 10 is 4 seconds. Now is it good or bad performance is always a subjective discussion and unless you set a benchmark, the number looks very bad, bad, ok, good, very good etc.
There can be multiple benchmarks
So as mentioned above, the real question is that against what number we are comparing our test results. Those are called benchmarks.
The benchmarks can be a variety of things including:
- Your competitor’s similar solution
- Your last commercial release that is with the client
- Your weekly/daily alpha release
- A number that comes out from the person who is responsible for release.
- A number that comes out of some Usability lab conducted with users.
It is also possible to record actual number and then represent it with compared with one or more or all benchmarks.
(the original photo is here: http://idmdude.com/2014/03/07/its-ok-to-get-stressed-out-with-openam/)
Play around only one variable
Be mindful that performance measurements need to be done in ‘controlled environment’. If you are comparing a number coming from Machine A with Machine B, that is not a comparison as more than variables at play. So rest of the stuff will stay the same serving as baseline and two readings will be taken by changing one variable.
Again looking at the above example of load time of website, you’ll have to check it on the same machine with same memory resources, same set of other processes loaded and then compare the time. So if you are doing a comparison with benchmark, on same machine repeat the test with one or more benchmark. Now you have results that you can compare.
Take multiple measurements
Having one reading can be misleading and may not present the true picture. Depending on the task being measured, take multiple measurements like 5 or 10 or more and then take an average to get the real value.
Tools will help you
There are many tools that will help you measure time or memory. They also help you orchestrate your tests one after another and then provide you with set of numbers. Once you know what kind of testing you want to do, then picking a tool can be easy. For example to check for load time of website, this tool will be handy for Firefox.
Did I miss any thing on the general overview of performance testing? What things you’d add to this list?
No one can look inside you to find out how good you are at something. Everyone judges you from what you say and what you do and that is why if you want to build your repute as a Knowledge Tester, you should stop saying these 3 things.
- Where are the Specs
When I started learning the craft of software development in late 90’s I liked the quotes that said that “You can only walk on water if it is frozen. And you can only work on requirements if they are frozen”. (I couldn’t find the origin of this; can any one help?)
Fast forward many years and the constant change has made it impossible to freeze the requirements. Our users, their requirement, their priorities, their environments, the technology changes at such a fast pace that freezing them is against the laws of Physics.
So when you start on a new project or feature as a tester, never ask ‘where are the Specs’. Because chances are that either they don’t exist or they are outdated/ out of sync from how the software currently behaves.
I know what you are thinking now that how we can move forward if we don’t know what the software is supposed to do. The answer to that is “find out”. Yes, we can find them out through exploration and invoking Three Amigos conversations with Programmers and Customers.
And if someday you accidentally find the Specs, add them to your list of resources that reveal the ‘truth of expected behavior’. Don’t take them as the absolute resource for this purpose.
(the photo is taken from this article: https://www.linkedin.com/pulse/20141020052253-64875646-50-things-you-must-stop-saying-at-work )
- How should I test this?
We often get into this trap that as we engage other members of the team in testing, we ask their advice on how the systems should be tested. There are two problems associated with that kind of statement.
First, people can start doubting your skills by thinking that ‘we hired these guys to test and they ask us how to test. If we knew that, we could have tested this at our own’. The second problem is that you start following the thought process that produced the solution rather than thinking of new ways to look at the problem. Also at times, Programmers give their undone projects to testing team as soon as they are asked how to add more tests.
The right question to ask is: ‘What should I test?’ and then keep on extracting information, absorb that information and then come up with a plan so as how you’ll test it. There is no harm in saying that ‘this is how I plan to test, what thoughts do you have’ as in that case you’ll get answers driven by your test strategy and not following what others suggested you.
- I need more time to test
This is a killer statement and if you ask Project Managers about what they hate to hear from testers, this statement will be on the top of the list.
The thing is that testing never ends and it is impossible to find all the time to run all possible scenarios to find all the problems. In fact as you spend more time in testing, you figure out that you need more time for it. See how similar is this to spending time on vacation: you always need more time.
But the fact is that we don’t have time.
So, rather than looking at the equation from testing perspective, we need to look from the time perspective. Set clear rules that if you are given one month to test, this is what you’ll do. If you are given one week to test, this is what you’ll do. And if you are given just one hour, this is what you’ll test. On one of the release decisions that I’m part of, we are down to only 24 hours throughput time on testing. You tell us today you want to release it, within 24 hours you know ‘yes’ or ‘no’.
You might be thinking that I’m promoting automation but in fact I’m not. It is true that quick feedback from testing can be fast tracked through automation but defining clear priorities on what will be tested is what will get you there.
Do you also think that we should stop saying these things as Testers? If no, please let me know why. If yes, what else we should stop saying