Thoughts on Performance Testing
Every now and then, I’m been involved in discussion with fellow testers on the topic of Performance Testing. I realized that I never covered this on the blog. So here you go.
Note: There are many types of Performance Testing and below notes are related to checking time based performance of systems. Though these apply to other types also, but it’s always good to have a context.
Performance Tests don’t pass or fail
In a way that the result of a test is always a binary: either they pass or fail. Whereas Performance tests give you a number and only you have to decide if that number is reasonable enough to call that a Passing test. For example, you observe that the load time of your website’s home page on IE 10 is 4 seconds. Now is it good or bad performance is always a subjective discussion and unless you set a benchmark, the number looks very bad, bad, ok, good, very good etc.
There can be multiple benchmarks
So as mentioned above, the real question is that against what number we are comparing our test results. Those are called benchmarks.
The benchmarks can be a variety of things including:
- Your competitor’s similar solution
- Your last commercial release that is with the client
- Your weekly/daily alpha release
- A number that comes out from the person who is responsible for release.
- A number that comes out of some Usability lab conducted with users.
It is also possible to record actual number and then represent it with compared with one or more or all benchmarks.
(the original photo is here: http://idmdude.com/2014/03/07/its-ok-to-get-stressed-out-with-openam/)
Play around only one variable
Be mindful that performance measurements need to be done in ‘controlled environment’. If you are comparing a number coming from Machine A with Machine B, that is not a comparison as more than variables at play. So rest of the stuff will stay the same serving as baseline and two readings will be taken by changing one variable.
Again looking at the above example of load time of website, you’ll have to check it on the same machine with same memory resources, same set of other processes loaded and then compare the time. So if you are doing a comparison with benchmark, on same machine repeat the test with one or more benchmark. Now you have results that you can compare.
Take multiple measurements
Having one reading can be misleading and may not present the true picture. Depending on the task being measured, take multiple measurements like 5 or 10 or more and then take an average to get the real value.
Tools will help you
There are many tools that will help you measure time or memory. They also help you orchestrate your tests one after another and then provide you with set of numbers. Once you know what kind of testing you want to do, then picking a tool can be easy. For example to check for load time of website, this tool will be handy for Firefox.
Did I miss any thing on the general overview of performance testing? What things you’d add to this list?