Using Logs for Performance Testing
Performance testing is one of the specialty area for testers and is one of the challenging tasks. Designing Performance tests is different than designing functional tests, then setting up testing environment for Performance also needs special care and finally figuring out if the results do indicate an issue is not straight forward. One thing that has helped us lately in troubleshooting Performance issue is logging.
It is a known fact that logging is a Tester’s friend. Rather I’d say that it is a friend for any troubleshooter. The logs are like the traces which a skilled investigator follows to find the culprit who committed the crime. These are “Khurras” (Punjabi meaning foot prints) which “Khojis” (Punjabi/Urdu/Hindi meaning Detectors) follow. If the crime was committed (read an issue was observed) but no “khurras” are there (read no logs are there), it is not possible for the “Khoji” to find out the criminal (read what is the root cause of the issue).
(Original photo is here)
The first question in setting up logs is what logs we need? These have to be key points in software behavior for which time/memory captures are added to the log. For example in one of our Display systems where we implemented logging, we added logging for the following:
- The time to open the file.
- The time to display the thumbnails in the file.
- The time to fill view with selected thumbnail.
- The time to show properties when an element is selected in the display.
Once you know what points you need to log, the next step is to hook in logging. I’m hoping that your software or the tools/technologies that is being used by your software already has logging infrastructure in it and you have to add a new logger for Performance. And if it is unfortunately not there, then you need to have such infrastructure first. This logging will be added by the Programmers in your team and thus this project is another example of how Programmers and Testers can work together to deliver quality.
Suppose you have those friendly Programmers in your team and Performance logging is enabled. Now all you have to do is turn it on and run your existing tests. That’s it. Along with the numbers that measured before, you’ll get the logs.
Remember when all is going good, no one needs logs.
So you’ll need the logs only when your results are indicated a decline in Performance numbers that you measure. And then you can pass it on logs directly to the Programmers to investigate. Or if you are a smart tester like me, you’d write a small Python script to parse the log and get the meaningful numbers that you report to your team.
We’ve tried the above and believe it is really helping in issue investigation. We have moved through this from “We have poor Performance numbers” to “The Performance is down because file goes through up gradation progress while opening” like concrete issues being reported and fixed. Try it and you’ll love it. It will take some effort from both the Programmers and Testers to do it but it is surely worth an effort if you are serious about Performance testing.
Do you use logging in Performance testing? Or you have some other interesting ways to investigate Performance issues?