Using Logs for Performance Testing

Performance testing is one of the specialty area for testers and is one of the challenging tasks. Designing Performance tests is different than designing functional tests, then setting up testing environment for Performance also needs special care and finally figuring out if the results do indicate an issue is not straight forward. One thing that has helped us lately in troubleshooting Performance issue is logging.

It is a known fact that logging is a Tester’s friend. Rather I’d say that it is a friend for any troubleshooter. The logs are like the traces which a skilled investigator follows to find the culprit who committed the crime. These are “Khurras” (Punjabi meaning foot prints) which “Khojis” (Punjabi/Urdu/Hindi meaning Detectors) follow. If the crime was committed (read an issue was observed) but no “khurras” are there (read no logs are there), it is not possible for the “Khoji” to find out the criminal (read what is the root cause of the issue).

(Original photo is here)

The first question in setting up logs is what logs we need? These have to be key points in software behavior for which time/memory captures are added to the log. For example in one of our Display systems where we implemented logging, we added logging for the following:

  • The time to open the file.
  • The time to display the thumbnails in the file.
  • The time to fill view with selected thumbnail.
  • The time to show properties when an element is selected in the display.

Once you know what points you need to log, the next step is to hook in logging. I’m hoping that your software or the tools/technologies that is being used by your software already has logging infrastructure in it and you have to add a new logger for Performance. And if it is unfortunately not there, then you need to have such infrastructure first. This logging will be added by the Programmers in your team and thus this project is another example of how Programmers and Testers can work together to deliver quality.

Suppose you have those friendly Programmers in your team and Performance logging is enabled. Now all you have to do is turn it on and run your existing tests. That’s it. Along with the numbers that measured before, you’ll get the logs.

Remember when all is going good, no one needs logs.

So you’ll need the logs only when your results are indicated a decline in Performance numbers that you measure. And then you can pass it on logs directly to the Programmers to investigate. Or if you are a smart tester like me, you’d write a small Python script to parse the log and get the meaningful numbers that you report to your team.

We’ve tried the above and believe it is really helping in issue investigation. We have moved through this from “We have poor Performance numbers” to “The Performance is down because file goes through up gradation progress while opening” like concrete issues being reported and fixed. Try it and you’ll love it. It will take some effort from both the Programmers and Testers to do it but it is surely worth an effort if you are serious about Performance testing.

Do you use logging in Performance testing? Or you have some other interesting ways to investigate Performance issues?

Tags: ,

2 responses to “Using Logs for Performance Testing”

  1. Aziz (@xeeez) says :

    Very informative article! A couple of things crossed my mind that I’d like to share 🙂

    1. From experience I feel that logging may not be the most reliable mechanism when done on a developer or tester’s machine. For instance, a performance issue that has been noted on production may not be best reproduced on a developer’s machine and hence may have a slight chance to lead you to false positives 🙂

    2. The way a tester/developer writes a log statement can help them analyze the results more efficiently. If someone doesn’t want to write separate code to analyze the log file, they can spend some time to properly design and write the log statements so they can perhaps be converted to a CSV and opened in a spreadsheet where one can leverage on built in formulas to perform better analysis e.g. what was the “Average Execution Time” of a certain function across 100 separate runs.

    Like

    • majd says :

      Thanks Aziz for adding experience based info. Yes, CSV or any other structured logging is more valuable than plain text.

      One point that I forgot to mention is that logging itself slows down Application performance. So logging should be used with care.

      Like

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s