Once More about Performance Testing Terminology and Skills
Reading about new SOASTA offers I stumbled upon the following statement: “The subtle difference between pure ‘load testing’ and performance testing is the former is about driving synthetic user traffic, and performance testing is about measuring critical data during the effort. The problem and difference between the two is the focus on the effort and results.”
While it is great that SOSTA started to value “measuring critical data during the effort” more, I don’t see it this way. Yes, word ‘load’ in load testing stress that you create multi-user load on the system – but to make it meaningful you must measure “critical data during the effort”. Otherwise it is not an effort to learn how the system behaves under load, but rather wasting resources and providing misleading information.
It is interesting that there are quite different (and often contradicting) definitions of load and performance testing. I believe that performance testing is just a wider term than load testing. For example, we may have single-user performance testing, but it doesn’t look like single-user load testing makes sense (unless, perhaps, when we create a single-user baseline before we start multi-user tests). But as we get to multi-user workload – I use these terms rather as synonyms. I may use one over another depending on what aspect I want to highlight – load or performance – but I definitely believe that the process here is the same. You must understand what are you doing and what is happening with the system regardless of what term you prefer to use – otherwise it just discredits the whole activity.
Why it bothered me enough to write about it? The terms still have a slight difference in emphasis. Just because the quote rather degrades the term ‘load testing’ placing closer to blind load generation. While, unfortunately, there are many cases when load testing is done this way – everything can be done in a wrong way – I don’t think that it is a good idea to equate ‘load testing’ to such bad practices. Simplistic blind load generation discredits the overall idea – and, unfortunately, I have heard too many times from different people that they don’t do load testing because [it doesn’t represent real load, it doesn’t provide real insights, etc., etc.]. No, such simplistic blind load generation is not load testing – it is just an example of bad work (of course, it may be cases when you need to create a simple load – but it should be with full understanding what and why you are doing and definitely not in a blind way).
Actually it is surprising that there is rather a condescending approach to load testing at companies putting very high value to the art of programming. They are very serious about programming skills – but somehow believe that every developer many do load testing if needed (and this way they get confirmed that it doesn’t bring value). Not saying that a good developer can’t do load testing – but it may take a couple years, preferable working with somebody experience in it, to get to real mastery. Even a worse case is when companies try to find inexpensive outside resources for load testing. Not saying that you can’t find good outside resources – but such resources are rather few and far between. So the problem of discrediting of load testing by the people who don’t know how to do it properly is very serious (exaggerated by the fact that management often don’t even know what to check).
That is the reason why I believe that people who know how to do load / performance testing should be careful and not to degrade terms just to make a marketing point – it would probably hurt everybody in a long run. It would be better to work all together to increase understanding and improve reputation of load testing in the industry.
