Load Testing: Its Present and Future

Recent trends of agile development, DevOps, Web and Social Media sites somewhat question importance of load testing. Some (not many) openly saying that they don’t need load testing, some still paying lip service to it – but just never get to it. In more traditional corporate world we still see performance testing groups and important systems usually get load tested before deployment.

Let’s first define load testing as far as terminology is rather vague here. I use it here as anything that requires applying multi-user synthetic load – in contrast with single-user performance (which is a subset of performance engineering and may include, for example, profiling or Web Performance Optimization as it is defined now). And I use it here as an umbrella term including all other variations of multi-user testing, such as performance, concurrency, stress, endurance, longevity, scalability, etc. – but you may replace it with any other term if you prefer.

Yes, it looks like some Web and Social Media sites managed to survive without load testing. However, it looks like many such companies match the following profile:
-Business is built around a single Web site, so everybody in the company follows what is going on in production.
-Overall architecture is still clear and relatively simple. Changes (however frequent) are rather minor and evolutional.
-There is decent instrumentation providing performance information.
-There is a possibility to remove changes relatively easy.
-Site downtime/a period of slow performance (until the problem would be noticed and fixed) is not extremely painful or dangerous to the business.

Load testing is a way to mitigate load- and performance-related risks. There are other approaches and techniques that also alleviate some performance risks:
-Good single-user performance engineering practices (single-user requests performance are constantly tracked).
-Good instrumentation/Application Performance Management providing insights in what is going on inside the system.
-[Auto] scalable architecture.
-Continuous integration allowing quickly deploy and remove changes.

Still all of these don’t completely replace load testing, but rather complement it. They definitely decrease performance risk comparing with situation when nothing was done about performance at all until the last moment before rolling out the system in production without any instrumentation at all, but it still leaves risks of crashing and performance degradation under multi-user load. And if the cost of it is high, you should do load testing (what exactly and how is another large topic – there is much more here than the stereotypical waterfall-like last-moment record-and-replay approach).

There is always a risk of crashing or performance issues under heavy load – and the only way to mitigate it is actually test it. Even stellar performance in production and highly scalable architecture don’t guarantee that it won’t crash with a slightly higher load. Truly speaking, even load testing doesn’t completely guarantee it (real-life workload may be different from what you have tested), but it drastically decreases the risk.

Another important value of load testing is making sure that changes don’t degrade multi-user performance. Unfortunately, better single-user performance doesn’t guarantee better multi-user performance. In many cases it improves multi-user performance too, but definitely not always. And the more complex system, the more probable exotic multi-user performance issues no one even thought of. And a way to ensure that you don’t have such issues is load testing.

When you do performance optimization, you need a reproducible way to evaluate the impact of changes on multi-user performance. The impact on multi-user performance probably won’t be proportional to what you see with single-user performance (even if it still would be somewhat correlated). Without multi-user testing the actual effect is difficult to quantify. The same with the issues happening only in specific cases that are difficult to troubleshoot and verify in production – using load testing can significantly simplify the process.

Summarizing, I don’t see that the need in load testing is going away. Even in case of Web and Social Media sites we would probably see load testing coming back as soon as systems become more complex and performance issues start to hurt business. Maybe it would be less need for “performance testers” as it was at the heyday due to better instrumenting, APM tools, continuous integration, etc. – but I’d expect more need for performance experts that would be able to see the whole picture using all available tools and techniques (although I don’t see it yet).