The Future of Performance Testing
Following up my post Are Times still Good for Load Testing? , I decided to answer multiple comments here separately.
First, I’d like to elaborate on “It may be less need for simple load testing due to increased scale and sophistication of systems”. I meant that the traditional way – testing the system before deploying in production using production-type workload – is not the only way anymore. While there are still quite a lot of cases where it is still applicable, it needs to evolve into more sophisticated processes tightly integrated with development and other parts of performance engineering. First of all, integrating into agile development (shift-left / continuous performance testing) and integrating into performance information loop with production (shift-right) to form a holistic performance view. It doesn’t mean that we don’t need traditional load testing anymore – it means that we need to build up on the top of it to bring more value to the table.
That dictates changes needed in performance testing (and, eventually, performance testing tools). Integrating with agile developments means that performance testing should be 1) continuous and 2) on different levels – often for separate components / parts of the system in different stages of readiness. In addition to obvious close integration with other Continuous Integration (CI) / DevOps tools, it means better ways to create/maintain load and better ways to analyze information.
Yes, the tools and process should be easier for non-experts to incorporate some performance testing into continuous development process. But it doesn’t mean that we need tools with minimal functionality. Of course, more sophisticated tools may definitely do better job in creating a simple and intuitive interface (still providing a way to invoke more sophisticated functionality when needed).
Those wondering what I am talking about may, for example, check my earlier posts The Role of Recording in Load Testing Tools and About the Role of Recording in Performance Testing discussing one part of needed functionality.
One of main challenges is definitely scripting and scripts fragility. Yes, there is a trend toward using API (and REST API in particular) that makes it somewhat easier to create and maintain scrips. But there is an opposite trend to use several layers of different frameworks and libraries which nobody fully understand and what is actually sent to the server remains unknown unless you use recording (or reconstruct it from a detailed log).
Make scripting easier (and maintain the existing scripts) is utmost importance here. It appears that there are significant efforts in that direction, often citing AI – however it is all rather for simple cases [yet] and using “AI” is often rather a marketing stretch. Mark Tomlinson’s webinar Solving Performance Problems with AI may be a good introduction into the topic.
An interesting direction is using GUI-level scripting that allows alleviate some issues of protocol-level scripting. Latest move into that area was SmartBear LoadNinja. But the approach, while getting more interest, still rather far from being the mainstream for serious systems and still has the same problem of script fragility.
Another major challenge is that continuous performance testing as part of the agile development lifecycle can’t be the traditional full-scale, realistic load/data/configuration load test in most practical cases (but it could – and should – be complemented by larger-scale tests; see, for example, my Context-Driven Performance Testing presentation).
As continuous performance testing should fit CI process, it may be limited in scope by involved functionality / components, data, configuration, and load. The point is not to test system to its limits, the point is to see if any change in performance is happening. It is basically regression performance testing [and that is why it should be complemented by other kinds of performance testing]. So here we need ways to find a good combination of SUTs/load/data/configuration – to be meaningful and representative in a way while fitting CI – which is rather a non-trivial task. Data preparation and service virtualization functionality/tools would be here very handy here.
With all its limitations, continuous performance testing provides a constant stream of performance information – which may be a great input to understand overall impact on system’s performance. And to make these projections modeling becomes extremely important here. Performance modeling significantly increases the value of performance testing. First, it is one more way to validate tests correctness and help to identify problems with the system. Second, it allows answering questions about sizing and capacity of the system (including forecasting and what-if analysis). Performance modeling is needed today more than ever – as we embrace continuous performance testing and full-scale tests are not always feasible – to make a bridge to a holistic view of performance. How the change in that service performance will impact overall performance of the system? It is rather paradoxical that once popular topic of performance modeling became practically forgotten – it is time to revive it!
Henrik Rexed provoked me to talk on that subject at the upcoming Performance Advisory Council (PAC) – so I have no way to further avoid formalizing my views on that rather sophisticated subject. Basically, it may be boiled down to two aphorisms “all models are wrong, but some are useful” (attributed to George Box) and they “should be made as simple as possible, but not simpler” (attributed to Albert Einstein) – you don’t need sophisticated math anymore to create a useful performance models (which doesn’t mean that it became trivial – far from that – but it is more due to sophistication of today’s systems than need in complex math).
One somewhat related thing that was mentioned in comments is moving to the cloud. And cloud vendors have complex relations with performance testing – up to prohibiting it. It definitely changes the performance engineering strategy and there are many questions to be sorted out eventually. However cloud practically assigns a price tag to application performance – so any performance improvement results in cost saving (in addition to all other reasons for good performance). Probably it should lead to some shift toward performance optimization instead of performance verification. And, of course, need for performance / cost modeling – to see how any change (for example, on a component level) will impact bottom line.
As a part of shift-right, performance and reliability testing in production got significant popularity recently – under more sexy “chaos engineering” name. But I believe all that functionality that can be found in Gremlin and multiple simpler open source products should be in every performance testing tool (at least those that pretend to some universality). And some functionality like network virtualization can be found in more sophisticated products – but it is time to have all of them (or at least have a good integration with “chaos engineering” tools).
And all of that on the top of more traditional needs – such as close integration with monitoring/APM/etc tools and advanced results analysis. However, with all that additional functionality, the tools somehow should be easy to use and lightweight. As stated in a comment by Farid Sharipov: “Apart from being free open source tool, JMeter is pretty light compared to commercial tools. Trend is going more into continuous performance testing where tests run automatically and analysis for basic stuff also automatic. I cannot imagine myself setting up such automation with LoadRunner. With JMeter we can easily setup new machine and install and run tests and terminate the machine.” On the first read I even didn’t quite get the point and wanted to say that you can do all that with LoadRunner (as a lot of CI-related functionality is there for a while). But then I recalled that installing LoadRunner was a real process – sometimes taking a half day, with a lot of choices to make. I am sure you can do it in an automatic way with some proprietary scripting – but it is indeed difficult to compare with JMeter’s setup which is just copying and extracting a zip file.
So we have here a challenge: from one side, we need more sophisticated functionality; from another side, we need tools to be simple to use and lightweight. The current trend in open source tools appears to be that they provide basic functionality and everybody who is interested in other things will figure out some plumbing with the tools providing such functionality. The problem is that it is far from trivial as other tools, even if existing, may use other paradigms and do not fit together well. Not mentioning that performance engineers usually need to solve performance problems and plumbing work may be not the best use of their time (although it appears that this notion gets just dismissed nowadays). Well, maybe modularity is indeed an answer – but we need then really simple and reliable integrations between modules and we definitely is not there yet…
Finally, in addition to the standard disclaimer that all these above are my personal thoughts only, I should admit that I have rather a bad record of business predictions – usually simplistic and limited technologies succeeded on the market over more deep technologies that appeared to be much more promising to me… Anyway, looking forward to see what the future holds to performance testing.
