Master Apache JMeter. From Load Testing to DevOps
I was honored to write a preface to the great new book Master Apache JMeter. From load testing to DevOps. Here it is:
An important event, directly related to this book, happened recently and probably went unnoticed. It appears that Apache JMeter became the most popular load testing tool. In 2014, I was preparing a presentation about load testing tools and criteria for their selection. One criterion was the existence of ecosystem (documents, expertise, people, services, etc). It may be not the defining factor, but it is an important factor to consider. To evaluate such ecosystems, in absence of more sophisticated data, I used the number of documents Google finds and the number of jobs Monster finds mentioning each product.
LoadRunner (then an HP product) clearly had first place in both categories with JMeter following not too far. SilkPerformer (then Borland) trailed third far behind. But now, in 2018, JMeter appears to be well ahead of LoadRunner (now MicroFocus) in both the number of documents and the number of jobs mentioning it, apparently becoming the most popular load testing tool. Of course, it doesn’t mean that JMeter became the best tool for every task, but its popularity, in addition to being an open source tool, definitely gets it high in the list of options to consider. It is also very important for an open source project to attract people who will work to improve it, thus ensuring the future of the product.
Another interesting trend is that JMeter scripts become a de facto standard, and many SaaS tools are built on the top of JMeter or at least support JMeter scripts. These tools complement JMeter in many important ways and bring its functionality and services to a new level, allowing to compete with commercial products in more sophisticated environments.
Several books about JMeter were published before, but this one is the first one from JMeter contributors who know its internals and are renowned experts in the area. JMeter is not a trivial product. It has a lot functionality, but it may be implemented by many different ways and it is not always easy to figure out the best way to do it. While, as it was already mentioned, there is an enormous number of posts on the Internet discussing some aspects of JMeter, the problem is that most of them are for beginners. And while there is more advanced stuff too, you better exactly know what you are looking for, it may be hard to find it looking for generic terms. This is where the book is invaluable setting a framework of knowledge and notions, so you at least would understand what you should be looking for.
The book, with all that valuable information in one place, is a must for everybody who is seriously working with JMeter. It may be a little too condensed for absolute beginners (although, as already mentioned, there is a lot of introductory materials on the Internet that can help here), but it is the best you can find if you already have some performance testing experience and need to get further. In particular, it would be invaluable to people who want to expand their JMeter knowledge into advanced topics or switch to JMeter from other load testing tools.
In particular, JMeter has many integrations with other popular tools and a large number of plugins. It quite could be that somebody already solved the problem you are confronting, so you may save a lot of efforts re-using the solution. It is great that the book is not limited by the core JMeter functionality and discusses available components and integrations when appropriate. While it is impossible to cover everything that is available in detail, the advantage of the book is that it puts everything into a system, allowing the reader to understand relations between different parts and technologies.
Load testing is an important part of the performance engineering process. However, the industry is changing and load testing should adjust to these changes. A stereotypical, last-moment performance check is not enough anymore. Performance testing should be interwoven into development process, starting early and continuing through the whole lifecycle. The importance of the transformation is stressed in the subtitle of the book, From load testing to DevOps, moving from standalone load testing, a mere step at the end of software development cycle, to performance testing fully integrated into DevOps. And while we are not fully there yet, the Integration of JMeter in the DevOps tool chain chapter directly dives into what can be done right now.
DevOps, putting together the development and operations sides, is supposed to drastically improve feedback from production to development and free flow of performance information both ways. So a holistic approach to performance should be one of its main advantages. Unfortunately, it doesn’t look like such a holistic approach happens often. Rather it looks like "DevOps" teams just drop more sophisticated parts of performance engineering (and performance testing usually gets into that category) and rely more on reactive approach to performance issues, concentrating more on quick fixing of issues than on their prevention. Still, load testing is a very important way of risk mitigation and it can’t be fully replaced by other performance engineering activities.
First, there are always risks of crashing a system or experiencing performance issues under heavy load, the only way to mitigate them is to actually test the system. Even stellar performance in production and a highly scalable architecture don’t guarantee that it won’t crash under a slightly higher load.
It is important to notice load testing doesn’t completely guarantee that the system won’t crash: for example, if the real-life workload is different from what was tested. So you need to monitor the production system to verify that your test workload is close enough. But load tests considerably reduce the risk if they are carried out correctly (and, of course, can be completely useless and misleading if they are not).
Another important benefit of load testing is to verify how changes affect multi-user performance. The impact on multi-user performance is generally not proportional to what is observed with single-user performance and can often be counter-intuitive. Sometimes, improving single-user performance can lead to a degradation of multi-user performance. The more complex the system, the more exotic multi-user performance problems can occur.
Another value of load testing is to provide a reliable and reproducible way to apply a multi-user load necessary for performance optimization and troubleshooting. You apply exactly the same workload and see if the change makes a difference. In most cases, you cannot do this in production when the workload changes – so you never know if the result is a change in code or a change in workload (except, perhaps, a rather rare case of very homogeneous and manageable workloads when you can apply a very precisely measured portion of the actual workload). And, of course, a reproducible workload greatly simplifies debugging and checking multi-user problems.
In addition, given current trends in system self-regulation (such as auto-scaling or load-dependent service level changes), load tests are required to verify this functionality. You must apply a heavy load to see how auto-scaling will work. Load testing thus becomes a means of testing the functionality of the system, blurring the traditional division between functional and non-functional testing.
You will find further examples of different types of performance tests and their links to different aspects of DevOps in the book. Although the book does not focus on the theoretical aspects of performance testing, it provides sufficient theoretical information to understand the concepts discussed and their practical applications in JMeter. Practical examples of integrating performance testing with DevOps are all the more important as this is a rather new field of expertise and is probably the main challenge of performance testing at the moment. Thus, the book leads the reader through the basics of working with today’s most popular load testing tool JMeter, through more advanced aspects of the tool and performance testing in general until its complete integration into DevOps.
The authors decided to publish the book on LeanPub, in order to keep it up to date and enrich it. There have already been some updates and others are in progress. Although I would like to have a printed copy, I must admit that it is probably a better choice for readers, as many improvements are made to JMeter with each version, not to mention the growing ecosystem of plug-ins, extensions, tools and products. Any paper version would be outdated soon enough – but with LeanPub, authors have the opportunity to keep the book up to date, which is a great service to the community.
