Do we have a revolution in load test tools?

It looks like the competition on the load testing tools market heats up again. And, of course, new players need to differentiate themselves. I started to see statements that zero-scripting tools is a new word here. See, for example, the Agile Thinking: A New Approach to Performance Testing paper by Graham Parsons.

While I completely agree with the first part (compare with my Agile Performance Testing article), I am rather confused with the second part. I don’t see why we can’t use well-established tools like LoadRunner or JMeter (if we speak about free tools) in agile development. When you know the tool, scripting is usually not the most difficult part of your performance project.

The zero-scripting approach (if I understand it correctly) looks for me rather limited to simple web sites (and if you have such sites – you don’t need to do much in LoadRunner either). How do you handle complex correlation and parametrization often required by today’s rich web clients? And as you start to introduce ways to do it through, for example, graphical interface – you start to create a proprietary way to do this.

Also there is very interesting Fred Beringer’s post. I completely agree the main idea of the post. Performance engineering is much more than just a gate-keeping function. The terminology is somewhat vague here, but I usually use ‘performance engineering’ when I want to stress that it is more than “gate-keeping”. ‘Wisdom’ sounds a little bit too pathetic to me and the definitions of knowledge and wisdom may be argued about, but the idea is well formulated and definitely valid.

But I still believe that you may be in the ‘wisdom’ business with whatever tools are available to you. I, of course, don’t know all the features of CloudTest (the materials I read were too marketing-oriented to figure out what are real advantages). Tools help you a lot, but still it is people who are in knowledge / wisdom business, not tools. Even the best tool won’t do the work for you – you need to know how to use it, how to interpret it, etc.

And I still don’t see that drastic difference with “traditional” load testing tools. I guess CloudTest has some advantages. Listening to a webinar I realized one – they can quickly and automatically deploy agents across as many cloud servers as needed. Well, this is a significant advantage when you work in the cloud or want to run a really large-scale test (and an interesting variable here is how many users you can simulate from one server). Probably there are other advantages. But the basic principles still look the same.

Going to how Fred describes the difference:

1. Scale. Maybe CloudTest has some advantages on the high-end. It usually is a problem when you want to simulate many thousands users. But why “traditional tools limit themselves to testing behind a firewall”? Nothing, up to my knowledge, prevents you from installing them outside firewall. Yes, most performance testing in large corporations is done in a lab inside firewall – but, I guess, it is because most performance testing they do is for internal applications. And nothing actually prevents you from generating traffic from different geographies. I often did it myself by installing LoadRunner agents in remote locations.

I am not sure that I got Fred’s point in the statement: Most traditional tools, including Loadrunner, use manual dynamic session or hard-coded rules. You end up with significant repetitive work and long support delays. . At least what I saw about templates during a CloudTest webinar, looks pretty similar to what is used in “traditional” tools.

2. Speed. Why “proprietary scripting language”? LoadRunner uses ANSI C for Web protocol (and other standard languages for other protocols), Oracle Application Testing Suite uses Java, etc. Sure, they use a lot of proprietary functions there, but languages are standards and you can add your code or use external functions when you need to extend script functionality. Yes, it requires some skills – but this is required only in really complicated cases. In simple cases, you don’t care much about what language is used in the script – just record it and play it back. Anyway, looks like SOASTA uses JavaScript as a scripting language.

3. Affordability. Well, LoadRunner is not the cheapest product on the market for sure. But whatever licensing model is used by CloudTest, it is probably still a noticeable sum. While we have multiple free products around. Some, like JMeter, are mature enough.

To summarize, it quite could be that new tools have some advantages over old players. Still I don’t see a revolution yet. Some progress for sure (especially after a long period of stagnation in this market), but not a revolution. Well, maybe I just not good in marketing – but I wish to see some webinars / papers for technical audience explaining real difference and advantages.