A New Move in the Load Testing Tool Market

SOASTA launched CloudTest Lite – a free edition of their performance testing solution. Basically, they give it free for up to 100 users. A serious move for sure. It should heat up the load testing tool market. It may work indeed – I guess they don’t have many paid customers in that range anyway, looks like CloudTest’s sweet spot is when you need a very large number of users. I am very interested to see how it will turn out.

Several rosy reviews were posted, for example, CloudTest Lite – A Game Changer in the Performance Tool Market by Scott Barber and SOASTA CloudTest Lite Hands-On by Bernard Golden.

As I already mentioned, it indeed is pretty interesting. However, I’d say that we need add some skepticism to be more realistic.

First, it is not the first and absolutely unique move in load testing tools. I recall a few somewhat similar moves before which then quietly disappeared. Well, I don’t remember what limitations were (maybe a little bit more restrictive). And the companies were not the leaders of the market. Moreover, there is a list of 50 open source load testing tools on opensourcetesting.com and some, like JMeter and OpenSTA, are pretty mature. Yes, open source in load testing area was not so successful as in other areas. Especially analysis is weak in most of these tools (if existent at all).

Second, releasing is just the first step. The challenge for SOASTA would be how they support a large number of non-paying users (although, of course, for a promising start-up the number of customers may be important by itself). The community maybe can help with “how-to” questions, but implementing, let’s say, enhancement requests is up to the SOASTA team. And the number of such requests may be pretty high as people start to use it with different applications.

For example, it looks like we can’t specify transactions during recording in CloudTest for the moment. Well, what I am supposed to do with a script with a few hundred identical requests in it (AJAX type, differ by incomprehensible http body content)? Track delays in the scripts and try to correlate them with recording steps? Not exactly my understanding of quick and easy.

Scott writes in his review “it is free from now until the sun explodes” Hmm… I’d rather prefer to hear this from the SOASTA team. Well, even if the SOASTA team is completely devoted to this edition, nobody can guarantee that SOASTA won’t be acquired and who knows what acquirer decides to do with the freemium edition…

Yes, Scott is not easily getting excited. Last time, as far as I remember, Scott got excited about a load testing tool when Microsoft released their tool as part of Visual Studio back in 2005. See, for example, the discussions around my old posts VisualStudio 2005 and Load Testing and Scripting Language in Performance Tools. Well, Microsoft didn’t live up to its promises and I haven’t heard about their load testing tools for a while (my understanding is that it is not dead, but doesn’t play any noticeable role). But who knew that Microsoft was losing its grip?

By the way, returning to this old post about scripting languages, I don’t object the idea of GUI-based load testing tools (I mean when script is represented by a kind of graphical tree or something like this) if we can extend it with code (as in CloudTest with JavaScript) or switch between tree and script view (as in Oracle Application Testing Suite). My concern was (and is) if you can’t extend you recording with code at all – I still believe that it limits the area of application significantly.

Anyway, it looks like we have several interesting developments in the load testing tool market that may be beneficial to the community. CloudTest and its Lite version are definitely on the list. LoadRunner AJAX True client may be introducing a new paradigm in load testing (or promoting it if follow e-Valid blog). Oracle Application Testing Suite (former Empirix) is practically a new product and is getting traction [at least in the Oracle Universe].