Thoughts about Load Testing Tools Market
An interesting article Performance Testing Market Under Attack by Startups was posted in the WSJ blog. The author clearly don’t know much about the performance testing market – and only meaningful fact that I see there is that SOASTA is a challenger in this market. I have no idea how other startups mentioned relate to the performance testing market (some may relate to other performance-related markets, but that’s it – and, of course, all of them hot). Well, this article is clearly poor on facts, but it looks like the title represents the current market perception. So let’s see closer what is going on.
There were always a lot of new load testing products on the market (considering rather limited size of the market). The reason is, I believe, that entry costs are very low- you may create a very simple program and it still would be useful for many users. The problem is that further advancements require a lot of efforts and investments – and most companies in the best case would feed their founders.
There are always predictions that newcomers will push incumbents. But we haven’t seen it much before – I guess mainly due to long and hard efforts needed. The last wave was, I believe, in the beginning of 00s. There was a lot of noise when Microsoft was entering performance testing market, many predicted that Microsoft will crash the market – see, for example, the VisualStudio 2005 and Load Testing discussion. They, of course, got their own niche – but not really fighting for the market (and never did). I can’t even find a good link to their load testing capabilities to add to my list of most popular performance testing tools. OpenSTA, one of the most promising then open source tools then, looks dead. Many other then promising vendors – while still around – are not considered market disruptors anymore if put it mildly.
And now we see a new wave. And while we see a lot of new tools – not many even pretend to be a market disruptor. Truly speaking, it is not surprising that the author of the abovementioned WSJ blog had problems mentioning promising load testing statups beyond SOASTA. A few companies succeeded in a very lucrative niche – large-scale Internet tests. SOASTA is probably the most notable example. Perhaps BlazeMeter is worth mentioning too: JMeter-based load testing service sounds as a promising approach considering that JMeter is clearly the most used open source product. Their success, in particular, may be explained by two major LoadRunner flaws: astronomic cost of license for a large number of users (so even companies having LoadRunner are looking for alternative solution to test their public portals) and a lack of ways to quickly deploy a larger number of load generators. However these are not inherent LoadRunner problems – and HP may eventually address them. So even the current beachhead is not guaranteed – and its extending will require a lot of efforts and investments.
Still it looks like LoadRunner, in spite of all its shortcomings – high prices, very restrictive licenses, bad support, and outdated architectural components – still have the majority of corporate market. The main challenge of all LoadRunner contenders in the corporate market is the limited number of supported protocols (usually limited by HTTP-related). Practically only Borland SilkPerformer tries to compete with LoadRunner in supporting multiple protocols. While indeed the number of application using non-Web protocols is diminishing and efforts to support them are very high – so decision not to go this way for startups with limited resources makes perfect sense – the contenders still should come up with an approach to fill this gap to get to lucrative corporate market.
One approach is recording/playback on GUI level, which makes used protocols completely irrelevant. While some load testing tools had some indirect ways to support it (as using QTP scripts in LoadRunner), it wasn’t very popular due to scalability limitations. Appearance of new contenders who add some scalability to the approach by allowing quick deployment of load generators will change the balance. And the most notable example here is probably NRGGlobal AppLoader. AppLoader records on bitmap level, so it is completely independent from the underlying technology, and it measures real-user, end-to-end timing. AppLoader’s approach, adding scalability by simplifying deployment of multiple load generators, may be a meaningful alternative to supporting numerous protocols.
