A New Move in the Load Testing Tool Market
SOASTA launched CloudTest Lite – a free edition of their performance testing solution. Basically, they give it free for up to 100 users. A serious move for sure. It should heat up the load testing tool market. It may work indeed – I guess they don’t have many paid customers in that range anyway, looks like CloudTest’s sweet spot is when you need a very large number of users. I am very interested to see how it will turn out.
Several rosy reviews were posted, for example, CloudTest Lite – A Game Changer in the Performance Tool Market by Scott Barber and SOASTA CloudTest Lite Hands-On by Bernard Golden.
As I already mentioned, it indeed is pretty interesting. However, I’d say that we need add some skepticism to be more realistic.
First, it is not the first and absolutely unique move in load testing tools. I recall a few somewhat similar moves before which then quietly disappeared. Well, I don’t remember what limitations were (maybe a little bit more restrictive). And the companies were not the leaders of the market. Moreover, there is a list of 50 open source load testing tools on opensourcetesting.com and some, like JMeter and OpenSTA, are pretty mature. Yes, open source in load testing area was not so successful as in other areas. Especially analysis is weak in most of these tools (if existent at all).
Second, releasing is just the first step. The challenge for SOASTA would be how they support a large number of non-paying users (although, of course, for a promising start-up the number of customers may be important by itself). The community maybe can help with “how-to” questions, but implementing, let’s say, enhancement requests is up to the SOASTA team. And the number of such requests may be pretty high as people start to use it with different applications.
For example, it looks like we can’t specify transactions during recording in CloudTest for the moment. Well, what I am supposed to do with a script with a few hundred identical requests in it (AJAX type, differ by incomprehensible http body content)? Track delays in the scripts and try to correlate them with recording steps? Not exactly my understanding of quick and easy.
Scott writes in his review “it is free from now until the sun explodes” Hmm… I’d rather prefer to hear this from the SOASTA team. Well, even if the SOASTA team is completely devoted to this edition, nobody can guarantee that SOASTA won’t be acquired and who knows what acquirer decides to do with the freemium edition…
Yes, Scott is not easily getting excited. Last time, as far as I remember, Scott got excited about a load testing tool when Microsoft released their tool as part of Visual Studio back in 2005. See, for example, the discussions around my old posts VisualStudio 2005 and Load Testing and Scripting Language in Performance Tools. Well, Microsoft didn’t live up to its promises and I haven’t heard about their load testing tools for a while (my understanding is that it is not dead, but doesn’t play any noticeable role). But who knew that Microsoft was losing its grip?
By the way, returning to this old post about scripting languages, I don’t object the idea of GUI-based load testing tools (I mean when script is represented by a kind of graphical tree or something like this) if we can extend it with code (as in CloudTest with JavaScript) or switch between tree and script view (as in Oracle Application Testing Suite). My concern was (and is) if you can’t extend you recording with code at all – I still believe that it limits the area of application significantly.
Anyway, it looks like we have several interesting developments in the load testing tool market that may be beneficial to the community. CloudTest and its Lite version are definitely on the list. LoadRunner AJAX True client may be introducing a new paradigm in load testing (or promoting it if follow e-Valid blog). Oracle Application Testing Suite (former Empirix) is practically a new product and is getting traction [at least in the Oracle Universe].
Hi Alex,
I’m glad you had a chance to install and start playing with SOASTA CloudTest Lite. As you rightly mention, many of the most mature open-source products always lack some critical features. Lack of useful analytics is of course fairly common and makes these products weak candidates to help diagnostic performance issues, which is in the end one of the primary goal of performance testing.
Releasing an enterprise-class product such as SOASTA CLoudTest Lite doesn’t come without challenges and support might be seen as one of them. By releasing CloudTest Lite to the general public, we’re embracing this challenge as it will help us deliver the best product possible as close as possible to performance testing experts expectations. We’ll listen to constructive feedback and we’re looking at the software testing community to influence the future of our product. We truly think we’re bringing some new concepts and innovation into the aging world of performance testing products and we’re hoping our community will help us take this innovation to the next level.
Thanks for sharing your first opinion about SOASTA CloudTest Lite. I hope you’ll be able to write a follow-up post after a few months of practice. I’m looking forward to it!
Fred
PS1: You can of course create transactions in CloudTest Lite. It is one of many wrappers you can’t create around messages. You can create transaction around any messages in the clip editor: Right-Click->”Create a Transaction”. See the following screenshot http://cl.ly/8kfG .
PS2: What do you mean by “My concern was (and is) if you can’t extend you recording with code at all”? Are you looking at adding new messages, new scripts or new browser actions to your recording? If this is the case, that’s possible with SOASTA CloudTest Lite, again in the clip editor. There are plenty of tutorials and videos to help you with these features on our CloudLink community. You can also check-out our online help included with CloudTest Lite.
Hi Fred,
I mean to specify transactions during recording. I understand that I can specify transactions in CloudTest later in the clip editor, but it leaves me to figure out what requests I need to group for specific transactions – which is quite challenging and time-consuming if you have a large number of non-distinct requests.
Yes, I have realized that that we can add custom JavaScript code in CloudTest – and my statement was that it is an important feature and it removes the main part of my concerns expressed in my old post Scripting Language in Performance Tools.
Alex
Got it Alex! You’ll be happy to know that being able to specify transactions during recording is on CloudTest’s roadmap. Indeed, a very useful feature as you rightly mention.
It’s funny, I nearly qualified my “until the sun explodes” comment to include an acquisition exception, but I decided that it was irrelevant since pretty much every company with more than one employee (as well as many of them) are for sale at the right price… that’s just the nature of business.
Yup, bunches of opensource tools that many large companies will never allow in their firewall. I simply can’t begin to tell you how many organizations I’ve worked with who both refuse to spend enough money to purchase an adequate number of seats/licenses of a non-opensource tool and refuse to allow any opensource software on their network – and I don’t mean middle managers who’ve just been told “no downloading or installing anything”, I mean that I’ve stood in front of executive board meetings making the case for using opensource tools and been told, for example, “Not as long as I’m CEO.” These are the same companies that won’t allow and *nix operating systems or tools on machines that don’t have a box, CD, license key, and purchase/depreciation numbers in their accounting systems. The simple truth is that even if the opensource tools were far and away the best tools on the market, until executives of companies whose primary business is not producing software change their attitude toward opensource, none of those tools will have a chance to drive large market trends.
And finally, I’m still a big fan of the perf tool in MS’s VSTS bundle. I think the tool does live up to its promise, but the business model isn’t what many folks expected & therefore the impact it has had isn’t as easy to see. Many folks expected the tool to be marketed and sold as a stand-alone ‘all things to all people’ tool, but that was never the plan. MS was building the tool in the hopes of making it the undeniably correct (and easy) choice for .NET shops. What they made is a tool that’s fully integrated with the .NET dev environment and is specialized for .NET testing. I still *love* this architecture for a performance testing tool… unfortunately, it’s an architecture model that’s only relevant for organizations that are developing in “pure” environments. So, if your company isn’t a (mostly) pure .NET shop, the tool probably isn’t the best choice for you, but my advice to any org that is (or even claims to be a .NET shop) is that, you’re a fool if you don’t at least give VSTS a healthy look — *especially* if your company has an MSDN license… cause then you likely own the tool already.
In terms of impact, I think VSTS solidified a significant trend in the perf tool market — the major players consolidating onto specific technologies/platforms. The exception being HP/LoadRunner (Whose product roadmap continues to elude me — even when a friend of mine was the product manager and I was talking to him regularly… of course, he’s no longer product manager in part because the product roadmap eluded him too — which, I think, says all that needs to be said).
So, in the end, I think VSTS was as much of a game changer as I anticipated, only along a different path. As I look back, I really believe that the release of the VSTS tool was the tipping point that lead to Segue going to Borland then MicoFocus, Empirix going to Oracle, Radview going semi-open source then nowhere, IBM digging their tool even deeper into Eclipse/Java/IBM architectures, and even the Compuware/Gomez/Dynatrace consolidation. And I think that all that consolidation was what opened the door for JMeter’s growing popularity and created opportunities for SOASTA, NeoTys, StressTester, BrowserMob and others. Malcolm Gladwell may disagree about that being the tipping point, but I doubt it’s a trend that interests him.
CTLite’s impact, is already being felt. I’ve talked to 3 tool vendors since the release who have admitted to me that they are scrambling to come up with “a competitive offering”. Clearly, I’m not the only one who thinks CTLite is a game changer.
–
Scott Barber
System Performance Strategist & CTO, PerfTestPlus, Inc.
Co-Author, Performance Testing Guidance for Web Applications
http://www.perftestplus.com
sbarber@perftestplus.com
“If you can see it in your mind…
you will find it in your life.”
I’m excited for CTLite to attract even more people to focus on performance and improve their capabilities around performance testing and “building-in” performance thinking and action. Actually – I like game changers, when it leads to more attention to performance and development of highly performing applications. Also – I like game changers when they help people to succeed more than just a tool company to succeed.
And for the record, the HP/LoadRunner roadmap didn’t elude me as much as you might think – I think elude is not the right word. It was just very frustrating to get attention on that one product amongst such a huge portfolio of all products in the HP Software suite. I’m proud of what we added and innovated in LoadRunner v11.0 – it’s good stuff with some great feedback so far.
Cheers,
-mt
I might as well throw my two cents in here, so we can make this a conversation featuring the usual suspects. Where is James Pulley’s response?
I sat down with Scott Barber to discuss this topic earlier this year. The reason open source won’t take over the enterprise is simple, and two-fold. I will express it below while channeling a Fortune 500 CIO:
1. Don’t tell me you are an “enterprise class” load testing solution when you only support 20% of my enterprise technologies and protocols. The strength LR still has on the market has to do with it’s wide protocol support. Most large enterprises still have mainframes (green screen terminal emulators), old client/server apps deployed through Citrix, some weird Java RMI thing running in the corner that no one even knows about, in addition to all their web development. I don’t have time to build a utility belt like Batman with single use solutions for specific apps. I want “one ring to rule them all”. Build me something that can handle all of my enterprise needs (i.e. multiple protocols) and for 1/10th of the price of LR, and then you can slam HP’s price tag and we can talk. Obviously, smaller companies that may only do web development can get by with something that only does web. More power to them. That is not our reality.
2. Follow the money. I’m a CIO with a 300 million dollar Oracle ERP rollout to test that covers Siebel, Peoplesoft, Hyperion, and other applications that we haven’t stood up yet. Forgive me if I don’t take the risk on an open source tool created by a 17 year old in Angola somewhere. I want someone I can sue if they ruin the project, don’t provide support for the product, etc…. Hello HP.
Perhaps I am short-sighted, but this is what I see in the field and has been my experience for years. I’m happy to see other load testing tools out there, and I hope they solve needs for many companies. I wish them luck. I’d just like to see more innovation (more protocols being developed) before those tool makers start slamming the price of LoadRunner and making straw man arguments about how LR is some old, outdated product. It really makes them look very small. Build something great, and you won’t have to tear something else down. The world will beat a pathway to your door.
Now, let’s go get some BBQ….
Scott Moore
http://www.loadtester.com
Hey guys, it’s starting to feel like a party here!
@Mark Tomlinson , @Scott Moore
No offense meant by “elude”. And it wasn’t features that I was referring to… so maybe “product roadmap” was where I use the wrong term. Point is, it’s always seemed like LR was so focused on adding features (and clients) that it missed business opportunities that confound some of us (meaning me and the talking mouse I carry around with me to keep people from thinking I’m talking to myself all the time ).
LR is a good (very good, even) product. It’s the most diverse tool out there. No doubt about it. And if you’re a company with enough $ to get “enough” LR to do the load/performance testing you need to do, excellent! Absolutely no argument from me.
These days (and this hasn’t always been the case, so I’ll qualify with “since HP bought LR”) my gripe is that there is no “entry level”… there’s no way for a small, or struggling, or starting up company to implement life-cycle performance engineering (including testing) — especially on multiple projects using LR. So some orgs go with other tools (sometimes tools that are not the best tool for the job), and some orgs choose to skimp on perf activities or handcuff their perf testers, or build this complicated license & load gen check-in, check-out system where you have to schedule your test in outlook 2 weeks in advance (once again, I couldn’t make this up… household name insurance company).
So maybe “roadmap” and “elude” were poor word choices. I certainly didn’t intend a knock on your Mark, I like what you influenced in the LR product and the stuff you shared with me about where you were headed with things before you moved east. Your technical vision is/was above reproach.
The business vision, the packaging, the pricing, the refusal to negotiate, and the apparent lack of caring about helping companies without big enough wallets to develop and deploy well performing applications is what I’m referring to. And unless I’m remembering incorrectly Mark, you mentioned to me once that even as Product Manager, you had very little influence in these areas. (I guess making you the “technical product manager” not the “business product manager” — for purposes of clarifying for the 6 other people on the planet who are still reading).
And Scott, you are correct on both counts — good, bad, or otherwise, my experience agrees with yours. Neither opensource nor CTLite “resolve” those objections. But those aren’t the people I’m hoping that CTLite has the biggest influence on. I’m hoping that CTLite enables good performance practices in the small, struggling, and/or start-up companies that can’t afford to implement LR, but have the foresight to implement a solution that will grow with them. (Insert caveats about supported technologies/protocols here).
So if LR can find a way to bring a fully featured version of LR “to the masses” with a “no pain” upgrade path, I’ll say good things about that tool (particularly targeting all those orgs that have non-web and/or multiple protocols to support). Honestly, I *want* to see signs that the business side of the LR department at HP cared a little more about helping clients succeed and a little less about profit margin.
Of course, that’s probably because I am a far better activist than I am a capitalist.
–
Scott Barber
System Performance Strategist & CTO, PerfTestPlus, Inc.
Co-Author, Performance Testing Guidance for Web Applications
http://www.perftestplus.com
sbarber@perftestplus.com
“If you can see it in your mind…
you will find it in your life.”
I have a few comments. Mercury pioneered the way with record-replay, like two decades ago and was great when you had a waterfall model where it took like 6 months to build the app and so you could click-monkey your way through the test scripts for a week before you can run the load tests. With PaaS and the power shift to developers on the cloud (2 guys running a large number of instances on the cloud) and the broader trend of RESTful APIs on the cloud, the scripting and the load testing tools have to play well with continuous integration. Besides, load testing is not just for performance “experts”. We need to strive to bring this to the developers. Think Mint vs. Quicken. Yeah, you could have a thousand knobs to get it just right and just perfect, but if your user has to read a 300 page manual before s/he can run the load test, then it’s the 90′s all over again.
- @pcapr
I’ve been reading through the SOASTA website content, and I suspect there are a few other limitations that we aren’t clearly called out. It appears there are distinctions between CloudTest versions other than simply the VU count: I think I’m also seeing distinctions (limitations) about where the tests (VU load) or target systems run from: http://www.soasta.com/cloudtest/cloudtest-editions/
Sifting through the content, I think I’m seeing – or perhaps more accurately inferring – a distinction between running internally (Intranet) vs externally; running VU’s from different locations; running VU’s from different geographic IP ranges; running internally and externally at the same time; interacting with mobile devices, etc. It’s not clear to me whether these distinctions relate to VU load generation or the system(s) being targeted for the test (or both).
So, if “location” is a factor in the license scaling, what limitations in practical terms does the free version have?
PS – @pcapr (kowsik) I think it’s a bit of an stretch claiming that Mercury pioneered the way with record-playback. There were various record-playback testing system available on the market well before Mercury were a significant player. However, I think it’s fair to say that they were part of the early wave of companies seriously commercialising record/ playback. I think of them more like a Microsoft: they’re the companies that take the “cool tech” of the early innovators and make a commercial success of it.
Alex,
Thanks for your comments and cynicism. I begin to have serious doubts when all I hear is praise about something. But in the midst of all these well-known QA type am I going to have point out an grammar error (sorry these really annoy me). in the 4th paragraph you say “Especially analysis is week in most of these tools (if existent at all). ” week is a measure of time. Granted Loadrunner analysis seems like it takes a week sometimes but weak is the the word you wanted.
I also like to comment on the comment “there is a list of 50 open source load testing tools on opensourcetesting.com” True, they are listed but how many of those listings have you actually looked at or investigated? I’ve looked at a large number of the 50 listed and run into many deal links, abandoned projects or projects that don’t have anything more then a sourceforge page. As for analysis of test results, very few have anything more then xUnit type reporting. Even stuff like the PushToTest Testmaker which has some basic analysis features is so buggy its useless.
Since everything I do must be inside the corp firewall, Public Cloud testing is, imho, a lot of hype. Cloudtest Lite is something that I can use. From what I’ve seen its pretty freaking impressive. Actual documentation that MATCHES the product. that in itself is impressive. What I’ve seen is equally impressive. But as you alude to, it remains to be seen if it can really handle multi-tier web 2.0 web apps. I hope it does change the market for performance testing tools and save me from TruClient. lol
Thank you for spotting the typo! Fixed.
I am saying that the idea is not new, not saying that it was very successful in the past. It looks like load testing tools is not a very exciting area for open source developers. Still there are a few decent products that you may use in some cases: JMeter, Grinder, OpenSTA. I am working with rather tricky products (from load testing point of view) and had a lot of issues with every tool I tried – but don’t see why you can’t use them in simpler cases.
It is good for the performance testing community that we have more options now (considering that nothing was happening for a while). And Cloudtest Lite is definitely an interesting option – but still keep you eyes opened and see how it works for you.
Web 2.0 apps is an interesting separate topic – unfortunately we can’t guarantee that you can always use protocol-level recording/playback approach. In some cases we may need other approaches like using GUI-based recording (like TruClient) or use APIs.
Great Article
will love to share information with your blog, visit me at http://www.qatestingtools.com