Alex Podelko

Performance testing is an interesting discipline: while you may see a lot of people doing it, not much is written about it. And what is written is often either introduction-level or tool-specific (other large groups of topics including performance analysis and performance troubleshooting are wider topics and not testing-specific). But not much about performance testing specific challenges beyond of what is covered by tool manuals.

My group specializes in performance testing of Hyperion / Enterprise Performance Management and Business Intelligence products at Oracle. A few of scripting challenges exist for almost every product. Nothing exceptional – you should resolve them if you have some experience and would be attentive enough, but time after time we are called to save performance testing projects to find out that there are serious problems with scripts and scenarios which make test results meaningless. Even very experienced testers stumble, but problems could be avoided if more time was spent analyzing what is going on.

Let’s consider an example – probably typical for challenges you can face with modern Web-based applications. Some operations, like financial consolidation, can take a long time. The client starts the operation on the server and then waits until it will finish, a progress bar is shown in meanwhile. When recorded, the script looks like (in LoadRunner pseudo-code):

web_custom_request(“XMLDataGrid.asp_7″,
“URL={URL}/Data/XMLDataGrid.asp?Action=EXECUTE&TaskID=1024&RowStart=1&ColStart=2&RowEnd=1&ColEnd=2&SelType=0&Format=JavaScript”,
LAST);

web_custom_request(“XMLDataGrid.asp_8″,
“URL={URL}/Data/XMLDataGrid.asp?Action=GETCONSOLSTATUS”,
LAST);

web_custom_request(“XMLDataGrid.asp_9″,
“URL={URL}/Data/XMLDataGrid.asp?Action=GETCONSOLSTATUS”,
LAST);

web_custom_request(“XMLDataGrid.asp_10″,
“URL={URL}/Data/XMLDataGrid.asp?Action=GETCONSOLSTATUS”,
LAST);

What each request is doing is defined by the ?Action= part. The number of GETCONSOLSTATUS requests recorded depends on the processing time. In the example above it was recorded three times; it means that the consolidation was done by the moment the third GETCONSOLSTATUS request was sent to the server. If playback this script, it will work in the following way: the script submits the consolidation in the EXECUTE request and then calls GETCONSOLSTATUS three times. If we have a timer around these requests, the response time will be almost instantaneous. While in reality the consolidation may take many minutes or even an hour (yes, this is a good example when sometimes people may be happy having one hour response time in a Web application). If we have several iterations in the script, we will submit several consolidations, which continue to work in background competing for the same data, while we report sub-second response times.

Consolidation scripts require creating an explicit loop around GETCONSOLSTATUS to catch the end of consolidation:

web_custom_request(“XMLDataGrid.asp_7″,
“URL={URL}/Data/XMLDataGrid.asp?Action=EXECUTE&TaskID=1024&RowStart=1&ColStart=2&RowEnd=1&ColEnd=2&SelType=0&Format=JavaScript”,
LAST);

do {

sleep(3000);

web_reg_find(“Text=1″,”SaveCount=abc_count”,LAST);

web_custom_request(“XMLDataGrid.asp_8″,
“URL={URL}/Data/XMLDataGrid.asp?Action=GETCONSOLSTATUS”,
LAST);

} while (strcmp(lr_eval_string(“{abc_count}”),”1″)==0);

Here the loop simulates the internal logic of the system sending GETCONSOLSTATUS requests each 3 sec until the consolidation is done. Without such loop the script just send requests and ends the iteration while the consolidation runs for a long time after that.

Some other examples can be found in my presentation A Load Testing Story at CMG’10 – a story about one my project which had a good share of testing challenges.

Many technical details about testing Hyperion products with LoadRunner are included in Oracle support document 1339743.1. A lot of this knowledge is now included in the Hyperion module of Oracle Application Testing Suite. But that is just about one family of products – which is a small sample of challenges you would see around.

According to my experience, creating a meaningful and realistic workload is the main performance testing challenge – and you probably face it in most non-trivial systems (including most of corporate applications). It is surprising that very few people talk about it: reading about performance testing you may get impression that the scripting is trivial and mundane (and it looks like many “performance testers” truly believe in it – and when they face any challenge, they start to talk about the death of performance testing).

Only few names come to my mind who write about specific challeneges. For example, see Northway (@northwaysg) / Scott Moore (@loadtester) performance testing blog and white papers or Mentora (@MentoraGroup ) / Dan Downing presentations and white papers. Maybe few more – but definitely they are few and far apart.

Classifying and evaluating load testing tools is not easy as they include different sets of functionality often crossing borders of whatever criteria are used. In most cases, any classification is either an oversimplification (which in some cases still may be useful) or a marketing trick to highlight advantages of specific tools. There are many criteria allowing to differentiate load testing tools and it is probably better to evaluate tools on each criterion separately.

First, there are three main approaches to workload generation and every tool may be evaluated on which of them it supports and how exactly.

Protocol-level recording and the list of supported protocols. Does the tool support protocol-level recording and, if it does, what protocols it supports. With quick Internet growth and popularity of browser-based clients, most products support HTTP only or a few Web-related protocols. According to my knowledge, only HP LoadRunner and Microfocus SilkPerformer try to keep up with support of all popular protocols. So if you need recording of a special protocol, you probably end up into looking at these two tools (unless you find a special niche tool supporting your specific protocol). That somewhat explains the popularity of LoadRunner at large corporations where you probably have almost all possible protocols used. The level of support of specific protocols differs significantly too. Some HTTP-based protocols are extremely difficult to correlate if there is no built-in support, so you may look for that kind of specific support. For example, Oracle Application Testing Suite may have better support of Oracle technologies.

UI-level recording. The option was available for a long time, but it is much more viable now. For example, there was a possibility to use Mercury/HP WinRunner or QuickTest Professional (QTP) scripts in load tests, but you needed a separate machine for each virtual user (or at least a separate terminal session). That limited the level of load you may achieve drastically. Other known options were, for example, Citrix and RDP (Remote Desktop Protocol) protocols in LoadRunner – which always were the last resort when nothing else was working, but were notoriously tricky to playback. New UI-level tools for browsers, such as Selenium, extended possibilities of the UI-level approach allowing to run multiple browser per machine (so scalability is limited by resources available to run browsers). Moreover, we got UI-less browsers, such as HtmlUnit, which require significantly less resources than real browsers. There are multiple tools supporting this approach now – such as PushToTest directly harnessing Selenium and HtmlUnit for load testing or LoadRunner TruClient protocol and SOASTA CloudTest using more proprietary solutions to achieve low-overhead playback. Still questions of supported technologies, scalability, and timing accuracy remain largely undocumented, so the approach requires evaluation in every specific non-trivial case.

Programming. There are cases when you can’t (or can, but it is more difficult) use recording at all. In such cases using API calls from the script may be an option. Other variations of this approach are web services scripting and using of unit testing scripts for load testing. And, of course, you may need to add some logic to your recorded script. You program the script using whatever way you have and use the tool to execute scripts, coordinate their executions, report and analyze results. To do this, the tool should have ability to add code to (or invoke code from) your script. And, of course, if tool’s language is different from the language of your API, you would need to figure out a way to plumb them. Tools, using standard languages such as C (e.g. LoadRunner) or Java (e.g. Oracle Application Testing Suite) may have an advantage here. However you should know all details of the communication between client and server that is often very challenging.

Other important criteria are related to the environment:

Deployment Model. There were a lot of discussions about different deployment models: lab vs. cloud vs. service. There are some advantages and disadvantage of each model. Depending on your goals and systems to test you may prefer one deployment model over another. But I still believe that for comprehensive performance testing you really need both lab testing (with reproducible results for performance optimization) and realistic outside testing from around the globe (to check real-life issues that you can’t simulate in the lab). Doing both would be expensive and makes sense when you really care about performance and have a global system – but it not rare and if you are not there yet, you can get there eventually. If there are such chances, it would be better to have a tool which supports different deployment models.

If it is lab or cloud, an important sub-question would be what kind of software / hardware / cloud the tool requires. Many tools use low-level system functionality, so is may be unpleasant surprises when the platform of your choice or your corporate browser standard is not supported.

Scaling. When you have a few users to simulate, it usually is not a problem. The more users you need to simulate, the more important it becomes. The tools differ drastically on how many resources they need per simulated user and how well they may handle large volumes of information. It may differ significantly even for specific tool depending on protocol used and specifics of your script. As soon as you get to thousands of users, it may become a major problem. For a very large number of users some automation, like automatic creation of a specified number of load generators across several clouds in SOASTA CloudTest, may be very handy.

Two other important sets of functionality are monitoring of the environment and result analysis. While theoretically it is possible to do it using other tools, it significantly degrades productivity and may require building some plumbing infrastructure. So while these two areas may look optional, integrated and powerful monitoring and result analysis are very important. And the more complex system and tests, the more important they are.

Of course, non-technical criteria are important too:

Cost. There are commercial tools (and license costs differ drastically) and free tools. And there are some choices in between: for example SOASTA has the CouldTest Light edition free up to 100 users. There are many free tools (some, as JMeter, are mature enough and well-known) and many inexpensive tools, but most of them are very limited in functionality.

Skills. Considering a large number of tools and a relatively small number of people working in the area, there is a kind of labor market only for the most popular tools. Even for the second-tier tools there are few people around and few positions available. So if you don’t choose the market leaders, you can’t count that you find people with this tool experience. Of course, an experienced performance engineer will learn any tool – but it may take some time until productivity will get to the expected level.

Support. Recording and load generation has a lot of sophistication in the background and issues may happen in every area. Availability of good support may significantly improve productivity.

This is, of course, not a comprehensive list of criteria – rather a few starting points. Unfortunately, in most cases you can’t just rank tools on the better – worse scale. It may be that a simple tool will work quite well in your case. If your business is built around a single web site, it doesn’t use sophisticated technologies, and load is not extremely high – almost every tool will work for you. The further you are from this state, the more challenging it would be to pick up the right tool. And it even may be that you need several tools.

And while you may evaluate tools with above mentioned criteria, it is not guaranteed that a specific tool will work with your specific product (unless it uses a well-known and straightforward technology). That actually means that if you have a few system to test, you need to evaluate the tools you consider using your systems and see if the tools can handle them. If you have many, choosing a tool supporting multiple load generation options is probably a good idea (and, of course, check it with at least the most important systems).

Recent trends of agile development, DevOps, Web and Social Media sites somewhat question importance of load testing. Some (not many) openly saying that they don’t need load testing, some still paying lip service to it – but just never get to it. In more traditional corporate world we still see performance testing groups and important systems usually get load tested before deployment.

Let’s first define load testing as far as terminology is rather vague here. I use it here as anything that requires applying multi-user synthetic load – in contrast with single-user performance (which is a subset of performance engineering and may include, for example, profiling or Web Performance Optimization as it is defined now). And I use it here as an umbrella term including all other variations of multi-user testing, such as performance, concurrency, stress, endurance, longevity, scalability, etc. – but you may replace it with any other term if you prefer.

Yes, it looks like some Web and Social Media sites managed to survive without load testing. However, it looks like many such companies match the following profile:
-Business is built around a single Web site, so everybody in the company follows what is going on in production.
-Overall architecture is still clear and relatively simple. Changes (however frequent) are rather minor and evolutional.
-There is decent instrumentation providing performance information.
-There is a possibility to remove changes relatively easy.
-Site downtime/a period of slow performance (until the problem would be noticed and fixed) is not extremely painful or dangerous to the business.

Load testing is a way to mitigate load- and performance-related risks. There are other approaches and techniques that also alleviate some performance risks:
-Good single-user performance engineering practices (single-user requests performance are constantly tracked).
-Good instrumentation/Application Performance Management providing insights in what is going on inside the system.
-[Auto] scalable architecture.
-Continuous integration allowing quickly deploy and remove changes.

Still all of these don’t completely replace load testing, but rather complement it. They definitely decrease performance risk comparing with situation when nothing was done about performance at all until the last moment before rolling out the system in production without any instrumentation at all, but it still leaves risks of crashing and performance degradation under multi-user load. And if the cost of it is high, you should do load testing (what exactly and how is another large topic – there is much more here than the stereotypical waterfall-like last-moment record-and-replay approach).

There is always a risk of crashing or performance issues under heavy load – and the only way to mitigate it is actually test it. Even stellar performance in production and highly scalable architecture don’t guarantee that it won’t crash with a slightly higher load. Truly speaking, even load testing doesn’t completely guarantee it (real-life workload may be different from what you have tested), but it drastically decreases the risk.

Another important value of load testing is making sure that changes don’t degrade multi-user performance. Unfortunately, better single-user performance doesn’t guarantee better multi-user performance. In many cases it improves multi-user performance too, but definitely not always. And the more complex system, the more probable exotic multi-user performance issues no one even thought of. And a way to ensure that you don’t have such issues is load testing.

When you do performance optimization, you need a reproducible way to evaluate the impact of changes on multi-user performance. The impact on multi-user performance probably won’t be proportional to what you see with single-user performance (even if it still would be somewhat correlated). Without multi-user testing the actual effect is difficult to quantify. The same with the issues happening only in specific cases that are difficult to troubleshoot and verify in production – using load testing can significantly simplify the process.

Summarizing, I don’t see that the need in load testing is going away. Even in case of Web and Social Media sites we would probably see load testing coming back as soon as systems become more complex and performance issues start to hurt business. Maybe it would be less need for “performance testers” as it was at the heyday due to better instrumenting, APM tools, continuous integration, etc. – but I’d expect more need for performance experts that would be able to see the whole picture using all available tools and techniques (although I don’t see it yet).

While many companies promote performance testing in the cloud (or from the cloud), it makes sense only for certain types of performance testing. For example, it should work fine if we want to test how many users the system supports, would it crash under load of X users, how many servers we need to support Y users, etc., but are not too concerned with exact numbers or variability of results (or even want to see some real-life variability).

Even in this case it assumes that we don’t introduce any bottleneck using the cloud (for example, saturating network bandwidth between load generators and the system under test) and leave the cloud provider to care that our test doesn’t impact other cloud tenants (that may be not too trivial in the case of PaaS or SaaS).

However it doesn’t work for performance optimization, when we make a change in the system and want to see how it impacts performance. Testing in a cloud with other tenants intrinsically has some results variability as far as we don’t control other activities in the cloud and in most cases don’t know exact hardware configuration. For example, if the system scales out by automatic creation of an additional application instance, the new instance may be outside of the network segment where other servers are. The effects may be even more sophisticated in case of PaaS and SaaS.

So when we talk about performance optimization, we still need an isolated lab. And, if the target environment for the system is a cloud, it should be an isolated private cloud with all hardware and software infrastructure of the target cloud. And we need monitoring access to underlying hardware to see how the system maps to the hardware resources and if it works as expected (for example, testing scaling out or evaluating impacts to/from other tenants – which probably should be one more kind of performance testing to do). Real-world network emulators should be used to make sure that performance testing is representative of how the system would be used in production – otherwise we don’t taking into account such factors as network latency, bandwidth, jitter, etc. This means that we need a way to plug in the network emulation appliance properly.

So if we need optimization for cloud software, we still need a lab – but the lab should be more sophisticated to emulate the cloud environment and real-world network conditions. An ultimate example of such lab probably is the lab Microsoft created for testing IE.

So factoring in the cloud into performance testing, we have two alternatives: coarse performance testing in/from the cloud with inherent variability (and perhaps some savings on hardware and configuration costs) or granular performance testing and optimization in a sophisticated isolated lab emulating the cloud (thus avoiding variability with probably higher hardware and configuration costs).

See my reponse Is the Current Model of Load/Performance Testing Broken? on Application Performance Engineering Hub to Stephen Thair’s post.

Oracle Application Testing Suite 9.3 was released some time ago. It is available for download (subject to OTN License Agreement). Some new features and updates in this release are described in the press release.

SOASTA launched CloudTest Lite – a free edition of their performance testing solution. Basically, they give it free for up to 100 users. A serious move for sure. It should heat up the load testing tool market. It may work indeed – I guess they don’t have many paid customers in that range anyway, looks like CloudTest’s sweet spot is when you need a very large number of users. I am very interested to see how it will turn out.

Several rosy reviews were posted, for example, CloudTest Lite – A Game Changer in the Performance Tool Market by Scott Barber and SOASTA CloudTest Lite Hands-On by Bernard Golden.

As I already mentioned, it indeed is pretty interesting. However, I’d say that we need add some skepticism to be more realistic.

First, it is not the first and absolutely unique move in load testing tools. I recall a few somewhat similar moves before which then quietly disappeared. Well, I don’t remember what limitations were (maybe a little bit more restrictive). And the companies were not the leaders of the market. Moreover, there is a list of 50 open source load testing tools on opensourcetesting.com and some, like JMeter and OpenSTA, are pretty mature. Yes, open source in load testing area was not so successful as in other areas. Especially analysis is weak in most of these tools (if existent at all).

Second, releasing is just the first step. The challenge for SOASTA would be how they support a large number of non-paying users (although, of course, for a promising start-up the number of customers may be important by itself). The community maybe can help with “how-to” questions, but implementing, let’s say, enhancement requests is up to the SOASTA team. And the number of such requests may be pretty high as people start to use it with different applications.

For example, it looks like we can’t specify transactions during recording in CloudTest for the moment. Well, what I am supposed to do with a script with a few hundred identical requests in it (AJAX type, differ by incomprehensible http body content)? Track delays in the scripts and try to correlate them with recording steps? Not exactly my understanding of quick and easy.

Scott writes in his review “it is free from now until the sun explodes” Hmm… I’d rather prefer to hear this from the SOASTA team. Well, even if the SOASTA team is completely devoted to this edition, nobody can guarantee that SOASTA won’t be acquired and who knows what acquirer decides to do with the freemium edition…

Yes, Scott is not easily getting excited. Last time, as far as I remember, Scott got excited about a load testing tool when Microsoft released their tool as part of Visual Studio back in 2005. See, for example, the discussions around my old posts VisualStudio 2005 and Load Testing and Scripting Language in Performance Tools. Well, Microsoft didn’t live up to its promises and I haven’t heard about their load testing tools for a while (my understanding is that it is not dead, but doesn’t play any noticeable role). But who knew that Microsoft was losing its grip?

By the way, returning to this old post about scripting languages, I don’t object the idea of GUI-based load testing tools (I mean when script is represented by a kind of graphical tree or something like this) if we can extend it with code (as in CloudTest with JavaScript) or switch between tree and script view (as in Oracle Application Testing Suite). My concern was (and is) if you can’t extend you recording with code at all – I still believe that it limits the area of application significantly.

Anyway, it looks like we have several interesting developments in the load testing tool market that may be beneficial to the community. CloudTest and its Lite version are definitely on the list. LoadRunner AJAX True client may be introducing a new paradigm in load testing (or promoting it if follow e-Valid blog). Oracle Application Testing Suite (former Empirix) is practically a new product and is getting traction [at least in the Oracle Universe].

It looks like the competition on the load testing tools market heats up again. And, of course, new players need to differentiate themselves. I started to see statements that zero-scripting tools is a new word here. See, for example, the Agile Thinking: A New Approach to Performance Testing paper by Graham Parsons.

While I completely agree with the first part (compare with my Agile Performance Testing article), I am rather confused with the second part. I don’t see why we can’t use well-established tools like LoadRunner or JMeter (if we speak about free tools) in agile development. When you know the tool, scripting is usually not the most difficult part of your performance project.

The zero-scripting approach (if I understand it correctly) looks for me rather limited to simple web sites (and if you have such sites – you don’t need to do much in LoadRunner either). How do you handle complex correlation and parametrization often required by today’s rich web clients? And as you start to introduce ways to do it through, for example, graphical interface – you start to create a proprietary way to do this.

Also there is very interesting Fred Beringer’s post. I completely agree the main idea of the post. Performance engineering is much more than just a gate-keeping function. The terminology is somewhat vague here, but I usually use ‘performance engineering’ when I want to stress that it is more than “gate-keeping”. ‘Wisdom’ sounds a little bit too pathetic to me and the definitions of knowledge and wisdom may be argued about, but the idea is well formulated and definitely valid.

But I still believe that you may be in the ‘wisdom’ business with whatever tools are available to you. I, of course, don’t know all the features of CloudTest (the materials I read were too marketing-oriented to figure out what are real advantages). Tools help you a lot, but still it is people who are in knowledge / wisdom business, not tools. Even the best tool won’t do the work for you – you need to know how to use it, how to interpret it, etc.

And I still don’t see that drastic difference with “traditional” load testing tools. I guess CloudTest has some advantages. Listening to a webinar I realized one – they can quickly and automatically deploy agents across as many cloud servers as needed. Well, this is a significant advantage when you work in the cloud or want to run a really large-scale test (and an interesting variable here is how many users you can simulate from one server). Probably there are other advantages. But the basic principles still look the same.

Going to how Fred describes the difference:

1. Scale. Maybe CloudTest has some advantages on the high-end. It usually is a problem when you want to simulate many thousands users. But why “traditional tools limit themselves to testing behind a firewall”? Nothing, up to my knowledge, prevents you from installing them outside firewall. Yes, most performance testing in large corporations is done in a lab inside firewall – but, I guess, it is because most performance testing they do is for internal applications. And nothing actually prevents you from generating traffic from different geographies. I often did it myself by installing LoadRunner agents in remote locations.

I am not sure that I got Fred’s point in the statement: Most traditional tools, including Loadrunner, use manual dynamic session or hard-coded rules. You end up with significant repetitive work and long support delays. . At least what I saw about templates during a CloudTest webinar, looks pretty similar to what is used in “traditional” tools.

2. Speed. Why “proprietary scripting language”? LoadRunner uses ANSI C for Web protocol (and other standard languages for other protocols), Oracle Application Testing Suite uses Java, etc. Sure, they use a lot of proprietary functions there, but languages are standards and you can add your code or use external functions when you need to extend script functionality. Yes, it requires some skills – but this is required only in really complicated cases. In simple cases, you don’t care much about what language is used in the script – just record it and play it back. Anyway, looks like SOASTA uses JavaScript as a scripting language.

3. Affordability. Well, LoadRunner is not the cheapest product on the market for sure. But whatever licensing model is used by CloudTest, it is probably still a noticeable sum. While we have multiple free products around. Some, like JMeter, are mature enough.

To summarize, it quite could be that new tools have some advantages over old players. Still I don’t see a revolution yet. Some progress for sure (especially after a long period of stagnation in this market), but not a revolution. Well, maybe I just not good in marketing – but I wish to see some webinars / papers for technical audience explaining real difference and advantages.

I uploaded my CMG’10 presentations to my site.

One was A Load Testing Story. It just described one of my recent load testing projects chronologically. The point was to share some challenges of load testing and demonstrate that it may be not as trivial as sometimes believed.

Another, The Obscure World of Performance Requirements, was a small update of my old performance requirements presentation / paper. It was still a collection of topics related to performance requirements without many interconnections, but preparing it this time I got an idea how to arrange it in a more cohesive way; maybe I’d do this by CMG’11.

Dan Bartow wrote the Stop Cheating in Your Performance Tests article at STP.

Dan said: There are two dangerous practices that are rampant in performance testing right now – shrinking or removing think times and extrapolation of test results.

While I agree with all Dan’s statements in general, I’d rather put another spin here. It is cheating when you use these two valid and sound techniques irresponsible / instead of full-scale tests. But if you understand what you are doing and use these techniques intelligently, both are very powerful tools.

For the shrinking and removing think time, I’d suggest to read Rapid Bottleneck Identification – A Better Way to do Load Testing, which shows how to use this approach for rapid bottleneck identification, and some related discussion here and here. I believe that it may be a very valuable technique for performance troubleshooting in addition to full-scale tests (and in some cases it is better than nothing if full-scale tests are not possible for whatever reasons).

By the way, I rather think that shrinking and removing think time put lower load on the system because it probably would be some resources associated with every connection – and we had significantly fewer connections. Together with the possible consequences Dan mentioned, you may have quite complicated implications here, so watch your step.

For extrapolation, Dan said: Just as dangerous as shrinking your think times down is trying to make guesses about how many users you can support in a production environment when you tested at 1/10th the anticipated load in a scaled down lab. This is like saying that because you can run a mile right now, you can run a marathon with no problems.

Extrapolation, of course, should be used carefully and has risk associated with it. But if we develop the idiom with a runner further, it is irresponsible to say that you run marathon if you run one mile. But if you run one mile, you may be pretty sure that you run a half of a mile. Saying that you would run two miles has some risk associated with it, but if you know that you are still fresh at the end of the first mile (have a lot of resources available) it is still a reasonable statement. And running a mile is definitely better that just laying on sofa. So if you can’t do full-scale tests, do tests as close to full-scale as you can – and using intelligent extrapolation / modeling may help here. Anyway, the whole capacity planning discipline is based on extrapolation. But if you can run a full-scale test, but are instead running only a small test and extrapolate – this would be definitely cheating.