Alex Podelko

Performance testing is an interesting discipline: while you may see a lot of people doing it, not much is written about it. And what is written is often either introduction-level or tool-specific (other large groups of topics including performance analysis and performance troubleshooting are wider topics and not testing-specific). But not much about performance testing specific challenges beyond of what is covered by tool manuals.

My group specializes in performance testing of Hyperion / Enterprise Performance Management and Business Intelligence products at Oracle. A few of scripting challenges exist for almost every product. Nothing exceptional – you should resolve them if you have some experience and would be attentive enough, but time after time we are called to save performance testing projects to find out that there are serious problems with scripts and scenarios which make test results meaningless. Even very experienced testers stumble, but problems could be avoided if more time was spent analyzing what is going on.

Let’s consider an example – probably typical for challenges you can face with modern Web-based applications. Some operations, like financial consolidation, can take a long time. The client starts the operation on the server and then waits until it will finish, a progress bar is shown in meanwhile. When recorded, the script looks like (in LoadRunner pseudo-code):

web_custom_request(“XMLDataGrid.asp_7″,
“URL={URL}/Data/XMLDataGrid.asp?Action=EXECUTE&TaskID=1024&RowStart=1&ColStart=2&RowEnd=1&ColEnd=2&SelType=0&Format=JavaScript”,
LAST);

web_custom_request(“XMLDataGrid.asp_8″,
“URL={URL}/Data/XMLDataGrid.asp?Action=GETCONSOLSTATUS”,
LAST);

web_custom_request(“XMLDataGrid.asp_9″,
“URL={URL}/Data/XMLDataGrid.asp?Action=GETCONSOLSTATUS”,
LAST);

web_custom_request(“XMLDataGrid.asp_10″,
“URL={URL}/Data/XMLDataGrid.asp?Action=GETCONSOLSTATUS”,
LAST);

What each request is doing is defined by the ?Action= part. The number of GETCONSOLSTATUS requests recorded depends on the processing time. In the example above it was recorded three times; it means that the consolidation was done by the moment the third GETCONSOLSTATUS request was sent to the server. If playback this script, it will work in the following way: the script submits the consolidation in the EXECUTE request and then calls GETCONSOLSTATUS three times. If we have a timer around these requests, the response time will be almost instantaneous. While in reality the consolidation may take many minutes or even an hour (yes, this is a good example when sometimes people may be happy having one hour response time in a Web application). If we have several iterations in the script, we will submit several consolidations, which continue to work in background competing for the same data, while we report sub-second response times.

Consolidation scripts require creating an explicit loop around GETCONSOLSTATUS to catch the end of consolidation:

web_custom_request(“XMLDataGrid.asp_7″,
“URL={URL}/Data/XMLDataGrid.asp?Action=EXECUTE&TaskID=1024&RowStart=1&ColStart=2&RowEnd=1&ColEnd=2&SelType=0&Format=JavaScript”,
LAST);

do {

sleep(3000);

web_reg_find(“Text=1″,”SaveCount=abc_count”,LAST);

web_custom_request(“XMLDataGrid.asp_8″,
“URL={URL}/Data/XMLDataGrid.asp?Action=GETCONSOLSTATUS”,
LAST);

} while (strcmp(lr_eval_string(“{abc_count}”),”1″)==0);

Here the loop simulates the internal logic of the system sending GETCONSOLSTATUS requests each 3 sec until the consolidation is done. Without such loop the script just send requests and ends the iteration while the consolidation runs for a long time after that.

Some other examples can be found in my presentation A Load Testing Story at CMG’10 – a story about one my project which had a good share of testing challenges.

Many technical details about testing Hyperion products with LoadRunner are included in Oracle support document 1339743.1. A lot of this knowledge is now included in the Hyperion module of Oracle Application Testing Suite. But that is just about one family of products – which is a small sample of challenges you would see around.

According to my experience, creating a meaningful and realistic workload is the main performance testing challenge – and you probably face it in most non-trivial systems (including most of corporate applications). It is surprising that very few people talk about it: reading about performance testing you may get impression that the scripting is trivial and mundane (and it looks like many “performance testers” truly believe in it – and when they face any challenge, they start to talk about the death of performance testing).

Only few names come to my mind who write about specific challeneges. For example, see Northway (@northwaysg) / Scott Moore (@loadtester) performance testing blog and white papers or Mentora (@MentoraGroup ) / Dan Downing presentations and white papers. Maybe few more – but definitely they are few and far apart.

Performance testing terminology is not well defined and one of the most ambiguous terms is user concurrency. Re-reading Load Testing: Concurrent Users verses Simultaneous Users by Scott Moore (@loadtester) and LoadRunner Concurrency video by Mark Tomlinson (@mtomlins) inspired me to post this comment. Here is what I wrote (and still believe in it) in my old CMG paper about performance requirements in 2007 ( the latest version of this paper was presented again at CMG’12):

Concurrency is the number of simultaneous users or threads. It is important too: connected, but inactive users still hold some resources. For example, the requirement may be to support up to 300 active users.

The number of online users (the number of parallel session) looks like the best metric for concurrency (complementing throughput and response time requirements).

To summarize my comment, I believe that the number of “really concurrent” users is not an appropriate input metric for performance engineering and performance testing. It perhaps may be an output metric characterizing system’s load if we find a way to measure it (in a way, it is the number of users in the system if we use queuing theory terminology).

After attending Sergey Chernyshev’s (@sergeyche) Scalability vs. Performance presentation at NY Web Performance Meetup and reading Scalability: it’s the question that drives us by Robert David Graham (@ErrataRob) and Scalability vs. Performance: it isn’t a battle by Theo Schlossnagle (@postwait) I would like to share my understanding. While I agree in general with everything said, I would rather word it differently. The topic became loaded, so accents are important. Robert, for example, states that “performance” and “scalability” are orthogonal problems. Well, no, they are not. They are different, but correlated notions. Even leaving aside that performance and scalability are somewhat vague terms.

If we speak about web systems now, it looks like we can roughly separate two main components in response time (which is the main performance metric): backend (server-side) time and frontend (network and client-side time). There are subtleties and grey areas, but I’d ignore them here. The frontend time, the subject of Web Performance Optimization (WPO), doesn’t relate to scalability as far as it is not involving server processing (again ignoring subtleties).

The proportion of frontend time vs. backend time may be any at all. According to Steve Souders (@souders), the founder of WPO, 80-90% of the end-user response time is spent on the frontend. But even for major web sites the backend time for requests involving database processing (such as submitting orders or querying order status) may be more noticeable. And there are plenty of corporate web applications where the share of frontend time is rather small. Of course, the starting point of any performance troubleshooting is to find where time is spent. And there is absolutely no sense to optimize parts where time is not spent.

However, there is one important “but”. While front-end time supposed to be constant (another simplification, but again ignoring subtleties), the backend (server) time depends on load. The heavier is load, the larger may be server time. And at some point it may skyrocket making the system practically unusable. So thinking about what you need to optimize, you need to check where time is spend under maximal load. Unless you don’t care about downtime and user experience, the way to do it is load testing.

And here we get to scalability. The frontend performance indeed doesn’t matter here and is independent of scalability. But the backend performance is directly related to scalability. The relationship, of course, may be non-linear and quite sophisticated – but it does exist.

To illustrate it, let’s consider one simple (but still typical) example. The backend processing takes X ms, the time is mainly spent in CPU and we don’t have any other bottlenecks. In this case the server response time would be mainly CPU processing time – and every request would take X ms of CPU time (if we don’t have parallelism here). As soon as we take most of available CPU time, server response time would skyrocket (that situation may be modeled using queuing theory). So there is a load when the system becomes practically unusable – and the question is just when we get to this load. We, of course, may get to problems sooner if we have any scalability problem inside the system or run out of another kind of resources.

Generally speaking, you can increase scalability by either optimizing server processing (using less resources) or providing more resources. Of course, if your architecture allows using these additional resources – so mainly scalability boils down to ability to parallelize your processing (and often limited by what you can’t fully parallelize – like a centralized database).

We do have two parts of response time – frontend and backend – which behaves differently and may need different approaches and tools to optimize. But the end user experience is the sum of these two parts – where the backend time is a function of load. You can’t say much about your end-to-end performance and its backend part until you check it under load – and load testing is the safe way to do so.

Historically performance engineering concentrated on backend – where main performance and scalability issues were – and practically ignored frontend (which indeed was usually pretty straightforward then). Several sub-disciplines were formed including performance analysis, capacity planning, and load testing. Later, when sophistication of frontend skyrocketed, a whole new discipline was established by Steve Souders and quickly grew around Velocity and Web Performance meetups. Unfortunately, it practically dismissed performance engineering developments of the last 40 years (maybe even more – the Computer Measurement Group (CMG) was founded in 1975). While frontend WPO definitely has its own specific, I’d still expect to see a holistic approach to performance engineering, taking in account all aspects of performance and scalability end-to-end.

And now, after my posts about agile performance testing and performance requirements in agile projects, we are getting to another fundamental issue behind performance testing issues in agile projects.

There are two very important assumptions for what I am saying below: stakeholders understand the need in performance engineering and the team knows the basics. How to get there is a very interesting topic, but it is outside of this post. For those, who are not there yet, the issues described below may sound abstract and they probably have more urgent problems to fight – but it is something you may want to be aware about while you are getting there.

The fundamental issue is, as I see it, that performance engineering teams don’t scale well, even assuming that they are competent and effective. At least not in their traditional form. They work well in traditional corporate environments where they check products for performance before release, but they face challenges as soon as we start to expand the scope of performance engineering (early involvement, more products/configurations/scenarios, etc.). And agile projects, when we need to test the product each iteration or build, expose the problem through the increased volume of work to do.

Just to avoid misunderstandings, I am a strong supporter of having performance teams and I believe that it is the best approach to building performance culture. Performance is a special area and performance specialists should have an opportunity to work together to grow professionally. The details of organization may vary (Scott Barber, for example, specified three models: “on demand”, “on retainer”, “full immersion”), but a center of performance expertise should exist. Only thing I am saying here is that while the approach works fine in traditional environment, it needs major changes in organization, tools, and skills when the scope of performance engineering should be extended (as in the case of agile projects).

Actually remedies are well known: automation, making performance everyone jobs (full immersion), etc. However they are not wide-spread yet.

Historically performance testing automation was almost non-existent (at least in traditional environments). Performance testing automation is much more difficult than, for example, functional testing automation (I use “automation” here as what should be done for “continuous testing”, i.e. process when you run test and get a report automatically for a new build without human intervention – not in its old way, when it meant using a tool; in performance we almost always use a tool). Setups are much more complicated. A list of possible issues is long. Results are complex (not just pass/fail). It is not easy to compare two result sets. So it is definitely much more difficult and would probably require much more human intervention, but it isn’t impossible.

However, the cost of performance testing automation is high. You need to know system well enough to make meaningful automation. Automation for a new system doesn’t make much sense – overheads are too high. So there was almost no automation in traditional environment [with testing in the end with a record/playback tool]. When you test the system once in a while before next major release, chances to re-use your artifacts are low.

It is opposite when the same system is tested again and again (as it should be in agile projects). It makes sense to invest in setting up automation. It rarely happened in traditional environments – even if you test each build, they are far apart and the difference between the builds prevents re-using the artifacts (especially with recorded scripts – API, for example, is usually more stable). So demand for automation was rather low and tool vendors didn’t pay much attention to it. Well, the situation is changing – we may see more automation-related features in load testing tools soon.

There are some vendor claiming that their load testing tools better fit agile processes, but it looks like in the best case it means that the tool is a little easier to handle (and, unfortunately, often just because there is not much functionality in it). Even if there is something that may be used for automation, like starting by a command line with parameters, it is difficult to find out.

At the moment, I read about few implementation of continuous performance testing – for example OpTier or Betfair (and few about measuring response times during functional testing in single-user mode – like this – which is a good step toward full-scale performance testing automation). Some prototypes, like this, are described– but without many details.

Probably we don’t see it more often because we don’t have much infrastructure for that kind of automation and performance testers may be not the best people to create complex integrated solutions from dozens of not-related pieces (as those who implemented it did). When we get more automation support in tools, we will probably see it changing.

By the way, I am not saying that automation would replace performance testing as we know it. Performance testing of new systems is agile and exploratory in nature and can’t be replaced by automation (well, at least not in the foreseen future). Automation would complement it – together with additional input from development offloading performance engineers from routine tasks not requiring sophisticated research and analysis.

It looks like testingreflections.com, where I had blogged for a while, doesn’t exist anymore. One more confirmation that you can’t rely on somebody else long-term – who knows what would happen… Well, I still have some drafts of my old posts. Here is what I wrote about agile performance testing in 2009:

Two Views of Agile Unit Testing reminded me (indirectly) the QAForums Agile Performance Testing – Oxymoron? discussion about agile development and performance testing some time ago

Well, if we talk about agile development I don’t see any problem with performance testing at all: you have a working build each iteration – you test it. What is a problem? It should be synonym, not oxymoron.

All issues mentioned in the discussion boil down to one: nobody wants to pay for it (so no testers, no equipment, etc). Don’t see anything especially agile in it. Maybe companies still don’t switch to agile development for major projects – and don’t want to spend money for minor projects. Another explanation may be that in most traditional projects performance testing happens at the very end instead of every iteration (so require less resources) – but it not the way to do it properly.

I see significantly more problems doing performance testing properly during traditional development. I wrote a paper that has the same title, Agile Performance Testing, but completely different content: how to do performance testing in an agile way for any kind of projects (where agile projects look rather like a trivial case for me).

Well, returning to the original question of the discussion, there are a few articles in the Internet about the subject:

Agile Performance Testing by Jamie Dobson

An Explanation of Performance Testing on an Agile Team Part 1 and Part2 by Scott Barber

Chapter 6 of Performance Testing Guidance for Web Applications. Managing an Agile Performance Test Cycle.

Well, I still believe so. I still believe that performance testing in agile projects should be easier conceptually (although, of course, it would be a lot of technical difficulties – but who said that it should be easy?).

There are a few good newer articles addressing some aspects of the subject:

Top Ten Secret Weapons For Agile Performance Testing by Patrick Kua

Agile Performance Testing process AgileLoad whitepaper

Performance Testing in the Agile Enterprise by Scott Barber

Performance By Design – an Agile Approach by Jason Buksh

Still the subject is not covered much (and it looks like not actually practiced much – apart from involvement of traditional performance testing team in agile projects).

I started to prepare my talk for Belgium Testing Days (which promises to be a very interesting global testing conference) and the subject got me to very philosophical questions – will try to share my further thoughts about the subject in following posts.

Updated the list of my publications. Suddenly a lot happened in December.

At the CMG Conference I presented the latest version of my performance requirements view: Performance Requirements: the Backbone of the Performance Engineering Process, paper and presentation.

Also at the CMG Conference I presented Load Testing: See a Bigger Picture, paper and presentation. A magazine version of it, A Bird’s-Eye View of Load Testing, was published in the December issue of Software Test and Quality Assurance Magazine.

My notes on Performance Testing in the Cloud: Look Beyond the Word were published in the December issue of Testing Experience.

My post Performance: See the Whole Picture was included in 2012 Performance Calendar.

My next talk, Agile Aspects of Performance Testing, would be at Belgium Testing Days in Brussels, Belgium on March 1, 2013. Belgium Testing Days is a great global testing conference in spite of its rather local name – just check the program. All areas of testing covered – even performance testing is well represented, that is rather unusual for testing conferences.

CMG’12 is an annual conference organized by Computer Management Group – a volunteer organization of professionals specialized in performance, capacity, and IT service management. This year it is held in Las Vegas, December 2-7, 2012.

Why I love CMG, spend a lot of my time organizing and promoting it, and coming there every year (sometimes on my own)? Well, because I believe that it is the best (and actually the only) conference on performance and capacity, the main topic of my interest for the last fifteen years. There are many conferences on specific topics. For example, the Velocity conference, devoted to web performance, is significantly larger and more popular – but it is still devoted mainly to single-use web performance, leaving all other performance and capacity questions to CMG. Let me share some of my excitement – of course, from my personal point of view (there is plenty of other highlights, but I am mentioning only the ones that are close to my heart).

This year the conference covers all aspects of performance (well, almost all – performance is so sophisticated subject that there is always much more to learn) from Web Performance Optimization (the conference opens by the keynote by Patrick Meenan, a web performance Google guru and the creator of WebPagetest) to mainframe performance (and everything in between).

The conference starts with a half-day workshops – see here the description. In addition to workshops, there are CMG-T sessions during the whole conference. Each CMG-T class spans 2 or 3 session spots, so it could easily be considered as a workshop or a training class. All led by renown experts with tons of experience, you hardly would get anybody even remotely close if you engage in a typical vendor class (not to mention a unique vendor-neutral or vendor-agnostic perspective you hardly find anywhere else). You have the CMG-T track through the whole conference and every one of them is a gem:

• Capacity Planning by Ray Wicks
• z/OS Basics by Glenn Anderson
• Java Performance Analysis and Tuning by Peter Johnson
• Model and Forecasting Basics by Dr. Michael Salsburg
• Network Performance Management by Manoj Nambiar
• Windows System Performance Management and Analysis by Jeffry Schwartz
• Using SAS to Communicate Your Message by MP Welch

CMG’12 has 4 keynote/plenary session and almost a hundred regular track sessions going on from mid-Monday to mid-Friday. The conference is 5 tracks wide. One track, as I already mentioned, is CMG-T 101– type classes (with 301-depth). Others four tracks shared between five subject areas: Performance Engineering and Testing, Capacity Planning, Application Performance Management, IT Service Management, and Hot Topics. It is difficult to list all highlights – too many. While I know many great presenters and am fascinated by many topics, commenting every single one would take too much time and space. Probably you just need to look at agenda – there are three different views: preliminary agenda (overview, a day on a page), a list of abstracts in a single pdf document and search/scheduler (click on the abstract number to see the abstract).

One track on Wednesday is a Michelson award track. CMG is presenting Michelson award since 1974 (if you wonder, Albert Abraham Michelson was known for his technical accomplishments in measuring the speed of light and for his role as teacher and inspirer of others – and measuring is the key to performance). This year we will see many Michelson winners presenting: Dr. Connie Smith, the founder of Software Performance Engineering, Dr. Daniel Menasce, the author of many great books about performance and capacity planning, Adam Grummit, the author of the great Capacity Management book (ITSM Library) and the CMG president, Dr. Pat Artis, Bruce McNutt, and Dr. Michael Salsburg.

I believe that the main advantage of attending CMG is networking with best world experts in almost all areas of performance and capacity. Nowadays you can find all technical information on the Internet, but there is no substitution to face-to-face conferences to learn how to use it and what were people experiences, and, of course, to see the whole picture. Especially in performance: performance is the result of every design and implementation detail and you need to be learning all the time to keep up with coming challenges.

I am presenting there too: Load Testing: See a Bigger Picture on Thursday and
Performance Requirements: the Backbone of the Performance Engineering Process on Friday. Nothing comparing to other CMG’12 highlights, but I hope to trigger discussions around these two very important topics.

And, of course, it is Las Vegas – and Rio’s rate is $55 per night until November 14th. See you there!

The Computer Measurement Group (CMG) calls for papers and presentations for CMG’s 38th International Conference to be held in Las Vegas, Nevada, December 3rd through 7th, 2012.

The 2012 CMG conference will cover all areas of systems management, including but not limited to: capacity planning, IT service management, application performance management, performance engineering and testing, as well as the latest developments in the overall field of computer performance evaluation. See the Call for Papers and Call for Workshops for details.

CMG is the source of unbiased and objective expert information and practical, real life experiences across all computing platforms in the computer industry for over 30 years. Share your knowledge and experiences: write a paper and submit it for presentation at CMG’12.

Paper are categorized as Introductory, Tutorial, Advanced, or User Experience. I want to especially encourage all of you to consider writing a User Experience paper. Every year, the conference evaluations show a common theme: “More User Experience Papers, please!” You don’t need to be one of the field’s superstars to write one — in fact, they seem to work better from people who are just working in the field, in non-IT companies and government bodies. Just tell us what problem you faced, how you went about figuring out what the cause was, and how you dealt with it. Mentors are available for writing assistance, and may be requested at any point in the writing process, including before the paper is started. Just write mentor@cmg.org and ask.

Please take the time to participate in the CMG’12 program. It will be rewarding for both authors and attendees, and as we all share our knowledge we all become more complete professionals.
Paper submission through the CMG website is now available. For more information go to paper submission and workshop submission.

The deadline for paper submissions is June 8, 2012.

Please send questions to CMG’12 Program Chair, Bill Jouris at cmgpc@cmg.org.

Classifying and evaluating load testing tools is not easy as they include different sets of functionality often crossing borders of whatever criteria are used. In most cases, any classification is either an oversimplification (which in some cases still may be useful) or a marketing trick to highlight advantages of specific tools. There are many criteria allowing to differentiate load testing tools and it is probably better to evaluate tools on each criterion separately.

First, there are three main approaches to workload generation and every tool may be evaluated on which of them it supports and how exactly.

Protocol-level recording and the list of supported protocols. Does the tool support protocol-level recording and, if it does, what protocols it supports. With quick Internet growth and popularity of browser-based clients, most products support HTTP only or a few Web-related protocols. According to my knowledge, only HP LoadRunner and Microfocus SilkPerformer try to keep up with support of all popular protocols. So if you need recording of a special protocol, you probably end up into looking at these two tools (unless you find a special niche tool supporting your specific protocol). That somewhat explains the popularity of LoadRunner at large corporations where you probably have almost all possible protocols used. The level of support of specific protocols differs significantly too. Some HTTP-based protocols are extremely difficult to correlate if there is no built-in support, so you may look for that kind of specific support. For example, Oracle Application Testing Suite may have better support of Oracle technologies.

UI-level recording. The option was available for a long time, but it is much more viable now. For example, there was a possibility to use Mercury/HP WinRunner or QuickTest Professional (QTP) scripts in load tests, but you needed a separate machine for each virtual user (or at least a separate terminal session). That limited the level of load you may achieve drastically. Other known options were, for example, Citrix and RDP (Remote Desktop Protocol) protocols in LoadRunner – which always were the last resort when nothing else was working, but were notoriously tricky to playback. New UI-level tools for browsers, such as Selenium, extended possibilities of the UI-level approach allowing to run multiple browser per machine (so scalability is limited by resources available to run browsers). Moreover, we got UI-less browsers, such as HtmlUnit, which require significantly less resources than real browsers. There are multiple tools supporting this approach now – such as PushToTest directly harnessing Selenium and HtmlUnit for load testing or LoadRunner TruClient protocol and SOASTA CloudTest using more proprietary solutions to achieve low-overhead playback. Still questions of supported technologies, scalability, and timing accuracy remain largely undocumented, so the approach requires evaluation in every specific non-trivial case.

Programming. There are cases when you can’t (or can, but it is more difficult) use recording at all. In such cases using API calls from the script may be an option. Other variations of this approach are web services scripting and using of unit testing scripts for load testing. And, of course, you may need to add some logic to your recorded script. You program the script using whatever way you have and use the tool to execute scripts, coordinate their executions, report and analyze results. To do this, the tool should have ability to add code to (or invoke code from) your script. And, of course, if tool’s language is different from the language of your API, you would need to figure out a way to plumb them. Tools, using standard languages such as C (e.g. LoadRunner) or Java (e.g. Oracle Application Testing Suite) may have an advantage here. However you should know all details of the communication between client and server that is often very challenging.

Other important criteria are related to the environment:

Deployment Model. There were a lot of discussions about different deployment models: lab vs. cloud vs. service. There are some advantages and disadvantage of each model. Depending on your goals and systems to test you may prefer one deployment model over another. But I still believe that for comprehensive performance testing you really need both lab testing (with reproducible results for performance optimization) and realistic outside testing from around the globe (to check real-life issues that you can’t simulate in the lab). Doing both would be expensive and makes sense when you really care about performance and have a global system – but it not rare and if you are not there yet, you can get there eventually. If there are such chances, it would be better to have a tool which supports different deployment models.

If it is lab or cloud, an important sub-question would be what kind of software / hardware / cloud the tool requires. Many tools use low-level system functionality, so is may be unpleasant surprises when the platform of your choice or your corporate browser standard is not supported.

Scaling. When you have a few users to simulate, it usually is not a problem. The more users you need to simulate, the more important it becomes. The tools differ drastically on how many resources they need per simulated user and how well they may handle large volumes of information. It may differ significantly even for specific tool depending on protocol used and specifics of your script. As soon as you get to thousands of users, it may become a major problem. For a very large number of users some automation, like automatic creation of a specified number of load generators across several clouds in SOASTA CloudTest, may be very handy.

Two other important sets of functionality are monitoring of the environment and result analysis. While theoretically it is possible to do it using other tools, it significantly degrades productivity and may require building some plumbing infrastructure. So while these two areas may look optional, integrated and powerful monitoring and result analysis are very important. And the more complex system and tests, the more important they are.

Of course, non-technical criteria are important too:

Cost. There are commercial tools (and license costs differ drastically) and free tools. And there are some choices in between: for example SOASTA has the CouldTest Light edition free up to 100 users. There are many free tools (some, as JMeter, are mature enough and well-known) and many inexpensive tools, but most of them are very limited in functionality.

Skills. Considering a large number of tools and a relatively small number of people working in the area, there is a kind of labor market only for the most popular tools. Even for the second-tier tools there are few people around and few positions available. So if you don’t choose the market leaders, you can’t count that you find people with this tool experience. Of course, an experienced performance engineer will learn any tool – but it may take some time until productivity will get to the expected level.

Support. Recording and load generation has a lot of sophistication in the background and issues may happen in every area. Availability of good support may significantly improve productivity.

This is, of course, not a comprehensive list of criteria – rather a few starting points. Unfortunately, in most cases you can’t just rank tools on the better – worse scale. It may be that a simple tool will work quite well in your case. If your business is built around a single web site, it doesn’t use sophisticated technologies, and load is not extremely high – almost every tool will work for you. The further you are from this state, the more challenging it would be to pick up the right tool. And it even may be that you need several tools.

And while you may evaluate tools with above mentioned criteria, it is not guaranteed that a specific tool will work with your specific product (unless it uses a well-known and straightforward technology). That actually means that if you have a few system to test, you need to evaluate the tools you consider using your systems and see if the tools can handle them. If you have many, choosing a tool supporting multiple load generation options is probably a good idea (and, of course, check it with at least the most important systems).

Recent trends of agile development, DevOps, Web and Social Media sites somewhat question importance of load testing. Some (not many) openly saying that they don’t need load testing, some still paying lip service to it – but just never get to it. In more traditional corporate world we still see performance testing groups and important systems usually get load tested before deployment.

Let’s first define load testing as far as terminology is rather vague here. I use it here as anything that requires applying multi-user synthetic load – in contrast with single-user performance (which is a subset of performance engineering and may include, for example, profiling or Web Performance Optimization as it is defined now). And I use it here as an umbrella term including all other variations of multi-user testing, such as performance, concurrency, stress, endurance, longevity, scalability, etc. – but you may replace it with any other term if you prefer.

Yes, it looks like some Web and Social Media sites managed to survive without load testing. However, it looks like many such companies match the following profile:
-Business is built around a single Web site, so everybody in the company follows what is going on in production.
-Overall architecture is still clear and relatively simple. Changes (however frequent) are rather minor and evolutional.
-There is decent instrumentation providing performance information.
-There is a possibility to remove changes relatively easy.
-Site downtime/a period of slow performance (until the problem would be noticed and fixed) is not extremely painful or dangerous to the business.

Load testing is a way to mitigate load- and performance-related risks. There are other approaches and techniques that also alleviate some performance risks:
-Good single-user performance engineering practices (single-user requests performance are constantly tracked).
-Good instrumentation/Application Performance Management providing insights in what is going on inside the system.
-[Auto] scalable architecture.
-Continuous integration allowing quickly deploy and remove changes.

Still all of these don’t completely replace load testing, but rather complement it. They definitely decrease performance risk comparing with situation when nothing was done about performance at all until the last moment before rolling out the system in production without any instrumentation at all, but it still leaves risks of crashing and performance degradation under multi-user load. And if the cost of it is high, you should do load testing (what exactly and how is another large topic – there is much more here than the stereotypical waterfall-like last-moment record-and-replay approach).

There is always a risk of crashing or performance issues under heavy load – and the only way to mitigate it is actually test it. Even stellar performance in production and highly scalable architecture don’t guarantee that it won’t crash with a slightly higher load. Truly speaking, even load testing doesn’t completely guarantee it (real-life workload may be different from what you have tested), but it drastically decreases the risk.

Another important value of load testing is making sure that changes don’t degrade multi-user performance. Unfortunately, better single-user performance doesn’t guarantee better multi-user performance. In many cases it improves multi-user performance too, but definitely not always. And the more complex system, the more probable exotic multi-user performance issues no one even thought of. And a way to ensure that you don’t have such issues is load testing.

When you do performance optimization, you need a reproducible way to evaluate the impact of changes on multi-user performance. The impact on multi-user performance probably won’t be proportional to what you see with single-user performance (even if it still would be somewhat correlated). Without multi-user testing the actual effect is difficult to quantify. The same with the issues happening only in specific cases that are difficult to troubleshoot and verify in production – using load testing can significantly simplify the process.

Summarizing, I don’t see that the need in load testing is going away. Even in case of Web and Social Media sites we would probably see load testing coming back as soon as systems become more complex and performance issues start to hurt business. Maybe it would be less need for “performance testers” as it was at the heyday due to better instrumenting, APM tools, continuous integration, etc. – but I’d expect more need for performance experts that would be able to see the whole picture using all available tools and techniques (although I don’t see it yet).