The Role of Recording in Load Testing Tools

An interesting discussion started around a very good post Open Source Load Testing Tool Review by Ragnar Lönn. Somehow the latest comment, which I got in my e-mail, didn’t get published. So I decided to copy the discussion as is and publish the comment (and my answer) here. Check the original post (as well as many other great posts) following the link above – but the discussion below is somewhat independent. Anyway, I believe the topic deserve a separate discussion even if the comment would be eventually published in the original post.

Alexander Podelko:

Hi Ragnar,

You practically don’t mention recording. In my experience it is a major factor – and a weak spot of most open source products (some of which don’t support it at all). Many technologies generate so sophisticated HTTP queries that recording is a must.

Regards,
Alex

Ragnar Lönn:

Hi Alexander,

I agree. I should be more clear around what use case(s) I am looking to support, because that shapes my opinions in the article. What I’m looking for primarily is functionality that supports load test automation.

I know that for “traditional” load testing of web sites, where you typically try to simulate a realistic, and complex, traffic pattern, then recordings are very important. But I don’t think the traditional, complex load test scenario is suitable for automation. Its complexity means that it requires frequent maintenance to keep up to date, and it also usually means that getting value out of the test results requires manual analysis. Instead, I believe that the most accessible way for many who want to automate their load testing, is to run very simple load tests. Single API end point testing is one obvious scenario I see, where the load test configuration is more or less equal to a functional test case, and manually written (not recorded). Maybe this should be a follow-up article…?

regards,
/Ragnar

David Luu:

I concur that record (and then playback) is a good feature to have, but disagree that it’s a must for sophisticated HTTP queries to load test against.

From experience, record & playback is brittle, same as for functional test automation. One could do the recording to analyze the queries to replicate in load yes, but in practice there are better and alternate ways to do that like with a network proxy, Wireshark, or the network analyzer tab/tool in browser’s debug toolset. From the analysis you then craft the desired templated/parameterized HTTP query to make at scale using dynamically generated data or pulled from some test data source.

If just replaying same old data at scale, then one could maybe make do with replaying web/app server access logs that any load tool can easily do with some configuration/coding to read the log and send the HTTP queries in the log.

Alexander Podelko:

So, if I hear you correctly, you are saying that extracting hundreds of requests from a network analyzer (even assuming that we don’t have many issues with formatting) and copying them into a load testing tool with further manual correlation from a scratch – and doing it often as you correctly mentioned that “record & playback is brittle” – is a “better” way than doing recording in the tool itself automatically with automatic correlation (based on a correlation library)? Is it sarcasm? Am I missing something? It is like saying that we don’t need a car as we have a better way – ride a bicycle. You can get anywhere around a town on a bicycle and it is better for your health. Which may be true unless you have an one-way 60-mile daily commute or need to get from NYC to California. And saying that an one-way 60-mile daily commute or driving from NYC to California is difficult doesn’t change the fact that with a car you can do it (whatever difficult it may be) and with a bicycle you can’t (in any practical sense – theoretically, I guess, it is indeed still possible).

Somehow David’s reply wasn’t posted to the blog – so posting it here and responding to it here:

David Luu:

Well, you didn’t elaborate on the “specifics” of your type of complex test scenario. Typically when you load test a website it calls some back end APIs, but the APIs follow a certain pattern (in the URL or POST/PUT body content) that varies if you go by a data driven (or random data generation) route. If a lot of users visit the website, a lot of calls are made to the APIs but with different data/ID parameters. So it’s just data driven load testing of the same API calls concurrently with different parameters. If a single webpage or website transaction calls 20 APIs that’s ok, and manageable. If that same transaction (for your organization) calls say 100+ APIs, someone did some bad system architecture. No one in their right mind would make a website/application that calls 100 APIs to do a transaction like purchase a product, etc.

As for the other aspect of website load testing, the URLs are likely statics (files), e.g. HTML/CSS/JS, often served by CDN. So (1) there is less reason to load test something in CDN, and (2) you can use record to capture that and replay those URLs or opt to get them from your Ops team via web server/app access logs which contains the same URLs without having to do record & playback.

What I was referring to previously was the case of dynamic URLs that change, you have to do a lot of patch work still with record & playback so that you are not trying to say place an order for a product or save a product to a user’s wishlist using existing IDs/records that screw up your database. So in that case, it is really more effective to study the API calls made and craft it from scratch than to have to look through the recorded APIs and parameterize them – it ends up being same amount of effort.

Record & playback only works best if what you replay is truly static in that you don’t need to change anything. Which typically is for things like static website, blogs, etc. but not so much for things where you save something to wishlist, place an order, post a comment to blog and delete it, etc. because replaying it could throw errors or have conflicting IDs.

Alex Podelko:

I am pretty sure that there are still simple enough systems where you may just study API calls. We may see it from a large number of load testing tools that doesn’t offer much other options. But, thanks to various frameworks, Rich Internet Applications (RIA), AJAX, and other similar technologies, it looks like we see less such systems – getting close to the challenges when we had fat clients and proprietary protocols. Working mostly with business applications, I hardly can remember such system that were easy to do just studying API calls. Each system had its own quirks – and testing them without a good load testing tool with powerful recording and correlation capabilities, even if it is theoretically possible, would be a nightmare.

Let’s take the system I am working now. I’d say that it is rather average in my practice: not very complicated, but not too easy either. It has a business process, which varies depending on content. So I ended up creating 6 scripts covering different parts of functionality/content (cutting a lot of corners, by the way, if we talk about coverage). The largest script has 27 transactions, from 1 to 56 HTTP requests each (on average 10-15). All requests are the same (index.jsp) and differs mainly by http request body. We have about a dozen of different parameters in different places that need to be correlated, one gets changed in every request. Of course, in addition to parameterization of objects names – as new objects are created in the process. Here are a couple of random requests for example:

POST http://server:port/epm/faces/oracle/epm/web/main/ui/page/index.jspx?_adf.ctrl-state=u7ry1kkia_9 HTTP/1.1 Accept: */* Accept-Language: en-US,en;q=0.5 Accept-Encoding: gzip, deflate Content-Type: application/x-www-form-urlencoded; charset=UTF-8 Adf-Rich-Message: true Referer: http://server:port/epm/faces/oracle/epm/web/main/ui/page/index.jspx?_afrLoop=422167606067871&_adf.ctrl-state=u7ry1kkia_9 Content-Length: 1417 Cookie: ORA_FEPM_AUTH=true; oracle.uix=0^^GMT-7:00^p; JSESSIONID=1_aqogMT4SvHbtYOBw_Vo5hYEBXvkf9dURxzRaVB7JqS7KuKJxQq!527691855 Connection: keep-alive

org.apache.myfaces.trinidad.faces.FORM=f1&javax.faces.ViewState=!-b3k0gj3gb&event=d1&racle.adf.view.rich.monitoring.UserActivityInfo= %3Cm+xmlns%3D%22http%3A%2F%2Foracle.com%2FrichClient%2Fcomm%22%3E%3Ck+v%3D%22pr0%22%3E%3Cm%3E%3Ck+v%3D%22cid%22%3E3Cs%3E905b879470a964e9%3A -38d161bb%3A15da521ad99%3A-8000-0000000000197a2e%3C%2Fs%3E%3C%2Fk%3E%3Ck+v%3D%22cst%22%3E%3Cs%3E1501806331330%3C%2Fs%3E%3C%2Fk%3E%3Ck+v %3D%22cet%22%3E%3Cs%3E1501806333109%3C%2Fs%3E%3C%2Fk%3E%3C%2Fm%3E%3C%2Fk%3E%3Ck+v%3D%22pr1%22%3E%3Cm%3E%3Ck+v%3D%22cid%22%3E %3Cs%3E905b879470a964e9%3A-38d161bb%3A15da521ad99%3A-8000-0000000000197a2a%3C%2Fs%3E%3C%2Fk%3E%3Ck+v%3D%22cst%22%3E%3Cs%3E1501806331288 %3C%2Fs%3E%3C%2Fk%3E%3Ck+v%3D%22cet%22%3E%3Cs%3E1501806331330%3C%2Fs%3E%3C%2Fk%3E%3Ck+v%3D%22rrt%22%3E%3Cs%3E1501806331328%3C%2Fs%3E%3C %2Fk%3E%3C%2Fm%3E%3C%2Fk%3E%3Ck+v%3D%22prm%22%3E%3Cm%3E%3Ck+v%3D%22cst%22%3E%3Cs%3E1501806333204%3C%2Fs%3E%3C%2Fk%3E%3Ck+v%3D%22eif%22%3E %3Cm%3E%3Ck+v%3D%22ety%22%3E%3Cs%3EonStartupHandler%3C%2Fs%3E%3C%2Fk%3E%3Ck+v%3D%22cld%22%3E%3Cs%3Ed1%3C%2Fs%3E%3C%2Fk%3E%3Ck+v%3D%22cty %22%3E%3Cs%3Eoracle.adf.RichDocument%3C%2Fs%3E%3C%2Fk%3E%3C%2Fm%3E%3C%2Fk%3E%3C%2Fm%3E%3C%2Fk%3E%3C%2Fm%3E&event.d1=%3Cm+xmlns%3D%22http%3A%2F%2Foracle.com%2FrichClient%2Fcomm%22%3E%3Ck+v%3D%22_custom%22%3E%3Cb%3E1%3C%2Fb%3E%3C%2Fk%3E%3Ck+v%3D%22type%22%3E %3Cs%3EonStartupHandler%3C%2Fs%3E%3C%2Fk%3E%3C%2Fm%3E&oracle.adf.view.rich.PROCESS=d1

POST http://server:port/epm/faces/oracle/epm/web/main/ui/page/index.jspx?_adf.ctrl-state=u7ry1kkia_9 HTTP/1.1 Accept: */* Accept-Language: en-US,en;q=0.5 Accept-Encoding: gzip, deflate Content-Type: application/x-www-form-urlencoded; charset=UTF-8 Adf-Ads-Page-Id: 1 Adf-Rich-Message: true Referer: http://server:port/epm/faces/oracle/epm/web/main/ui/page/index.jspx?_afrLoop=422167606067871&_adf.ctrl-state=u7ry1kkia_9 Content-Length: 1543 Cookie: ORA_FEPM_AUTH=true; oracle.uix=0^^GMT-7:00^p; JSESSIONID=1_aqogMT4SvHbtYOBw_Vo5hYEBXvkf9dURxzRaVB7JqS7KuKJxQq!527691855 Connection: keep-alive

epmhr1:0:r9:1:pt1:epmsrct=&org.apache.myfaces.trinidad.faces.FORM=f1&javax.faces.ViewState=!-b3k0gj3gb& oracle.adf.view.rich.DELTAS=%7Bepmhr1%3A0%3Ar9%3A1%3Apt1%3Aepm_led%3D%7BviewportSize%3D12%7D%2Cepmhr1%3A0%3Ar9%3A1%3Apt1%3Aepm_lec %3D%7BviewportSize%3D10%7D%7D&event=epmhr1%3A0%3Ar9%3A1%3Apt1%3Aepm_lec%3A0%3Acl1j_id_3& oracle.adf.view.rich.monitoring.UserActivityInfo=%3Cm+xmlns%3D%22http%3A%2F%2Foracle.com%2FrichClient%2Fcomm%22%3E%3Ck+v%3D%22pr0%22%3E %3Cm%3E%3Ck+v%3D%22cid%22%3E%3Cs%3E905b879470a964e9%3A-38d161bb%3A15da521ad99%3A-8000-0000000000197ad8%3C%2Fs%3E%3C%2Fk%3E%3Ck+v%3D%2 2cst%22%3E%3Cs%3E1501806631340%3C%2Fs%3E%3C%2Fk%3E%3Ck+v%3D%22cet%22%3E%3Cs%3E1501806632071%3C%2Fs%3E%3C%2Fk%3E%3Ck+v%3D%22rrt%22%3E %3Cs%3E1501806631904%3C%2Fs%3E%3C%2Fk%3E%3C%2Fm%3E%3C%2Fk%3E%3Ck+v%3D%22prm%22%3E%3Cm%3E%3Ck+v%3D%22cst%22%3E%3Cs%3E1501806669571%3C%2Fs %3E%3C%2Fk%3E%3Ck+v%3D%22eif%22%3E%3Cm%3E%3Ck+v%3D%22ety%22%3E%3Cs%3Eaction%3C%2Fs%3E%3C%2Fk%3E%3Ck+v%3D%22cld%22%3E %3Cs%3Eepmhr1%3A0%3Ar9%3A1%3Apt1%3Aepm_lec%3A0%3Acl1j_id_3%3C%2Fs%3E%3C%2Fk%3E%3Ck+v%3D%22cty%22%3E%3Cs%3Eoracle.adf.RichCommandImageLink %3C%2Fs%3E%3C%2Fk%3E%3Ck+v%3D%22cdn%22%3E%3Cs%3ERP_user0001_00001%3C%2Fs%3E%3C%2Fk%3E%3Ck+v%3D%22rvd%22%3E%3Cs%3E %2FlibraryBaseTF%2FlibraryBase%3C%2Fs%3E%3C%2Fk%3E%3C%2Fm%3E%3C%2Fk%3E%3C%2Fm%3E%3C%2Fk%3E %3C%2Fm%3E&event.epmhr1:0:r9:1:pt1:epm_lec:0:cl1j_id_3=%3Cm+xmlns%3D%22http%3A%2F%2Foracle.com%2FrichClient%2Fcomm%22%3E%3Ck+v%3D%22type %22%3E%3Cs%3Eaction%3C%2Fs%3E%3C%2Fk%3E%3C%2Fm%3E&oracle.adf.view.rich.PROCESS=epmhr1%3A0%3Ar9

Yes, it is fragile. Sometimes I can track the change and fix a script, sometimes I need to re-create all six scripts from a scratch. As it uses a framework, developers don’t actually know how their change impact protocol level. With a good tool and my set of correlation rules I may recreate them in a couple of days. Not a fun for sure, but works. May I do it with a tool without good reliable recording and correlation rules functionality? Don’t want even think about it.

Can load generation done with that framework API? Well, perhaps. You develop a kind of a shadow system with almost all security and functionality built-in. Maybe some developers may go this way – which may have some advantages and some shortcomings – when it is their only main system. Not exactly my case here. And we are discussing load testing tools – which are created to test systems without major development efforts. And in many cases (at least in corporate IT) you may have no access to developers at all.

I don’t quite understand the notion “Record & playback only works best if what you replay is truly static in that you don’t need to change”. It works best in most cases (while, of course, there are exceptions) – and it is the reason while it is the main approach for testing non-trivial systems. We don’t speak here about beginners’ errors – like replaying without correlation or relying on automatic correlation only. Maybe we have a didactic aspect here: how you want to teach people to show what is behind the scene. I may see a point here. But assuming that you know what you are doing – what may be a drawback of recording? Even if you have good REST APIs – what may be wrong in seeing what are the actual sequence of requests and actual parameters used instead of trying to figure out them yourself from a scratch? Using a separate tool for recording (like Fiddler) is exactly the same except the lack of integration and need to copy it manually between the tools.

So I strongly believe that load testing tools must have powerful recording and correlation. Well, at least if they want to get into corporate market beyond some niche usage. Yes, we have some niches for load testing tools without recording. As well as there are ways to create load without using load testing tools at all – for example, by creating your own framework or re-directing real traffic. But not having recording heavily limits tool’s usage to simpler cases and make it practically useless for corporate IT.

14 Responses to "The Role of Recording in Load Testing Tools"

David Luu says:

August 16, 2017 at 3:40 am

Good response Alex. David here responding back. I will state that from my experience with the load test tooling (jmeter, I believe quite a few of the other tools are similar), the tool itself is simply a web proxy that requires the HTTP traffic and spits out a generated test script based on the recording.

You do have a point, depending on one’s use case & specific complexity, it may be worthwhile to just edit the recording generated script filling in the needed parameters and making other tweaks. However, in other cases (I believe it’s at least 50/50) the gain from just editing the generated script versus building one from scratch is marginal.

And when I mean create from scratch I didn’t mean totally from scratch, you still base data off captures from other tooling Firebug / browser developer tool network captures, other web proxies, or network sniffer like Wireshark. What I’ve found in this area is that the data sometimes is much more easily analyzed from these tools (say in bulk of many requests) versus looking through the generated script – although it’s a bit biased depending on the tooling, I do know jmeter’s isn’t that user friendly for analysis/modification of many requests “samplers” generated from the recording. Maybe some of the other tools are simpler.

Also, some tooling/protocols like XMPP, SIP, etc. likely don’t have recording for their load generators, or good ones at all compared to HTTP, so one has to learn to rely on industry standard tooling like Wireshark (and related TCP dump) tooling to do that job.

It really comes down to one’s own preference too. Maybe to non-programmers, recording + playback is straightforward.

As a developer and data analyzer, I want to easily work with the data & and analyze it in tabular/CSV form (or scriptable). So unless the recording generated test is a DSL (domain specific language) or real programming script (not some script editable via many GUI actions like jmeter), I’d prefer a load test tool that provides an interface to 1st preview the the recorded data to see what you working with and make tweaks there before having it generate the actual/final script. And of course, one can tweak it more at the end as well after generation.

Per your initial comment and this discussion, it probably just means we need a major disruption in how record & playback functionality is implemented in load test tools (for it to be much better than what we see on the market). Perhaps you could look into how you could contribute/guide the progress of such for the open source load test tools. Evaluate them, and voice your feedback to the individual projects for the tools you find of interest. Feedback is crucial. And (code or other type of) contributions also help too, the core project developers can’t do everything themselves.

I will tell you one thing I’ve personally experienced & found is sometimes (for open source), you make a contribution to benefit yourself as well as thinking it could help others, and you later find surprisingly that others may build upon your initial contributions and make things much better going forward. This I experienced in the realm of functional test automation, but not yet in load test tooling since I’ve not been involved much other than some usage discussions on jmeter users mailing list.
Ben Xu says:

August 16, 2017 at 6:28 am

I am a dev manager working on Oracle application testing suite , which is a commercial tool and providing both functional and load testing products for testing any web application, especially expert at testing all kinds of Oracle applications.

All PSR teams inside Oracle using our tool for load testing. These load testing can be trigger tens of thousands of VU, and run days …

I personally can’t imagine a good load testing script creation tool without recording support to solve correlation issues.

For any complex application like Oracle Fusion, Siebel, or old EBS, without recording support in testing tool to solve correlation issues automatically, how can load testing engineer create a meaning load testing scenario?

For anyone still has question on this point, the only reason is that they never done real load testing for any complex application (whether it is Oracle or SAP or any other big application…).

There is site which build on JSON restful web service, and can just test the web service API.
BUT if you need to some real performance testing using a scenario. Good recording and correlation support is still needed.

In short, if anyone has done real performance testing on big and complex system, he or she will not have any question and recording support for load testing tool.
David Luu says:

August 16, 2017 at 1:27 pm

I partly agree and disagree with you guys. Yes, recording feature will help the load testing complex applications. However, it doesn’t always. It may also be industry specific, from this blog post & Ben’s reply, seems like your issues deal with software from Oracle. Not to be biased, perhap’s Oracle’s tooling is rather “complex” and they need to make things simpler.

Bear in mind in this big wide world of tech industry, a lot of people do testing in other fields, so you can’t say based on your specific industry/software that it applies to all others in general. Well build websites and applications don’t suffer from this complexity you mention, even for their JSON REST APIs. This Oracle stuff I would theorize has its complexity partly based on legacy code they keep around rather than a modern rewrite, which a lot of companies tend to do with their legacy systems, just patching them for improvements over time to keep making use of it.

As for recording part, while its good to have, it’s also not necessary to need it to be incorporated natively into the tool. Take for instance Selenium WebDriver. Sure there’s an IDE and other third party tools, but recording & playback isn’t a major focus of that tool, and yet its the industry standard for web UI automation. You can always do the recording part & script generation from it separately.

Good load testing also benefits from a performance test engineer with good developer/coding experience, as well as performance & testing experience. With such skills one doesn’t have to rely on built-in recording of a load test tool to help do the testing and correlation. Take for example, what if you work in a special industry (or new technology areas) where you don’t have good or any (existing) tooling to help you generate load or do the recording all-in-one? You’re screwed if you rely on existing tools, so in this case you have to build your own tooling and use different toolsets in a chained workflow to do what you want – and that’s hard to do without good development experience as a performance test engineer. Built-in recording is more of a requirement for those lacking such skills to deal with or workaround those problems.

Btw, this discussion made me recall something I built a long time ago for some work I had to do, that would take captured data (not from a load test tool) and generate a script that would automate the workflow – it could be adapted to generate script output for any load test tool if you knew the tools script format for generating load/requests and analyzing the responses. This was easy to piece together really, and I had to do it because there wasn’t any good tool to record & playback what I wanted, and that was free, since we’re not a company that could afford fancy tools (plus there’s vendor lockin if you chose to go that route having been dependent on some fancy pricey test tool). I’d do something like this if the load test tooling I use was lacking.

https://www.codeproject.com/Articles/15128/Web-HTTP-Automation-with-Perl

later ported it for some other languages as well https://www.codeproject.com/Articles/19600/Web-HTTP-Request-and-Automation-Libraries
David Luu says:

August 16, 2017 at 1:35 pm

Btw, if I had to test what sounds to me as ugly complex Oracle software interface (based on you guy’s comments), I’d look for ways to solve it with custom coding and open source tooling rather than hoping for a nice tool that does recording and stuff to help me out, or pay for some $$$ commercial tool that supposedly helps me test Oracle software, unless of course my employer has the $$$ to not care to spend on such commercial tool, then I’d be ok with that. But some companies are cash strapped and prefer not to spend on such software/licenses if they don’t have to. Not for Oracle software, but I’ve done that kind of thing with previous jobs for other software / technology stack. A good (test) engineer will find ways to solve a problem regardless if you give him a basic screwdriver or a fancy powered cordless drill + screwdriver tool as an analogy.
Alex Podelko says:

August 16, 2017 at 2:39 pm

David,

You brought in a lot of topics… Let me to answer some:

-JMeter recording/correlation functionality looks rather minimal for me, not even close to what you see in LoadRunner or Oracle Application Testing Suite. Cordless drill vs. screwdriver would be a good analogy.

-My understanding is that recording/correlation functionality are very challenging and time-consuming parts to implement and even with advanced commercial tools there are a lot of issues there. It looks like in open source nobody is ready to dive into that. It looks like organizations building on the top of open source products – such as BlazeMeter or OctoPerf – starting to address it.

-Nope, it is not an Oracle-specific thing. First, many Oracle applications were acquired (Siebel, Peoplesoft, Hyperion, etc.) – so the current Oracle application set is a selections of the best applications available on the market. Second, I am 20 years in that business and paying attention to what going on there. Believe me, very few business app will be simple enough to fit your description. There are many reasons for that – need to be more generic, complex business processes vs. single-purpose web sites, need to support existing functionality/data (yes, legacy), etc.

-No, I don’t see that it becomes easier. Vise versa, I rather see that it becomes more sophisticated that to various frameworks, Rich Internet Applications (RIA), AJAX, and other similar technologies. REST APIs is a good trend – but 1) it is not everywhere 2) it becomes complex for complex applications.

-If you don’t want to pay $$$ for good tools, you pay $$$ testers for wrestling with simple tools / plumbing and for losses for work not done / not done properly. Of course, the balance depends on context. Some don’t test at all to save money. Have you seen any professional builders who use basic screwdrivers only instead of powered cordless drill to save moneys on tools?

-Said that, I am pretty sure that there are many contexts where tools without recording / with minimal one may work – early startups, simple / limited number of systems, etc. Yes, many people over-rely on recording / automatic correlation. Still a bicycle is a bicycle – to travel farther or faster you need a car…

Regards,
Alex
Richard Bishop says:

August 17, 2017 at 8:38 am

I can see both sides to the above argument but the answer to the question, “Do I need a record/replay tool or not?” is very difficult to answer without a detailed understanding of the specific application that you’re looking to test as well as the type of testing that you’re trying to do.

Examples where you could make a case for testing without record/replay:
Unit tests – e.g. testing a single API’s responsiveness when making changes to a back end.
Break tests of a simple process – e.g. testing a load balancer or CDN
Simple comparative (regression) tests – e.g. Did my change make things better or worse?

Examples where more complex record/replay testing may be appropriate:
End to end testing of a complex application with multiple dependencies on other systems. e.g. testing a banking application that relies on multiple back end systems, all of which could be experiencing demands from other applications / business units.
Recording any non-web applications. Applications like JMeter and other applications with an open-source heritage often provide record replay options for standard web applications. Anything that needs to be recorded at a higher level can only be done with one of the enterprise grade test tools.
Applications which rely on Citrix, Oracle Forms, Green Screen, SAP GUI, RDP, ODBC or other protocols are best left to the enterprise grade tools. These enterprise tools often have better reporting and analytics functionality as well.

Basically it’s “horses for courses”, people will always answer that tool X is better than tool Y, but it may not be better in every use case. It’s important to choose a tool based on a deep understanding of all possible use cases otherwise you end up with multiple test tools to test multiple applications and don’t benefit from any economies of scale.

I’m with Alex on this one, I’d rather use a car than a bike (most of the time). 😊
James Pulley says:

August 17, 2017 at 10:45 am

I think David’s assumption on the static content being serviced by CDN exclusively is a poor assumption. Admittedly I see this often as an excuse to simplify scripts.

This assumption is one that dramatically increases the load against the web tier in production when the assumption proves untrue. In my experience, this has been provably untrue 100% of the time that the cache model is covering all static components effectively, reducing the load on the web tier for those components to a whisper. But, if you never test for this configuration then you cannot cite a defect and have the issue addressed. Also “assumes” a CDN, which is still not universal.

As others have noted, included Mr Bishop just before me, is that this programmatic driven model works well in the shift left areas of development when you are trying to shake out a given API as performance unit tests, but where you are trying to model actual user behavior to understand risk and performance of users in production then you need a precise model of the user behavior to start your modifications.

This is where recording is key.

As Alex noted, Developers too often have little understanding of the impact of their work at the HTTP layer, the layer most often used for performance testing websites. As a result, they cannot reliably predict and hand code an accurate model of what a set of user transfers will contain
Greg Hinsley says:

August 17, 2017 at 12:00 pm

Agree with Alex, James and Bish here.

Playing devil’s advocate though I do sometimes prefer standalone recorders (Charles, Fiddler etc) to built in one’s.

Granted you lose automatic correlation needed to ease the pain of recording aforementioned complex applications, but there can be benefits:

– subjectively easier to manipulate a recorded session, filter response type / domain etc
– these tools tend to have additional features like stubbing / manipulating responses etc

The last one is a good skill to have, particularly as we all become more T-shaped
David Luu says:

August 17, 2017 at 3:40 pm

More participants in the discussion 🙂

RE: CDN and load testing, pardon my wording, I didn’t mean that it shouldn’t be tested but that in the grand scheme of things, it’s lower in priority to scale test than the APIs and more dynamic components of a system since these are static regardless if served over CDN, a caching layer, or non-cached but files served directly by the web/app server. It is good to test it in the final end-to-end tests. But having to account for all static/CDN assets in initial testing (and keeping it updated by periodically doing record & playback or manual tweaks since the marketing team tends to update these periodically/often) is effort better spent after verifying the dynamic data that’s served since that’s typically the primary bottleneck before you hit static asset bottlenecks, and mostly the interesting content is the dynamic part and not static, particularly in e-commerce/transactions.

All good points, I haven’t been familiar with other industries like finance and government, etc. so pardon my bias towards the modern web & telecom for which I’m more familiar with.

I wonder what you all think regarding load testing around modern cloud and micro service based systems around streaming data and big data systems. I may be mistaken (or stand corrected), but I doubt commercial load test tooling (and even open source tooling) has kept up with these new modern architectures surrounding technologies like Apache kafka, storm, hadoop, etc. We’re using different toolsets and build in house tooling for load generation and correlation for this new architecture.

And speaking of testing the above new architecture, as well as testing the complex applications you all speak of (that’s less web/HTTP based), I wonder how well the commercial load test tooling’s (auto) correlation capabilities are. Since I imagine a portion of things to correlate are actually business logic or user specific, to which a general purpose commercial and open source tool can not fully cover. In such cases, generating and tracking custom metrics or KPIs is what helps, and one might have to customize their own correlation around that via eyeing customized dashboards or running script/code that would analyze those custom KPIs/metrics. I’d imagine the load tools correlation capabilities that are general purpose useful are more likely request/response latencies, request/response throughput, and user/client counts simulated. Or are there more that I’m not aware of that are available in commercial load test tooling? Although I suppose if the test tooling solution is an “all-in-one” that comes with components like install an agent on the systems under test and we can also give you additional general purpose metrics to auto-correlate like CPU/memory/disk I/O and network I/O utilization, etc.
Ben Xu says:

August 18, 2017 at 10:34 am

As a proud Oracler, I feel strange for people to say “ugly complex Oracle software interface”.

If the biggest objective is to build a clean and most non-complex software, I will write a “hello world”.

Oracle is the one who put 10K+ developer, continuously developing 10+ years, for Oracle’s SAAS cloud, biggest enterprise SAAS solution in this world.

Also strange for others to have idea these who do performance testing at Oracle doesn’t know programming well, so that use a recording thing…

Doing things like use an open source testing framework to test restful web service is easy, any intern has some java experience who didn’t do such things before can quickly program such “clean” test in days…

If you are inside Oracle or have friends at Oracle, check these PSR team, these are one of the most competent people. You can easily find a VP/Senior director which only have a much smaller team than product development team, many PSR engineer are principal engineer and above…
They not only do performance testing to find issues. PSR team also involved at find root cause of these issue and sometimes also involved in solve it ……

It’s always that most complex issues need most competent people to solve.
Using all kinds of open source solution, set up a test framework, manually write some code which seems cool is not the objective for testing.

Testing need to find and solve issues. Using any kinds of way which save time and more efficient, and when you know there is a tool which can help you do things like recording script / auto correlation, which saves your effort, try to use it and check if it works.

I am proud to being a team member to build such tool. To free our user’s time spend on using the tool, and let them focus on finding and solving real issues(which is much more complex and challenge than focused on using tools/program test code or create some test framework, also has much bigger value and impact ).

It’s really easy to confuse means and ends, for all area including performance testing or the bigger software testing scope.
Ragnar Lonn says:

November 1, 2017 at 8:50 am

Late comment, didn’t realize the discussion happened here 🙂

Maybe it was already obvious, but I just want to add that while I strongly believe that “simple” load testing of e.g. individual API end points (what we have started calling “unit load tests”) is the way to go for most developers, and that this use case is *not* very dependent on recording capabilities, I am not in any way saying there is no need for recording capabilities in load testing tools. I expect the “traditional” type of complex, realistic-traffic-scenario-type tests to stay relevant for the forseeable future, for those organizations that can afford to run them. And for that use case, recording capabilities are probably very relevant.

For most developers out there though, load testing has long been completely out of reach. Enabling them to run *some* load tests, and to run them in an automated fashion, is a huge improvement. I see this as a very good first step. When they have some user-facing API end point tests (“unit load tests”) in place they can start thinking about testing deeper – e.g. testing their microservices – or testing with more complex traffic (perhaps they can build complexity out of the simple unit load tests). Then they will need more features – ability to test other protocols, recording capabilities etc.

So I think that the use case I am most interested in, is perhaps not the same use case that most experts in this discussion are most interested in. Just to explain again why recording wasn’t something I looked much at when writing the article.
David says:

May 9, 2018 at 3:10 pm

Btw, I was reminded of this discussion from a recent discussion elsewhere that made use of the term “correlation”. I didn’t inquire it earlier in this discussion or the original one that this blog post links to. (https://groups.google.com/forum/#!topic/gatling/j-8OwSL3GrY)

But if I’m not mistaken, the term “correlation” here is not the same as the other typical meaning of “correlation” in terms of evaluating and comparing data from test results and other items like metrics and KPIs to establish/confirm a pattern or trend for potential issues that can occur (as peformance drop in relation to load increase or this and that, etc.). Correlation as used by the experts here seem to be in relation to data extraction, parsing, and parameterization in terms of templatizing a recorded load test scenario, to minimize or avoid doing that work manually.

Am I correct or mistaken? If correct, do we have some article(s) somewhere that elaborate on this terminology of correlation vs the other meaning of correlation?
Alex Podelko says:

May 9, 2018 at 6:42 pm

You are absolutely correct. The word correlation is used completely differently in load testing than is statistics or as relation. Probably was introduced by LoadRunner and widely used in load testing, so now is often used by other tools too. However there are opinions that correlation shouldn’t be used this way – I remember seeing some interesting discussions on the subject, but can’t find them. There are numerous posts explaining correlation on the Internet – you may look, for example, this one demonstrating what LoadRunner does in that area.
Mira says:

July 21, 2020 at 5:47 am

Recording test cases has proved to be a very useful functionality for QA professionals when using test case recording tools like QARA Enterprise or TestStudio.

Leave a Reply