The Role of Recording in Load Testing Tools

An interesting discussion started around a very good post Open Source Load Testing Tool Review by Ragnar Lönn. Somehow the latest comment, which I got in my e-mail, didn’t get published. So I decided to copy the discussion as is and publish the comment (and my answer) here. Check the original post (as well as many other great posts) following the link above – but the discussion below is somewhat independent. Anyway, I believe the topic deserve a separate discussion even if the comment would be eventually published in the original post.

Alexander Podelko:

Hi Ragnar,

You practically don’t mention recording. In my experience it is a major factor – and a weak spot of most open source products (some of which don’t support it at all). Many technologies generate so sophisticated HTTP queries that recording is a must.

Regards,
Alex

Ragnar Lönn:

Hi Alexander,

I agree. I should be more clear around what use case(s) I am looking to support, because that shapes my opinions in the article. What I’m looking for primarily is functionality that supports load test automation.

I know that for “traditional” load testing of web sites, where you typically try to simulate a realistic, and complex, traffic pattern, then recordings are very important. But I don’t think the traditional, complex load test scenario is suitable for automation. Its complexity means that it requires frequent maintenance to keep up to date, and it also usually means that getting value out of the test results requires manual analysis. Instead, I believe that the most accessible way for many who want to automate their load testing, is to run very simple load tests. Single API end point testing is one obvious scenario I see, where the load test configuration is more or less equal to a functional test case, and manually written (not recorded). Maybe this should be a follow-up article…?

regards,
/Ragnar

David Luu:

I concur that record (and then playback) is a good feature to have, but disagree that it’s a must for sophisticated HTTP queries to load test against.

From experience, record & playback is brittle, same as for functional test automation. One could do the recording to analyze the queries to replicate in load yes, but in practice there are better and alternate ways to do that like with a network proxy, Wireshark, or the network analyzer tab/tool in browser’s debug toolset. From the analysis you then craft the desired templated/parameterized HTTP query to make at scale using dynamically generated data or pulled from some test data source.

If just replaying same old data at scale, then one could maybe make do with replaying web/app server access logs that any load tool can easily do with some configuration/coding to read the log and send the HTTP queries in the log.

Alexander Podelko:

So, if I hear you correctly, you are saying that extracting hundreds of requests from a network analyzer (even assuming that we don’t have many issues with formatting) and copying them into a load testing tool with further manual correlation from a scratch – and doing it often as you correctly mentioned that “record & playback is brittle” – is a “better” way than doing recording in the tool itself automatically with automatic correlation (based on a correlation library)? Is it sarcasm? Am I missing something? It is like saying that we don’t need a car as we have a better way – ride a bicycle. You can get anywhere around a town on a bicycle and it is better for your health. Which may be true unless you have an one-way 60-mile daily commute or need to get from NYC to California. And saying that an one-way 60-mile daily commute or driving from NYC to California is difficult doesn’t change the fact that with a car you can do it (whatever difficult it may be) and with a bicycle you can’t (in any practical sense – theoretically, I guess, it is indeed still possible).

Somehow David’s reply wasn’t posted to the blog – so posting it here and responding to it here:

David Luu:

Well, you didn’t elaborate on the “specifics” of your type of complex test scenario. Typically when you load test a website it calls some back end APIs, but the APIs follow a certain pattern (in the URL or POST/PUT body content) that varies if you go by a data driven (or random data generation) route. If a lot of users visit the website, a lot of calls are made to the APIs but with different data/ID parameters. So it’s just data driven load testing of the same API calls concurrently with different parameters. If a single webpage or website transaction calls 20 APIs that’s ok, and manageable. If that same transaction (for your organization) calls say 100+ APIs, someone did some bad system architecture. No one in their right mind would make a website/application that calls 100 APIs to do a transaction like purchase a product, etc.

As for the other aspect of website load testing, the URLs are likely statics (files), e.g. HTML/CSS/JS, often served by CDN. So (1) there is less reason to load test something in CDN, and (2) you can use record to capture that and replay those URLs or opt to get them from your Ops team via web server/app access logs which contains the same URLs without having to do record & playback.

What I was referring to previously was the case of dynamic URLs that change, you have to do a lot of patch work still with record & playback so that you are not trying to say place an order for a product or save a product to a user’s wishlist using existing IDs/records that screw up your database. So in that case, it is really more effective to study the API calls made and craft it from scratch than to have to look through the recorded APIs and parameterize them – it ends up being same amount of effort.

Record & playback only works best if what you replay is truly static in that you don’t need to change anything. Which typically is for things like static website, blogs, etc. but not so much for things where you save something to wishlist, place an order, post a comment to blog and delete it, etc. because replaying it could throw errors or have conflicting IDs.

Alex Podelko:

I am pretty sure that there are still simple enough systems where you may just study API calls. We may see it from a large number of load testing tools that doesn’t offer much other options. But, thanks to various frameworks, Rich Internet Applications (RIA), AJAX, and other similar technologies, it looks like we see less such systems – getting close to the challenges when we had fat clients and proprietary protocols. Working mostly with business applications, I hardly can remember such system that were easy to do just studying API calls. Each system had its own quirks – and testing them without a good load testing tool with powerful recording and correlation capabilities, even if it is theoretically possible, would be a nightmare.

Let’s take the system I am working now. I’d say that it is rather average in my practice: not very complicated, but not too easy either. It has a business process, which varies depending on content. So I ended up creating 6 scripts covering different parts of functionality/content (cutting a lot of corners, by the way, if we talk about coverage). The largest script has 27 transactions, from 1 to 56 HTTP requests each (on average 10-15). All requests are the same (index.jsp) and differs mainly by http request body. We have about a dozen of different parameters in different places that need to be correlated, one gets changed in every request. Of course, in addition to parameterization of objects names – as new objects are created in the process. Here are a couple of random requests for example:

POST http://server:port/epm/faces/oracle/epm/web/main/ui/page/index.jspx?_adf.ctrl-state=u7ry1kkia_9 HTTP/1.1
Accept: */*
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate
Content-Type: application/x-www-form-urlencoded; charset=UTF-8
Adf-Rich-Message: true
Referer: http://server:port/epm/faces/oracle/epm/web/main/ui/page/index.jspx?_afrLoop=422167606067871&_adf.ctrl-state=u7ry1kkia_9
Content-Length: 1417
Cookie: ORA_FEPM_AUTH=true; oracle.uix=0^^GMT-7:00^p; JSESSIONID=1_aqogMT4SvHbtYOBw_Vo5hYEBXvkf9dURxzRaVB7JqS7KuKJxQq!527691855
Connection: keep-alive

org.apache.myfaces.trinidad.faces.FORM=f1&javax.faces.ViewState=!-b3k0gj3gb&event=d1&racle.adf.view.rich.monitoring.UserActivityInfo=
%3Cm+xmlns%3D%22http%3A%2F%2Foracle.com%2FrichClient%2Fcomm%22%3E%3Ck+v%3D%22pr0%22%3E%3Cm%3E%3Ck+v%3D%22cid%22%3E3Cs%3E905b879470a964e9%3A
-38d161bb%3A15da521ad99%3A-8000-0000000000197a2e%3C%2Fs%3E%3C%2Fk%3E%3Ck+v%3D%22cst%22%3E%3Cs%3E1501806331330%3C%2Fs%3E%3C%2Fk%3E%3Ck+v
%3D%22cet%22%3E%3Cs%3E1501806333109%3C%2Fs%3E%3C%2Fk%3E%3C%2Fm%3E%3C%2Fk%3E%3Ck+v%3D%22pr1%22%3E%3Cm%3E%3Ck+v%3D%22cid%22%3E
%3Cs%3E905b879470a964e9%3A-38d161bb%3A15da521ad99%3A-8000-0000000000197a2a%3C%2Fs%3E%3C%2Fk%3E%3Ck+v%3D%22cst%22%3E%3Cs%3E1501806331288
%3C%2Fs%3E%3C%2Fk%3E%3Ck+v%3D%22cet%22%3E%3Cs%3E1501806331330%3C%2Fs%3E%3C%2Fk%3E%3Ck+v%3D%22rrt%22%3E%3Cs%3E1501806331328%3C%2Fs%3E%3C
%2Fk%3E%3C%2Fm%3E%3C%2Fk%3E%3Ck+v%3D%22prm%22%3E%3Cm%3E%3Ck+v%3D%22cst%22%3E%3Cs%3E1501806333204%3C%2Fs%3E%3C%2Fk%3E%3Ck+v%3D%22eif%22%3E
%3Cm%3E%3Ck+v%3D%22ety%22%3E%3Cs%3EonStartupHandler%3C%2Fs%3E%3C%2Fk%3E%3Ck+v%3D%22cld%22%3E%3Cs%3Ed1%3C%2Fs%3E%3C%2Fk%3E%3Ck+v%3D%22cty
%22%3E%3Cs%3Eoracle.adf.RichDocument%3C%2Fs%3E%3C%2Fk%3E%3C%2Fm%3E%3C%2Fk%3E%3C%2Fm%3E%3C%2Fk%3E%3C%2Fm%3E&event.d1=%3Cm+xmlns%3D%22http%3A%2F%2Foracle.com%2FrichClient%2Fcomm%22%3E%3Ck+v%3D%22_custom%22%3E%3Cb%3E1%3C%2Fb%3E%3C%2Fk%3E%3Ck+v%3D%22type%22%3E
%3Cs%3EonStartupHandler%3C%2Fs%3E%3C%2Fk%3E%3C%2Fm%3E&oracle.adf.view.rich.PROCESS=d1

POST http://server:port/epm/faces/oracle/epm/web/main/ui/page/index.jspx?_adf.ctrl-state=u7ry1kkia_9 HTTP/1.1
Accept: */*
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate
Content-Type: application/x-www-form-urlencoded; charset=UTF-8
Adf-Ads-Page-Id: 1
Adf-Rich-Message: true
Referer: http://server:port/epm/faces/oracle/epm/web/main/ui/page/index.jspx?_afrLoop=422167606067871&_adf.ctrl-state=u7ry1kkia_9
Content-Length: 1543
Cookie: ORA_FEPM_AUTH=true; oracle.uix=0^^GMT-7:00^p; JSESSIONID=1_aqogMT4SvHbtYOBw_Vo5hYEBXvkf9dURxzRaVB7JqS7KuKJxQq!527691855
Connection: keep-alive

epmhr1:0:r9:1:pt1:epmsrct=&org.apache.myfaces.trinidad.faces.FORM=f1&javax.faces.ViewState=!-b3k0gj3gb&
oracle.adf.view.rich.DELTAS=%7Bepmhr1%3A0%3Ar9%3A1%3Apt1%3Aepm_led%3D%7BviewportSize%3D12%7D%2Cepmhr1%3A0%3Ar9%3A1%3Apt1%3Aepm_lec
%3D%7BviewportSize%3D10%7D%7D&event=epmhr1%3A0%3Ar9%3A1%3Apt1%3Aepm_lec%3A0%3Acl1j_id_3&
oracle.adf.view.rich.monitoring.UserActivityInfo=%3Cm+xmlns%3D%22http%3A%2F%2Foracle.com%2FrichClient%2Fcomm%22%3E%3Ck+v%3D%22pr0%22%3E
%3Cm%3E%3Ck+v%3D%22cid%22%3E%3Cs%3E905b879470a964e9%3A-38d161bb%3A15da521ad99%3A-8000-0000000000197ad8%3C%2Fs%3E%3C%2Fk%3E%3Ck+v%3D%2
2cst%22%3E%3Cs%3E1501806631340%3C%2Fs%3E%3C%2Fk%3E%3Ck+v%3D%22cet%22%3E%3Cs%3E1501806632071%3C%2Fs%3E%3C%2Fk%3E%3Ck+v%3D%22rrt%22%3E
%3Cs%3E1501806631904%3C%2Fs%3E%3C%2Fk%3E%3C%2Fm%3E%3C%2Fk%3E%3Ck+v%3D%22prm%22%3E%3Cm%3E%3Ck+v%3D%22cst%22%3E%3Cs%3E1501806669571%3C%2Fs
%3E%3C%2Fk%3E%3Ck+v%3D%22eif%22%3E%3Cm%3E%3Ck+v%3D%22ety%22%3E%3Cs%3Eaction%3C%2Fs%3E%3C%2Fk%3E%3Ck+v%3D%22cld%22%3E
%3Cs%3Eepmhr1%3A0%3Ar9%3A1%3Apt1%3Aepm_lec%3A0%3Acl1j_id_3%3C%2Fs%3E%3C%2Fk%3E%3Ck+v%3D%22cty%22%3E%3Cs%3Eoracle.adf.RichCommandImageLink
%3C%2Fs%3E%3C%2Fk%3E%3Ck+v%3D%22cdn%22%3E%3Cs%3ERP_user0001_00001%3C%2Fs%3E%3C%2Fk%3E%3Ck+v%3D%22rvd%22%3E%3Cs%3E
%2FlibraryBaseTF%2FlibraryBase%3C%2Fs%3E%3C%2Fk%3E%3C%2Fm%3E%3C%2Fk%3E%3C%2Fm%3E%3C%2Fk%3E
%3C%2Fm%3E&event.epmhr1:0:r9:1:pt1:epm_lec:0:cl1j_id_3=%3Cm+xmlns%3D%22http%3A%2F%2Foracle.com%2FrichClient%2Fcomm%22%3E%3Ck+v%3D%22type
%22%3E%3Cs%3Eaction%3C%2Fs%3E%3C%2Fk%3E%3C%2Fm%3E&oracle.adf.view.rich.PROCESS=epmhr1%3A0%3Ar9

Yes, it is fragile. Sometimes I can track the change and fix a script, sometimes I need to re-create all six scripts from a scratch. As it uses a framework, developers don’t actually know how their change impact protocol level. With a good tool and my set of correlation rules I may recreate them in a couple of days. Not a fun for sure, but works. May I do it with a tool without good reliable recording and correlation rules functionality? Don’t want even think about it.

Can load generation done with that framework API? Well, perhaps. You develop a kind of a shadow system with almost all security and functionality built-in. Maybe some developers may go this way – which may have some advantages and some shortcomings – when it is their only main system. Not exactly my case here. And we are discussing load testing tools – which are created to test systems without major development efforts. And in many cases (at least in corporate IT) you may have no access to developers at all.

I don’t quite understand the notion “Record & playback only works best if what you replay is truly static in that you don’t need to change”. It works best in most cases (while, of course, there are exceptions) – and it is the reason while it is the main approach for testing non-trivial systems. We don’t speak here about beginners’ errors – like replaying without correlation or relying on automatic correlation only. Maybe we have a didactic aspect here: how you want to teach people to show what is behind the scene. I may see a point here. But assuming that you know what you are doing – what may be a drawback of recording? Even if you have good REST APIs – what may be wrong in seeing what are the actual sequence of requests and actual parameters used instead of trying to figure out them yourself from a scratch? Using a separate tool for recording (like Fiddler) is exactly the same except the lack of integration and need to copy it manually between the tools.

So I strongly believe that load testing tools must have powerful recording and correlation. Well, at least if they want to get into corporate market beyond some niche usage. Yes, we have some niches for load testing tools without recording. As well as there are ways to create load without using load testing tools at all – for example, by creating your own framework or re-directing real traffic. But not having recording heavily limits tool’s usage to simpler cases and make it practically useless for corporate IT.

Share

Leave a Reply

Your email address will not be published. Required fields are marked *