Performance Testing: The Art of Scripting
Performance testing is an interesting discipline: while you may see a lot of people doing it, not much is written about it. And what is written is often either introduction-level or tool-specific (other large groups of topics including performance analysis and performance troubleshooting are wider topics and not testing-specific). But not much about performance testing specific challenges beyond of what is covered by tool manuals.
My group specializes in performance testing of Hyperion / Enterprise Performance Management and Business Intelligence products at Oracle. A few of scripting challenges exist for almost every product. Nothing exceptional – you should resolve them if you have some experience and would be attentive enough, but time after time we are called to save performance testing projects to find out that there are serious problems with scripts and scenarios which make test results meaningless. Even very experienced testers stumble, but problems could be avoided if more time was spent analyzing what is going on.
Let’s consider an example – probably typical for challenges you can face with modern Web-based applications. Some operations, like financial consolidation, can take a long time. The client starts the operation on the server and then waits until it will finish, a progress bar is shown in meanwhile. When recorded, the script looks like (in LoadRunner pseudo-code):
web_custom_request(“XMLDataGrid.asp_7”,
“URL={URL}/Data/XMLDataGrid.asp?Action=EXECUTE&TaskID=1024&RowStart=1&ColStart=2&RowEnd=1&ColEnd=2&SelType=0&Format=JavaScript”,
LAST);
web_custom_request(“XMLDataGrid.asp_8”,
“URL={URL}/Data/XMLDataGrid.asp?Action=GETCONSOLSTATUS”,
LAST);
web_custom_request(“XMLDataGrid.asp_9”,
“URL={URL}/Data/XMLDataGrid.asp?Action=GETCONSOLSTATUS”,
LAST);
web_custom_request(“XMLDataGrid.asp_10”,
“URL={URL}/Data/XMLDataGrid.asp?Action=GETCONSOLSTATUS”,
LAST);
What each request is doing is defined by the ?Action= part. The number of GETCONSOLSTATUS requests recorded depends on the processing time. In the example above it was recorded three times; it means that the consolidation was done by the moment the third GETCONSOLSTATUS request was sent to the server. If playback this script, it will work in the following way: the script submits the consolidation in the EXECUTE request and then calls GETCONSOLSTATUS three times. If we have a timer around these requests, the response time will be almost instantaneous. While in reality the consolidation may take many minutes or even an hour (yes, this is a good example when sometimes people may be happy having one hour response time in a Web application). If we have several iterations in the script, we will submit several consolidations, which continue to work in background competing for the same data, while we report sub-second response times.
Consolidation scripts require creating an explicit loop around GETCONSOLSTATUS to catch the end of consolidation:
web_custom_request(“XMLDataGrid.asp_7”,
“URL={URL}/Data/XMLDataGrid.asp?Action=EXECUTE&TaskID=1024&RowStart=1&ColStart=2&RowEnd=1&ColEnd=2&SelType=0&Format=JavaScript”,
LAST);
do {
sleep(3000);
web_reg_find(“Text=1″,”SaveCount=abc_count”,LAST);
web_custom_request(“XMLDataGrid.asp_8”,
“URL={URL}/Data/XMLDataGrid.asp?Action=GETCONSOLSTATUS”,
LAST);
} while (strcmp(lr_eval_string(“{abc_count}”),”1″)==0);
Here the loop simulates the internal logic of the system sending GETCONSOLSTATUS requests each 3 sec until the consolidation is done. Without such loop the script just send requests and ends the iteration while the consolidation runs for a long time after that.
Some other examples can be found in my presentation A Load Testing Story at CMG’10 – a story about one my project which had a good share of testing challenges.
Many technical details about testing Hyperion products with LoadRunner are included in Oracle support document 1339743.1. A lot of this knowledge is now included in the Hyperion module of Oracle Application Testing Suite. But that is just about one family of products – which is a small sample of challenges you would see around.
According to my experience, creating a meaningful and realistic workload is the main performance testing challenge – and you probably face it in most non-trivial systems (including most of corporate applications). It is surprising that very few people talk about it: reading about performance testing you may get impression that the scripting is trivial and mundane (and it looks like many “performance testers” truly believe in it – and when they face any challenge, they start to talk about the death of performance testing).
Only few names come to my mind who write about specific challeneges. For example, see Northway (@northwaysg) / Scott Moore (@loadtester) performance testing blog and white papers or Mentora (@MentoraGroup ) / Dan Downing presentations and white papers. Maybe few more – but definitely they are few and far apart.
