Alex Podelko

Dan Bartow wrote the Stop Cheating in Your Performance Tests article at STP.

Dan said: There are two dangerous practices that are rampant in performance testing right now – shrinking or removing think times and extrapolation of test results.

While I agree with all Dan’s statements in general, I’d rather put another spin here. It is cheating when you use these two valid and sound techniques irresponsible / instead of full-scale tests. But if you understand what you are doing and use these techniques intelligently, both are very powerful tools.

For the shrinking and removing think time, I’d suggest to read Rapid Bottleneck Identification – A Better Way to do Load Testing, which shows how to use this approach for rapid bottleneck identification, and some related discussion here and here. I believe that it may be a very valuable technique for performance troubleshooting in addition to full-scale tests (and in some cases it is better than nothing if full-scale tests are not possible for whatever reasons).

By the way, I rather think that shrinking and removing think time put lower load on the system because it probably would be some resources associated with every connection – and we had significantly fewer connections. Together with the possible consequences Dan mentioned, you may have quite complicated implications here, so watch your step.

For extrapolation, Dan said: Just as dangerous as shrinking your think times down is trying to make guesses about how many users you can support in a production environment when you tested at 1/10th the anticipated load in a scaled down lab. This is like saying that because you can run a mile right now, you can run a marathon with no problems.

Extrapolation, of course, should be used carefully and has risk associated with it. But if we develop the idiom with a runner further, it is irresponsible to say that you run marathon if you run one mile. But if you run one mile, you may be pretty sure that you run a half of a mile. Saying that you would run two miles has some risk associated with it, but if you know that you are still fresh at the end of the first mile (have a lot of resources available) it is still a reasonable statement. And running a mile is definitely better that just laying on sofa. So if you can’t do full-scale tests, do tests as close to full-scale as you can – and using intelligent extrapolation / modeling may help here. Anyway, the whole capacity planning discipline is based on extrapolation. But if you can run a full-scale test, but are instead running only a small test and extrapolate – this would be definitely cheating.

Oracle Application Testing Suite 9.20 was released (former Empirix e-TEST Suite). It is available for download (subject to OTN License Agreement). New features and updates in this Release from Readme:

I guess the following note from Readme is important for those who is using the product for a long time:

The legacy Oracle Functional Testing, VB-based scripting platform and associated components (such as Job Scheduler) are no longer included with the ATS installer starting with version 9.2. Customers that still require this product should use ATS 9.1 or older versions supported under Oracle’s Lifetime Support Policy.

Hyperformix was acquired by CA. Hyperformix was one of the leading companies providing capacity planning/management solutions based on mathematical modeling. They probably had the best product for Software Performance Engineering. They probably did the best job integrating modeling/capacity planning with load testing (and LoadRunner in particular). Once upon a time Hyperformix Performance Optimizer was also marketed as Mercury Capacity Planning.

I believe that this is the first acquisition in this area since BGS was acquired by BMC in 1998 with then the leading BEST/1 product (now, I guess, BMC Capacity Management).

It is interesting that modeling (capacity management based on the true modeling) remained an area with small player for a long time. In addition to Hyperformix and BMC, it is possible to mention TeamQuest, Metron, OPNET, and BEZ Systems. It is especially interesting because large players (like IBM, Oracle, HP, and Microsoft) don’t have any real modeling solutions in their portfolio (some products may have trending/forecasting features, but I haven’t heard about anything more sophisticated). But with high-tech buying spree around, this area surprisingly remained untouched.

I was also surprised that it was CA who acquired Hyperformix. As far as I understand, CA is not known for investing too much in the acquired companies. Looks like they put more stress on capacity management. So I am afraid that most interesting features of Hyperformix (in my opinion, of course) like using for new system modeling (Software Performance Engineering) and integration with load testing may be lost.