When extrapolation is cheating

Dan Bartow wrote the Stop Cheating in Your Performance Tests article at STP.

Dan said: There are two dangerous practices that are rampant in performance testing right now – shrinking or removing think times and extrapolation of test results.

While I agree with all Dan’s statements in general, I’d rather put another spin here. It is cheating when you use these two valid and sound techniques irresponsible / instead of full-scale tests. But if you understand what you are doing and use these techniques intelligently, both are very powerful tools.

For the shrinking and removing think time, I’d suggest to read Rapid Bottleneck Identification – A Better Way to do Load Testing, which shows how to use this approach for rapid bottleneck identification, and some related discussion here and here. I believe that it may be a very valuable technique for performance troubleshooting in addition to full-scale tests (and in some cases it is better than nothing if full-scale tests are not possible for whatever reasons).

By the way, I rather think that shrinking and removing think time put lower load on the system because it probably would be some resources associated with every connection – and we had significantly fewer connections. Together with the possible consequences Dan mentioned, you may have quite complicated implications here, so watch your step.

For extrapolation, Dan said: Just as dangerous as shrinking your think times down is trying to make guesses about how many users you can support in a production environment when you tested at 1/10th the anticipated load in a scaled down lab. This is like saying that because you can run a mile right now, you can run a marathon with no problems.

Extrapolation, of course, should be used carefully and has risk associated with it. But if we develop the idiom with a runner further, it is irresponsible to say that you run marathon if you run one mile. But if you run one mile, you may be pretty sure that you run a half of a mile. Saying that you would run two miles has some risk associated with it, but if you know that you are still fresh at the end of the first mile (have a lot of resources available) it is still a reasonable statement. And running a mile is definitely better that just laying on sofa. So if you can’t do full-scale tests, do tests as close to full-scale as you can – and using intelligent extrapolation / modeling may help here. Anyway, the whole capacity planning discipline is based on extrapolation. But if you can run a full-scale test, but are instead running only a small test and extrapolate – this would be definitely cheating.

Kelvin Gething says:

April 12, 2011 at 4:10 pm

Alex
Your counter argument on shrinking and extrapolation are well stated. I think it ultimately comes down to designing a suitable test to meet the performance test objective(s). Doing the full test – i.e. no extrapolation – gives you a greater confidence in the result. It would be interesting if this type of confidence could be expressed mathematically?

Alex Podelko says:

April 12, 2011 at 6:47 pm

You probably can extrapolate relatively reliable (up to some degree, of course) if the only restriction is resources (for example, CPU available). Unfortunately model (unless you explicitly built it into it) won’t forecast when you run out of threads, handles, locks, and whatever else you usually have no idea about. So there always is a risk. However even full-scale performance testing is not a full guarantee either: it is as accurate as you simulate real workload. It is still quite possible that something in production you never thought about will kill the system. So you always have some risk. Just the more you test / analyze / investigate, the smaller it is.

Leave a Reply