Dan said: There are two dangerous practices that are rampant in performance testing right now – shrinking or removing think times and extrapolation of test results.
While I agree with all Dan’s statements in general, I’d rather put another spin here. It is cheating when you use these two valid and sound techniques irresponsible / instead of full-scale tests. But if you understand what you are doing and use these techniques intelligently, both are very powerful tools.
For the shrinking and removing think time, I’d suggest to read Rapid Bottleneck Identification – A Better Way to do Load Testing, which shows how to use this approach for rapid bottleneck identification, and some related discussion here and here. I believe that it may be a very valuable technique for performance troubleshooting in addition to full-scale tests (and in some cases it is better than nothing if full-scale tests are not possible for whatever reasons).
By the way, I rather think that shrinking and removing think time put lower load on the system because it probably would be some resources associated with every connection – and we had significantly fewer connections. Together with the possible consequences Dan mentioned, you may have quite complicated implications here, so watch your step.
For extrapolation, Dan said: Just as dangerous as shrinking your think times down is trying to make guesses about how many users you can support in a production environment when you tested at 1/10th the anticipated load in a scaled down lab. This is like saying that because you can run a mile right now, you can run a marathon with no problems.
Extrapolation, of course, should be used carefully and has risk associated with it. But if we develop the idiom with a runner further, it is irresponsible to say that you run marathon if you run one mile. But if you run one mile, you may be pretty sure that you run a half of a mile. Saying that you would run two miles has some risk associated with it, but if you know that you are still fresh at the end of the first mile (have a lot of resources available) it is still a reasonable statement. And running a mile is definitely better that just laying on sofa. So if you can’t do full-scale tests, do tests as close to full-scale as you can – and using intelligent extrapolation / modeling may help here. Anyway, the whole capacity planning discipline is based on extrapolation. But if you can run a full-scale test, but are instead running only a small test and extrapolate – this would be definitely cheating.