User Concurrency

Performance testing terminology is not well defined and one of the most ambiguous terms is user concurrency. Re-reading Load Testing: Concurrent Users verses Simultaneous Users by Scott Moore (@loadtester) and LoadRunner Concurrency video by Mark Tomlinson (@mtomlins) inspired me to post this comment. Here is what I wrote (and still believe in it) in my old CMG paper about performance requirements in 2007 ( the latest version of this paper was presented again at CMG’12):

Concurrency is the number of simultaneous users or threads. It is important too: connected, but inactive users still hold some resources. For example, the requirement may be to support up to 300 active users.

When we speak about the number of users, the terminology is somewhat vague. Usually three metrics are used:

• Total or named users: all registered or potential users. That is a metric of data the system works with. It also indicates the upper potential limit of concurrency.

• Active or concurrent users: users logged in at a specific moment of time. That one is the real measure of concurrency in the sense it is used here.

• Really concurrent: users actually running requests at the same time. While that metric looks appealing and is used quite often, it is almost impossible to measure and rather confusing: the number of “really concurrent” requests depends on the processing time for this request. For example, let’s assume that we got a requirement to support up to 20 “concurrent” users. If one request takes 10 sec, 20 “concurrent” requests mean throughput of 120 requests per minute. But here we get an absurd situation that if we improve processing time from 10 to 1 second and keep the same throughput, we miss our requirement because we have only 2 “concurrent” users. To support 20 “concurrent” users with 1 second response time we really need to increase throughput 10 times to 1,200 requests per minute.

It is important to understand what users you are speaking about: the difference between each of these three metrics for some systems may be drastic. Of course, it heavily depends on the nature of the system.

The number of online users (the number of parallel session) looks like the best metric for concurrency (complementing throughput and response time requirements).

To summarize my comment, I believe that the number of “really concurrent” users is not an appropriate input metric for performance engineering and performance testing. It perhaps may be an output metric characterizing system’s load if we find a way to measure it (in a way, it is the number of users in the system if we use queuing theory terminology).

5 Responses to "User Concurrency"

Trevor Warren says:

December 24, 2013 at 2:29 am

Alex,

Let me thank you for mentioning the “Performance Requirements” paper. It’s a really well written doco. I’ll be writing a piece about your work and link to it if you don’t mind.

Coming back to the piece, yes I do appreciate your point of view and agree that there is generally lack of agreement industry wide on the definition of most performance related quantities including specifically “User concurrency”. It’s one of the reasons why the journey is so essential when a Performance Architect starts on a new program to ensure that everyone is on the same page.

I also agree that User Concurrency isn’t a measure of system performance. It’s rather one of the quantities that goes towards defining your workload models and the overall system Non Functional Requirements.

Everyone seems to have their own definitions of User concurrency and i completely understand where you are coming from. To avoid all ambiguity i define user concurrency of all types within the confines of what can be modeled using Little’s Law. That makes sure that we’ve analytically accounted for all the users in a sense that’s realistic (assuming the think times, response time assumptions are sensible) and achievable.

The article was less about the technicality but rather stressing on the fact that while tech jargon is important, if you don’t take the program on the journey and get them on the same page your work is never going to be read in the right light and measure.

Appreciate the response. Great job, i like your theme and your website.

Cheers,
Trevor
Practical Performance Analyst
Neil Gunther says:

January 17, 2014 at 1:03 pm

I completely agree that all performance terms/metrics need to be clearly defined. But it seems to me you haven’t done that consistently in your own examples. Let’s look more closely.

Your example test “requirement (is) to support up to 20 ‘concurrent’ users.” In other words, the *independent variable* is the user load (N) and the test is to determine what thruput (X), on average, will be sustained under that load. Here, X(N) is the *dependent variable* to be measured as an output or response of the system (test rig) to the applied load (N). Put another way, the user load is the *stimulus* and the thruput is the system *response*.

Now, you inform us that each “request takes 10 seconds” or 1/6 minutes (note the change of timebase). But takes 10 seconds where? You need to clearly define this time also. There are many possible times that can be defined in performance analysis. Which one is this?

Since you conclude that the thruput X = 120 RPM, it is *not* the “processing time” to service the request. It must be the TOTAL TIME the request spends in the system (let’s call it, R). Check using Little’s law: X = N/R = 20/(1/6) = 20 * 6 = 120 RPM.

Next, you say “we get an absurd situation that if we improve processing time from 10 to 1 second and keep the same
throughput, we miss our requirement because we have only 2 ‘concurrent’ users.”

I disagree with this conclusion. If I improve the performance of the system (i.e., reduce R by 10x to 1/60 mins.), I fully expect it to respond with higher thruput (X = 20 * 6 = 120 RPM) under the same load (N = 20). If it doesn’t (during the actual test), either we’ve hit a previously hidden limit of the system (bottleneck) or something has gone wrong with the test (happens all the time). Guerrilla mantra 2.13: If the data don’t agree with the model (Little’s law), change the data.

It only *seems* absurd b/c you switched gears and specified a DIFFERENT test requirement, i.e., “keep the same throughput”. Naturally, if performance has been improved, I then expect to apply less load to get the same thruput of 120 RPM. Check using Little’s law: N = X * R = 120 * 1/60 = 2 users.

Either test is completely legitimate, but they represent two different test requirements. Same comment applies to your CMG’12 paper, where the 60x factor simply corresponds to two *different* tests.
Neil Gunther says:

January 17, 2014 at 1:07 pm

Arrrgh! That should read “higher thruput (X = 20 * 60 = 1200 RPM) under the same load (N = 20).”
Alex Podelko says:

January 20, 2014 at 11:06 pm

Neil,

I probably was not clear: I meant unrestricted system with constant arrival rate. So when we improve processing time, we still have the same throughput just because we don’t have more requests to process (as we had processed all requests anyway).

As the system is unrestricted and we look for a steady mode, arrival rate and throughput would be the same.

My point was that real business requirements are throughput (equal here to request arrival rate as we probably want the system to be able to process all arriving requests in a steady mode) and response time (which indeed would be the sum of processing time and request waiting time). We can present throughput (==arriving rate here) as the number of users * the number of transaction each users generates (sometimes indirectly defined by think time), but we can’t present it with “really concurrent” users as it is illustrated by the example. The number of “really concurrent” users is the number of requests in the system – it is a metric of the system, but not a business requirements.

Thanks,
Alex
Kelvin Gething says:

May 7, 2016 at 10:42 am

Alex
I have been through the concurrency requirements questions during performance test planning many times. I agree with your definitions. I personally use the term “concurrent user sessions”. I find this has meaning to both technical and non-technical stakeholders.
Kelvin

Leave a Reply