Alex Podelko

Performance testing terminology is not well defined and one of the most ambiguous terms is user concurrency. Re-reading Load Testing: Concurrent Users verses Simultaneous Users by Scott Moore (@loadtester) and LoadRunner Concurrency video by Mark Tomlinson (@mtomlins) inspired me to post this comment. Here is what I wrote (and still believe in it) in my old CMG paper about performance requirements in 2007 ( the latest version of this paper was presented again at CMG’12):

Concurrency is the number of simultaneous users or threads. It is important too: connected, but inactive users still hold some resources. For example, the requirement may be to support up to 300 active users.

The number of online users (the number of parallel session) looks like the best metric for concurrency (complementing throughput and response time requirements).

To summarize my comment, I believe that the number of “really concurrent” users is not an appropriate input metric for performance engineering and performance testing. It perhaps may be an output metric characterizing system’s load if we find a way to measure it (in a way, it is the number of users in the system if we use queuing theory terminology).

After attending Sergey Chernyshev’s (@sergeyche) Scalability vs. Performance presentation at NY Web Performance Meetup and reading Scalability: it’s the question that drives us by Robert David Graham (@ErrataRob) and Scalability vs. Performance: it isn’t a battle by Theo Schlossnagle (@postwait) I would like to share my understanding. While I agree in general with everything said, I would rather word it differently. The topic became loaded, so accents are important. Robert, for example, states that “performance” and “scalability” are orthogonal problems. Well, no, they are not. They are different, but correlated notions. Even leaving aside that performance and scalability are somewhat vague terms.

If we speak about web systems now, it looks like we can roughly separate two main components in response time (which is the main performance metric): backend (server-side) time and frontend (network and client-side time). There are subtleties and grey areas, but I’d ignore them here. The frontend time, the subject of Web Performance Optimization (WPO), doesn’t relate to scalability as far as it is not involving server processing (again ignoring subtleties).

The proportion of frontend time vs. backend time may be any at all. According to Steve Souders (@souders), the founder of WPO, 80-90% of the end-user response time is spent on the frontend. But even for major web sites the backend time for requests involving database processing (such as submitting orders or querying order status) may be more noticeable. And there are plenty of corporate web applications where the share of frontend time is rather small. Of course, the starting point of any performance troubleshooting is to find where time is spent. And there is absolutely no sense to optimize parts where time is not spent.

However, there is one important “but”. While front-end time supposed to be constant (another simplification, but again ignoring subtleties), the backend (server) time depends on load. The heavier is load, the larger may be server time. And at some point it may skyrocket making the system practically unusable. So thinking about what you need to optimize, you need to check where time is spend under maximal load. Unless you don’t care about downtime and user experience, the way to do it is load testing.

And here we get to scalability. The frontend performance indeed doesn’t matter here and is independent of scalability. But the backend performance is directly related to scalability. The relationship, of course, may be non-linear and quite sophisticated – but it does exist.

To illustrate it, let’s consider one simple (but still typical) example. The backend processing takes X ms, the time is mainly spent in CPU and we don’t have any other bottlenecks. In this case the server response time would be mainly CPU processing time – and every request would take X ms of CPU time (if we don’t have parallelism here). As soon as we take most of available CPU time, server response time would skyrocket (that situation may be modeled using queuing theory). So there is a load when the system becomes practically unusable – and the question is just when we get to this load. We, of course, may get to problems sooner if we have any scalability problem inside the system or run out of another kind of resources.

Generally speaking, you can increase scalability by either optimizing server processing (using less resources) or providing more resources. Of course, if your architecture allows using these additional resources – so mainly scalability boils down to ability to parallelize your processing (and often limited by what you can’t fully parallelize – like a centralized database).

We do have two parts of response time – frontend and backend – which behaves differently and may need different approaches and tools to optimize. But the end user experience is the sum of these two parts – where the backend time is a function of load. You can’t say much about your end-to-end performance and its backend part until you check it under load – and load testing is the safe way to do so.

Historically performance engineering concentrated on backend – where main performance and scalability issues were – and practically ignored frontend (which indeed was usually pretty straightforward then). Several sub-disciplines were formed including performance analysis, capacity planning, and load testing. Later, when sophistication of frontend skyrocketed, a whole new discipline was established by Steve Souders and quickly grew around Velocity and Web Performance meetups. Unfortunately, it practically dismissed performance engineering developments of the last 40 years (maybe even more – the Computer Measurement Group (CMG) was founded in 1975). While frontend WPO definitely has its own specific, I’d still expect to see a holistic approach to performance engineering, taking in account all aspects of performance and scalability end-to-end.

It is interesting to look how handling performance changed with time. Probably performance went beyond single-user profiling when mainframes started to support multiprogramming. It was mainly batch loads with sophisticated ways to schedule and ration consumed resources as well as pretty powerful OS-level instrumentation allowing to track down performance issues. The cost of mainframe resources was high, so there were capacity planners and performance analysts to optimize mainframe usage.

Then the paradigm changed to client-server and distributed systems. Available operating systems didn’t have almost any instrumentation and workload management capabilities, so load testing became almost only remedy in addition to system-level monitoring to handle multi-user performance. Deploying across multiple machines was more difficult and the cost of rollback was significant, especially for Commercial Of-The-Shelf (COTS) software which may be deployed by thousands of customers. Load testing became probably the main way to ensure performance of distributed systems and performance testing groups became the centers of performance-related activities in many organizations.

While cloud looks quite different from mainframes, there are many similarities between them, especially from the performance point of view. Such as availability of computer resources to be allocated, an easy way to evaluate the cost associated with these resources and implement chargeback, isolation of systems inside a larger pool of resources, easier ways to deploy a system and pull it back if needed without impacting other systems.

However there are notable differences and they make managing performance in cloud more challenging. First of all, there is no instrumentation on the OS level and even resource monitoring becomes less reliable. So all instrumentation should be on the application level. Second, systems are not completely isolated from the performance point of view and they could impact each other. And, of course, we mostly have multi-user interactive workloads which are difficult to predict and manage. That means that such performance risk mitigation approaches as APM, load testing, and capacity management are very important in cloud.

It is interesting that while performance is the result of all design and implementation details, performance engineering area remains very siloed. Those who do capacity planning are usually not involved much in performance testing or software performance engineering. The new and fastest growing group, web performance specialists, remains mainly isolated from other performance-related groups. People and organizations trying to span all performance-related activities together are few and far apart.

I don’t see that the need that need for specific performance-related expertise, such as load testing or capacity planning, is going away. Even in case of web operations, we would probably see load testing coming back as soon as systems become more complex and performance issues start to hurt business. There perhaps would be less need for “performance testers” as it was at the heyday due to better instrumenting, APM tools, continuous integration, resource availability, etc. – but I’d expect more need for performance experts who would be able to see the whole picture using all available tools and techniques.

CMG’12 is an annual conference organized by Computer Management Group – a volunteer organization of professionals specialized in performance, capacity, and IT service management. This year it is held in Las Vegas, December 2-7, 2012.

Why I love CMG, spend a lot of my time organizing and promoting it, and coming there every year (sometimes on my own)? Well, because I believe that it is the best (and actually the only) conference on performance and capacity, the main topic of my interest for the last fifteen years. There are many conferences on specific topics. For example, the Velocity conference, devoted to web performance, is significantly larger and more popular – but it is still devoted mainly to single-use web performance, leaving all other performance and capacity questions to CMG. Let me share some of my excitement – of course, from my personal point of view (there is plenty of other highlights, but I am mentioning only the ones that are close to my heart).

This year the conference covers all aspects of performance (well, almost all – performance is so sophisticated subject that there is always much more to learn) from Web Performance Optimization (the conference opens by the keynote by Patrick Meenan, a web performance Google guru and the creator of WebPagetest) to mainframe performance (and everything in between).

The conference starts with a half-day workshops – see here the description. In addition to workshops, there are CMG-T sessions during the whole conference. Each CMG-T class spans 2 or 3 session spots, so it could easily be considered as a workshop or a training class. All led by renown experts with tons of experience, you hardly would get anybody even remotely close if you engage in a typical vendor class (not to mention a unique vendor-neutral or vendor-agnostic perspective you hardly find anywhere else). You have the CMG-T track through the whole conference and every one of them is a gem:

• Capacity Planning by Ray Wicks
• z/OS Basics by Glenn Anderson
• Java Performance Analysis and Tuning by Peter Johnson
• Model and Forecasting Basics by Dr. Michael Salsburg
• Network Performance Management by Manoj Nambiar
• Windows System Performance Management and Analysis by Jeffry Schwartz
• Using SAS to Communicate Your Message by MP Welch

CMG’12 has 4 keynote/plenary session and almost a hundred regular track sessions going on from mid-Monday to mid-Friday. The conference is 5 tracks wide. One track, as I already mentioned, is CMG-T 101– type classes (with 301-depth). Others four tracks shared between five subject areas: Performance Engineering and Testing, Capacity Planning, Application Performance Management, IT Service Management, and Hot Topics. It is difficult to list all highlights – too many. While I know many great presenters and am fascinated by many topics, commenting every single one would take too much time and space. Probably you just need to look at agenda – there are three different views: preliminary agenda (overview, a day on a page), a list of abstracts in a single pdf document and search/scheduler (click on the abstract number to see the abstract).

One track on Wednesday is a Michelson award track. CMG is presenting Michelson award since 1974 (if you wonder, Albert Abraham Michelson was known for his technical accomplishments in measuring the speed of light and for his role as teacher and inspirer of others – and measuring is the key to performance). This year we will see many Michelson winners presenting: Dr. Connie Smith, the founder of Software Performance Engineering, Dr. Daniel Menasce, the author of many great books about performance and capacity planning, Adam Grummit, the author of the great Capacity Management book (ITSM Library) and the CMG president, Dr. Pat Artis, Bruce McNutt, and Dr. Michael Salsburg.

I believe that the main advantage of attending CMG is networking with best world experts in almost all areas of performance and capacity. Nowadays you can find all technical information on the Internet, but there is no substitution to face-to-face conferences to learn how to use it and what were people experiences, and, of course, to see the whole picture. Especially in performance: performance is the result of every design and implementation detail and you need to be learning all the time to keep up with coming challenges.

I am presenting there too: Load Testing: See a Bigger Picture on Thursday and
Performance Requirements: the Backbone of the Performance Engineering Process on Friday. Nothing comparing to other CMG’12 highlights, but I hope to trigger discussions around these two very important topics.

And, of course, it is Las Vegas – and Rio’s rate is $55 per night until November 14th. See you there!

Reading about performance modeling / simulation and system sizing, you often see two completely opposite views of the subject. Either authors describe in detail how you can model performance using some math and you may feel that as soon as you comprehend that math, you won’t have any problem with modeling. Or authors say that it is a black magic and you’d better stay away from it or do it in a minimal way with simple trending (while you probably won’t see that view in serious books, it is often can be seen in Internet discussions).

The truth, as usual, is in the middle. Modeling is a very helpful and works well if you use it properly and understand its limitations. And there are two main challenges here that rarely get highlighted – while everybody who wants to approach the topic should understand them clearly.

The first challenge is that modeling works well for known resource limitations. You should know these limitations in advance (and how your system uses that limited resource – which is also a challenge, but more technical one). For example, if your system is processor-bound and you know how much cpu it takes per transaction, you may build a rather simple and pretty accurate model (using queuing theory or even something pretty simple – for example, if you stay away from heavy cpu utilization, linear model may work well with multi-processor systems).

But that model would never tell you when you run out of another resource and run into another kind of bottleneck until you build it into the model. And beyond a few common resources (processor, memory, disk, network) and explicitly introduced throttling, you usually don’t know about bottlenecks until you run into them. This is the primary reason that results of your model (which may be perfect from the mathematical point of view) are not reliable if you model significantly higher load than you tested / validated – as far as there is a high probability that you run into another bottleneck you are not taking in consideration now. However, the model would provide the best possible case (which turns true when you fix all other bottlenecks you didn’t take in consideration at the moment of modeling), which is important information by itself. A model would also be very useful to see if the system behaves up to expectations – or there are internal issues degrading performance and preventing scalability (that may be not so trivial to catch in complex systems).

Another challenge is a lack of performance-related metrics of hardware to use in modeling. You can find detailed hardware specifications, but they won’t tell you how fast your systems would work on this hardware. As far as I understand, the only relatively objective approach (without testing the real system on the real hardware – which is, of course, the best) is to use existing benchmark results to compare performance (keeping in mind that they represent results of this specific benchmark, not your systems). Most serious commercial modeling tools come with a library of hardware configurations and their performance metrics, allowing what-if performance analysis. It looks like keeping such libraries is a pretty time-consuming task and their quality may differ. Such a library is usually a major advantage of commercial modeling tools in comparison with free or inexpensive modeling tools (which may be quite good from the mathematical point of view, but you need to provide all numbers yourself).

IDC made an interesting move here introducing QPI (Qualified Performance Indicator) as a part of IDC’s Server Decision Suite Metrics (free 30-days trial available). A kind of independent performance library that may be used for proper performance modeling / sizing (and, as far as I understand, going well beyond performance, integrating this information with other IT-related metrics such as price, power, and size – it should be a very interesting optimization task to find the best hardware configuration based on all these metrics).

The Computer Measurement Group (CMG) calls for papers and presentations for CMG’s 38th International Conference to be held in Las Vegas, Nevada, December 3rd through 7th, 2012.

The 2012 CMG conference will cover all areas of systems management, including but not limited to: capacity planning, IT service management, application performance management, performance engineering and testing, as well as the latest developments in the overall field of computer performance evaluation. See the Call for Papers and Call for Workshops for details.

CMG is the source of unbiased and objective expert information and practical, real life experiences across all computing platforms in the computer industry for over 30 years. Share your knowledge and experiences: write a paper and submit it for presentation at CMG’12.

Paper are categorized as Introductory, Tutorial, Advanced, or User Experience. I want to especially encourage all of you to consider writing a User Experience paper. Every year, the conference evaluations show a common theme: “More User Experience Papers, please!” You don’t need to be one of the field’s superstars to write one — in fact, they seem to work better from people who are just working in the field, in non-IT companies and government bodies. Just tell us what problem you faced, how you went about figuring out what the cause was, and how you dealt with it. Mentors are available for writing assistance, and may be requested at any point in the writing process, including before the paper is started. Just write mentor@cmg.org and ask.

Please take the time to participate in the CMG’12 program. It will be rewarding for both authors and attendees, and as we all share our knowledge we all become more complete professionals.
Paper submission through the CMG website is now available. For more information go to paper submission and workshop submission.

The deadline for paper submissions is June 8, 2012.

Please send questions to CMG’12 Program Chair, Bill Jouris at cmgpc@cmg.org.

Recent trends of agile development, DevOps, Web and Social Media sites somewhat question importance of load testing. Some (not many) openly saying that they don’t need load testing, some still paying lip service to it – but just never get to it. In more traditional corporate world we still see performance testing groups and important systems usually get load tested before deployment.

Let’s first define load testing as far as terminology is rather vague here. I use it here as anything that requires applying multi-user synthetic load – in contrast with single-user performance (which is a subset of performance engineering and may include, for example, profiling or Web Performance Optimization as it is defined now). And I use it here as an umbrella term including all other variations of multi-user testing, such as performance, concurrency, stress, endurance, longevity, scalability, etc. – but you may replace it with any other term if you prefer.

Yes, it looks like some Web and Social Media sites managed to survive without load testing. However, it looks like many such companies match the following profile:
-Business is built around a single Web site, so everybody in the company follows what is going on in production.
-Overall architecture is still clear and relatively simple. Changes (however frequent) are rather minor and evolutional.
-There is decent instrumentation providing performance information.
-There is a possibility to remove changes relatively easy.
-Site downtime/a period of slow performance (until the problem would be noticed and fixed) is not extremely painful or dangerous to the business.

Load testing is a way to mitigate load- and performance-related risks. There are other approaches and techniques that also alleviate some performance risks:
-Good single-user performance engineering practices (single-user requests performance are constantly tracked).
-Good instrumentation/Application Performance Management providing insights in what is going on inside the system.
-[Auto] scalable architecture.
-Continuous integration allowing quickly deploy and remove changes.

Still all of these don’t completely replace load testing, but rather complement it. They definitely decrease performance risk comparing with situation when nothing was done about performance at all until the last moment before rolling out the system in production without any instrumentation at all, but it still leaves risks of crashing and performance degradation under multi-user load. And if the cost of it is high, you should do load testing (what exactly and how is another large topic – there is much more here than the stereotypical waterfall-like last-moment record-and-replay approach).

There is always a risk of crashing or performance issues under heavy load – and the only way to mitigate it is actually test it. Even stellar performance in production and highly scalable architecture don’t guarantee that it won’t crash with a slightly higher load. Truly speaking, even load testing doesn’t completely guarantee it (real-life workload may be different from what you have tested), but it drastically decreases the risk.

Another important value of load testing is making sure that changes don’t degrade multi-user performance. Unfortunately, better single-user performance doesn’t guarantee better multi-user performance. In many cases it improves multi-user performance too, but definitely not always. And the more complex system, the more probable exotic multi-user performance issues no one even thought of. And a way to ensure that you don’t have such issues is load testing.

When you do performance optimization, you need a reproducible way to evaluate the impact of changes on multi-user performance. The impact on multi-user performance probably won’t be proportional to what you see with single-user performance (even if it still would be somewhat correlated). Without multi-user testing the actual effect is difficult to quantify. The same with the issues happening only in specific cases that are difficult to troubleshoot and verify in production – using load testing can significantly simplify the process.

Summarizing, I don’t see that the need in load testing is going away. Even in case of Web and Social Media sites we would probably see load testing coming back as soon as systems become more complex and performance issues start to hurt business. Maybe it would be less need for “performance testers” as it was at the heyday due to better instrumenting, APM tools, continuous integration, etc. – but I’d expect more need for performance experts that would be able to see the whole picture using all available tools and techniques (although I don’t see it yet).

There are no standards on titles and skill sets related to performance dimension of IT. I decided to put together how I understand them (most terms are vague, so it is quite possible that other people understand them differently). Of course, it is a simplification – but the topic is probably too heavy influenced by organization history and politics in every particular organization to be clear cut anyway.

I still think that we can break the whole area into three major categories: design (and development), testing, and production (maybe somewhat matching ITIL terms of Service Design, Service Transition, and Service Operation). The term Performance Engineering may be related to the whole area (or maybe related to the design category – in this case sometimes referred as Software Performance Engineering, SPE).

Performance Design. Talking about the design category (I used the ‘Performance Design’ term to group all performance-related activities during design and development , although it isn’t used this way – probably reflecting that the whole area is not quite existing as a separate discipline), we have specific areas of performance engineering knowledge for each specific technology. Such as Java performance, .Net performance, etc. One relatively new, but large and popular area is Web Performance Optimization, covering end-user Web performance. And, of course, we have Software Performance Engineering (SPE) trying to establish generic approaches – although SPE progress wasn’t too impressive since Dr. Connie Smith published ‘Performance Engineering of Software Systems’ in 1990.

It is definitely supposed to be an important part of the skill set of software architects (on a higher level, SPE, etc.) and software developers (maybe on a lower level, how efficiently design specific component using the chosen technology – but good understanding of high-level performance engineering won’t hurt either).

And while many architects and developers have some understanding of performance, often the main stress is on functionality and deadlines, so performance is left to the very end – where it sometimes may be indeed tuned in (usually when technologies are mature and the team is quite experienced), and sometimes require major changes (and late changes are very expensive).

It looks like the idea to have an explicit person responsible for performance from the beginning (starting from requirements) and working with other architects and developers to build it in makes sense. The title may be ‘performance architect’ or ‘performance champion’. Although such people are rare – rather we could see a proactive person from performance engineering or performance testing groups trying to ask performance questions early.

Performance Testing. Including, of course, all other variations and names, such as load, stress, endurance, etc. testing. ITIL matching term would probably Service Validation and Testing. All ways to apply synthetic load to the system and analyze system’s behavior. In the narrow sense, ‘performance tester’ is responsible for creating and applying such load (test scripting and execution). In a wider sense, it also includes workload characterization (workload modeling), performance analysis and performance troubleshooting – and often such person is referred as ‘performance engineer’. In some cases they are different people: performance tester is responsible for applying the load and performance engineer (maybe performance analyst in this case) is responsible for system analysis and optimization.

I definitely put performance testing in a separate category due to specific set of skills required: workload generation. And, perhaps, techniques to find and fix issues in the system applying an appropriate workload. But definitely not because “testing should go after development before production” as it use to be in the waterfall approach – testing should start as early as possible mostly overlapping with development and may continue in production. Monitoring the system using synthetic workload, for example, I’d rather also put in this testing category – it is actually testing the production system in parallel to production workload.

Performance Management, perhaps, may be a good name for the collection of performance-related activities and skills in production (and around).

It is interesting that ITIL places Capacity Management and Service Level Management processes into Service Design. I see a point here – you definitely need to allocate capacity before deploying the system, and Service Levels should probably come directly from the performance requirements. Still real people working in these areas are usually part of operations. Capacity Planners are responsible for allocating resources, although fewer and fewer people have such title and these responsibilities get spread between other groups (which, unfortunately, often don’t have appropriate skills).

Service Level Management would probably handled by Performance Monitoring (Analysis). ITIL matching term would probably Service Measurement. Title ‘Performance Analysts’ used often in the past – but not very popular anymore. Probably title ‘Performance Engineer’ is more popular now. And, of course, it may be specialized, like Database Monitoring, System Monitoring, Application Server Monitoring. These may be done by respective administrators (DBA, system administrator, etc.).

Application Monitoring – relatively new staff. Usually referred as Application Performance Monitoring. The idea is to measure application-specific metrics (including business-related metric, end-user metrics, etc.) in addition to those system-level metrics that used to be measured earlier. Importance of application monitoring is definitely growing. From one side, system-level metrics becomes less relevant in today’s infrastructure with virtualization, multi-tenancy, cloud, etc. From another side, the system becomes so complicated that trying to figure out what is going on using low-level metrics becomes nightmare. Form the third side, full monitoring from the business point of view becomes a business requirement – and it is where IT can provide unique business advantage.

Probably Application Performance Management (APM) would the right category encompassing most production-related categories such as Performance Monitoring, Capacity Management, Diagnostics (troubleshooting) and Tuning (and Optimization – although this may somewhat get into re-design category). We probably not there yet and Application Performance Management is rather a vague vision than reality. Gartner, for example, stresses that APM is Application Performance Monitoring, not Management. And I am not sure what would be a title of the person doing this. Management is a favorite word for an area of expertise (as in Performance Management or Capacity Management), but Manager (at least in the US) still means a person who manages other people. So the title, I guess, would be the same ubiquitous ‘performance engineer’.

Performance Troubleshooting or Diagnostics is definitely important part of Performance Management and is an application of performance engineering to existing performance issues. While it is probably the most typical performance-related activity at many corporations, very few have anything formal around it and usually all other performance-related groups get involved. And we need performance engineering kind of skills to investigate and fix performance problems in production.

It looks like that in the new generation of Web companies monitoring and capacity planning often included into ‘Site Reliability’, adding, I guess, some confusion to the already existing mess of terms and notions.

P.S. By the way, the only conference covering almost all topics mentioned above is CMG. Call for papers and workshops is opened now.

Posted How Do We Measure Computer Resources? on Application Performance Engineering Hub. It looks like an important issue for the high-tech industry for me – it is a pity that it continues to be unnoticed.

Bernd Harzog’s post Why is Application Performance Management so Screwed Up? started a lot of discussions on the Internet. The post is a very good list of existing issues you may face when you try to use APM tools. I’d add one more – overheads. At least for the first generation, the claim that you may use APM in production worked only if you did very selective monitoring.

My view of APM is that first generation of APM tools so well described by Bernd was very immature. Not that something was explicitly wrong with the APM in general – really wrong was the drastic contrast between what the tools actually could do and marketing promises of tool vendors. The vendors talked more about the APM vision and how the APM tools are supposed to work – but not about the exact things these tools are able to do. Which you figured out in the best case after you spent a few days evaluating the product.

If check Garter Magic Quadrant for Application Performance Monitoring or my list of tools, it is clear that the market is very crowded and not well defined. There is no good criteria you can compare tools and different tools may actually do pretty different things, although it may be difficult to understand reading about them on vendor’s sites.

However I’d say that now we are getting the second generation of APM tools which are much closer to the APM promise for some technologies. I don’t want to list names here and separate “first” and “second” generations. I’d guess that some “first” generation tools might advance to the “second” generation if kept progress – but, as I said, it is difficult to say without actual evaluation of the tools. So I am hearing a lot of stories that people were able to successfully implement APM for system X using tool Y without many problems.

Still you doesn’t have a product which will do APM across all platforms and system if you have a full zoo of different technologies some of which are older than most of your IT employees (as many large corporations do). And don’t believe to anybody who tells you that they can do that. Still it looks like you can do it now for more systems with fewer problems – and start reaping the benefits of APM. Actually I don’t see any other alternative to APM in the long run – although it is a topic for a separate post. But be aware of all points mentioned in Bernd’s post – and check if the product you are going to use doing what you need in the way you want.

P.S. Just before posting noticed another Bernd Harzog post where he shares his view of next generation APM products.