Alex Podelko

Performance testing is an interesting discipline: while you may see a lot of people doing it, not much is written about it. And what is written is often either introduction-level or tool-specific (other large groups of topics including performance analysis and performance troubleshooting are wider topics and not testing-specific). But not much about performance testing specific challenges beyond of what is covered by tool manuals.

My group specializes in performance testing of Hyperion / Enterprise Performance Management and Business Intelligence products at Oracle. A few of scripting challenges exist for almost every product. Nothing exceptional – you should resolve them if you have some experience and would be attentive enough, but time after time we are called to save performance testing projects to find out that there are serious problems with scripts and scenarios which make test results meaningless. Even very experienced testers stumble, but problems could be avoided if more time was spent analyzing what is going on.

Let’s consider an example – probably typical for challenges you can face with modern Web-based applications. Some operations, like financial consolidation, can take a long time. The client starts the operation on the server and then waits until it will finish, a progress bar is shown in meanwhile. When recorded, the script looks like (in LoadRunner pseudo-code):

web_custom_request(“XMLDataGrid.asp_7″,
“URL={URL}/Data/XMLDataGrid.asp?Action=EXECUTE&TaskID=1024&RowStart=1&ColStart=2&RowEnd=1&ColEnd=2&SelType=0&Format=JavaScript”,
LAST);

web_custom_request(“XMLDataGrid.asp_8″,
“URL={URL}/Data/XMLDataGrid.asp?Action=GETCONSOLSTATUS”,
LAST);

web_custom_request(“XMLDataGrid.asp_9″,
“URL={URL}/Data/XMLDataGrid.asp?Action=GETCONSOLSTATUS”,
LAST);

web_custom_request(“XMLDataGrid.asp_10″,
“URL={URL}/Data/XMLDataGrid.asp?Action=GETCONSOLSTATUS”,
LAST);

What each request is doing is defined by the ?Action= part. The number of GETCONSOLSTATUS requests recorded depends on the processing time. In the example above it was recorded three times; it means that the consolidation was done by the moment the third GETCONSOLSTATUS request was sent to the server. If playback this script, it will work in the following way: the script submits the consolidation in the EXECUTE request and then calls GETCONSOLSTATUS three times. If we have a timer around these requests, the response time will be almost instantaneous. While in reality the consolidation may take many minutes or even an hour (yes, this is a good example when sometimes people may be happy having one hour response time in a Web application). If we have several iterations in the script, we will submit several consolidations, which continue to work in background competing for the same data, while we report sub-second response times.

Consolidation scripts require creating an explicit loop around GETCONSOLSTATUS to catch the end of consolidation:

web_custom_request(“XMLDataGrid.asp_7″,
“URL={URL}/Data/XMLDataGrid.asp?Action=EXECUTE&TaskID=1024&RowStart=1&ColStart=2&RowEnd=1&ColEnd=2&SelType=0&Format=JavaScript”,
LAST);

do {

sleep(3000);

web_reg_find(“Text=1″,”SaveCount=abc_count”,LAST);

web_custom_request(“XMLDataGrid.asp_8″,
“URL={URL}/Data/XMLDataGrid.asp?Action=GETCONSOLSTATUS”,
LAST);

} while (strcmp(lr_eval_string(“{abc_count}”),”1″)==0);

Here the loop simulates the internal logic of the system sending GETCONSOLSTATUS requests each 3 sec until the consolidation is done. Without such loop the script just send requests and ends the iteration while the consolidation runs for a long time after that.

Some other examples can be found in my presentation A Load Testing Story at CMG’10 – a story about one my project which had a good share of testing challenges.

Many technical details about testing Hyperion products with LoadRunner are included in Oracle support document 1339743.1. A lot of this knowledge is now included in the Hyperion module of Oracle Application Testing Suite. But that is just about one family of products – which is a small sample of challenges you would see around.

According to my experience, creating a meaningful and realistic workload is the main performance testing challenge – and you probably face it in most non-trivial systems (including most of corporate applications). It is surprising that very few people talk about it: reading about performance testing you may get impression that the scripting is trivial and mundane (and it looks like many “performance testers” truly believe in it – and when they face any challenge, they start to talk about the death of performance testing).

Only few names come to my mind who write about specific challeneges. For example, see Northway (@northwaysg) / Scott Moore (@loadtester) performance testing blog and white papers or Mentora (@MentoraGroup ) / Dan Downing presentations and white papers. Maybe few more – but definitely they are few and far apart.

Performance testing terminology is not well defined and one of the most ambiguous terms is user concurrency. Re-reading Load Testing: Concurrent Users verses Simultaneous Users by Scott Moore (@loadtester) and LoadRunner Concurrency video by Mark Tomlinson (@mtomlins) inspired me to post this comment. Here is what I wrote (and still believe in it) in my old CMG paper about performance requirements in 2007 ( the latest version of this paper was presented again at CMG’12):

Concurrency is the number of simultaneous users or threads. It is important too: connected, but inactive users still hold some resources. For example, the requirement may be to support up to 300 active users.

The number of online users (the number of parallel session) looks like the best metric for concurrency (complementing throughput and response time requirements).

To summarize my comment, I believe that the number of “really concurrent” users is not an appropriate input metric for performance engineering and performance testing. It perhaps may be an output metric characterizing system’s load if we find a way to measure it (in a way, it is the number of users in the system if we use queuing theory terminology).

After attending Sergey Chernyshev’s (@sergeyche) Scalability vs. Performance presentation at NY Web Performance Meetup and reading Scalability: it’s the question that drives us by Robert David Graham (@ErrataRob) and Scalability vs. Performance: it isn’t a battle by Theo Schlossnagle (@postwait) I would like to share my understanding. While I agree in general with everything said, I would rather word it differently. The topic became loaded, so accents are important. Robert, for example, states that “performance” and “scalability” are orthogonal problems. Well, no, they are not. They are different, but correlated notions. Even leaving aside that performance and scalability are somewhat vague terms.

If we speak about web systems now, it looks like we can roughly separate two main components in response time (which is the main performance metric): backend (server-side) time and frontend (network and client-side time). There are subtleties and grey areas, but I’d ignore them here. The frontend time, the subject of Web Performance Optimization (WPO), doesn’t relate to scalability as far as it is not involving server processing (again ignoring subtleties).

The proportion of frontend time vs. backend time may be any at all. According to Steve Souders (@souders), the founder of WPO, 80-90% of the end-user response time is spent on the frontend. But even for major web sites the backend time for requests involving database processing (such as submitting orders or querying order status) may be more noticeable. And there are plenty of corporate web applications where the share of frontend time is rather small. Of course, the starting point of any performance troubleshooting is to find where time is spent. And there is absolutely no sense to optimize parts where time is not spent.

However, there is one important “but”. While front-end time supposed to be constant (another simplification, but again ignoring subtleties), the backend (server) time depends on load. The heavier is load, the larger may be server time. And at some point it may skyrocket making the system practically unusable. So thinking about what you need to optimize, you need to check where time is spend under maximal load. Unless you don’t care about downtime and user experience, the way to do it is load testing.

And here we get to scalability. The frontend performance indeed doesn’t matter here and is independent of scalability. But the backend performance is directly related to scalability. The relationship, of course, may be non-linear and quite sophisticated – but it does exist.

To illustrate it, let’s consider one simple (but still typical) example. The backend processing takes X ms, the time is mainly spent in CPU and we don’t have any other bottlenecks. In this case the server response time would be mainly CPU processing time – and every request would take X ms of CPU time (if we don’t have parallelism here). As soon as we take most of available CPU time, server response time would skyrocket (that situation may be modeled using queuing theory). So there is a load when the system becomes practically unusable – and the question is just when we get to this load. We, of course, may get to problems sooner if we have any scalability problem inside the system or run out of another kind of resources.

Generally speaking, you can increase scalability by either optimizing server processing (using less resources) or providing more resources. Of course, if your architecture allows using these additional resources – so mainly scalability boils down to ability to parallelize your processing (and often limited by what you can’t fully parallelize – like a centralized database).

We do have two parts of response time – frontend and backend – which behaves differently and may need different approaches and tools to optimize. But the end user experience is the sum of these two parts – where the backend time is a function of load. You can’t say much about your end-to-end performance and its backend part until you check it under load – and load testing is the safe way to do so.

Historically performance engineering concentrated on backend – where main performance and scalability issues were – and practically ignored frontend (which indeed was usually pretty straightforward then). Several sub-disciplines were formed including performance analysis, capacity planning, and load testing. Later, when sophistication of frontend skyrocketed, a whole new discipline was established by Steve Souders and quickly grew around Velocity and Web Performance meetups. Unfortunately, it practically dismissed performance engineering developments of the last 40 years (maybe even more – the Computer Measurement Group (CMG) was founded in 1975). While frontend WPO definitely has its own specific, I’d still expect to see a holistic approach to performance engineering, taking in account all aspects of performance and scalability end-to-end.

Just read Gartner Q&A: Analytics vs. APM with Will Cappelli, Gartner Research VP in Enterprise Management, part one, part two, and part three about his latest report: Will IT Operations Analytics Platforms Replace APM Suites?

The discussion is very interesting and informative, touches areas that interested me for a long time, but the title doesn’t make sense for me. Maybe I just got behind in terminology – but here is as I understand it.

I joined Hyperion in 1999 and, with one break, did, in a way, performance management of performance management applications. I was responsible for application performance of business performance management and business intelligence applications. At this point you probably can guess some questions that came to my mind. Can we use anything from application performance management in business performance management? Or, vise versa, can we use business analytic software for application performance analysis? Well, I haven’t still figured out a meaningful way to do it.

I am also involved in CMG (worldwide organization of IT professionals specializing in performance, capacity and service management), where there was an idea that skills of application performance and capacity management may be used for business performance and capacity. I waited with interest if the best experts in application performance and capacity management (and CMG is the place where you find them) would be able to come with interesting ideas of using their skills for business (beyond the trivial fact that performance impacts business). Still waiting.

While these two areas, application performance and business performance, look very similar and, of course, often use the same underlying approaches and math, the devil is in details. They are just two different areas and not much may be re-used between them (as soon as we get to details).

Reading the discussion, I came to one explanation of what is going on. It also explains one more phenomenon surprising me for a while. The phenomenon was that there are many new companies on the market providing only end-user monitoring (EUM, or real-user monitoring, RUM) and apparently doing pretty well. While EUM is definitely a very important APM area, my understanding was that it is just one part of APM and, without other parts, provides very limited APM value. My understanding was that we should rather move toward integrated APM solutions. Still the impression is that some such EUM companies doing even better than companies providing integrated APM solutions.

Well, the explanation is that web analytics (even with EUM) is just a completely different area from APM (although EUM is an important part of APM too). Web analytics is a business application (and yes, end-user performance is one of components of business analysis now) and APM is an “IT” application. If we agree on this, it explains all these strange titles about death of APM or replacing APM by analytics. End-user performance with web analytics became a business intelligence application – and the business intelligence market is many times larger than IT intelligence market (where we still have all true APM solutions – with all their problems because the APM is, in essence, much more sophisticated area technically than business analytics). It was always this way – recall market size of business intelligence (Hyperion, Cognos, Business Objects, etc.) vs. market size of companies providing monitoring and deep diagnostics solutions.

Business intelligence/analytics moved to a new level: from sales analysis we are getting to user clicks analysis. And there we have another level of data volume (==big data) and we are getting to the point when end-user performance is just one more dimension of business data. Performance impacts business – and business wants to analyze it. Well, in application performance management we worked with “big data” for ages – and have you heard about it as a world problem until it got into the business realm?

But nothing changed in APM – if we find a way to separate these business-related applications, I guess we find that the APM market is slowly growing and that APM applications are slowly becoming better and better. To manage your applications, you need an APM solution, not web analytics and not EUM only (although EUM is important for APM too).

As I highlighted in my performance requirements article (here is the last version just presented at CMG’12), we have two completely different views: business view and “IT” view. Business is interested in one (and only one) performance metric – end-user response time. That’s it, nothing else at all. All other stuff – throughput, resource utilization, bandwidth, latency, etc. – is for IT internal use only (even if “IT” is the core of the company as with many web services). Yes, business people know these words too – they hear them all the time as explanation why we can’t get requested performance or why we need to spend another pile of money on something IT needs – but if “IT” delivered the requested performance they were quite happy not even knowing these words. And that is exactly what we see here: we have solutions for business (EUM – which is much simpler of integrated APM solutions and provides everything business needs) and solutions for “IT” (with all these stuff you needed to manage applications performance – but much more sophisticated and difficult to implement).

And now, after my posts about agile performance testing and performance requirements in agile projects, we are getting to another fundamental issue behind performance testing issues in agile projects.

There are two very important assumptions for what I am saying below: stakeholders understand the need in performance engineering and the team knows the basics. How to get there is a very interesting topic, but it is outside of this post. For those, who are not there yet, the issues described below may sound abstract and they probably have more urgent problems to fight – but it is something you may want to be aware about while you are getting there.

The fundamental issue is, as I see it, that performance engineering teams don’t scale well, even assuming that they are competent and effective. At least not in their traditional form. They work well in traditional corporate environments where they check products for performance before release, but they face challenges as soon as we start to expand the scope of performance engineering (early involvement, more products/configurations/scenarios, etc.). And agile projects, when we need to test the product each iteration or build, expose the problem through the increased volume of work to do.

Just to avoid misunderstandings, I am a strong supporter of having performance teams and I believe that it is the best approach to building performance culture. Performance is a special area and performance specialists should have an opportunity to work together to grow professionally. The details of organization may vary (Scott Barber, for example, specified three models: “on demand”, “on retainer”, “full immersion”), but a center of performance expertise should exist. Only thing I am saying here is that while the approach works fine in traditional environment, it needs major changes in organization, tools, and skills when the scope of performance engineering should be extended (as in the case of agile projects).

Actually remedies are well known: automation, making performance everyone jobs (full immersion), etc. However they are not wide-spread yet.

Historically performance testing automation was almost non-existent (at least in traditional environments). Performance testing automation is much more difficult than, for example, functional testing automation (I use “automation” here as what should be done for “continuous testing”, i.e. process when you run test and get a report automatically for a new build without human intervention – not in its old way, when it meant using a tool; in performance we almost always use a tool). Setups are much more complicated. A list of possible issues is long. Results are complex (not just pass/fail). It is not easy to compare two result sets. So it is definitely much more difficult and would probably require much more human intervention, but it isn’t impossible.

However, the cost of performance testing automation is high. You need to know system well enough to make meaningful automation. Automation for a new system doesn’t make much sense – overheads are too high. So there was almost no automation in traditional environment [with testing in the end with a record/playback tool]. When you test the system once in a while before next major release, chances to re-use your artifacts are low.

It is opposite when the same system is tested again and again (as it should be in agile projects). It makes sense to invest in setting up automation. It rarely happened in traditional environments – even if you test each build, they are far apart and the difference between the builds prevents re-using the artifacts (especially with recorded scripts – API, for example, is usually more stable). So demand for automation was rather low and tool vendors didn’t pay much attention to it. Well, the situation is changing – we may see more automation-related features in load testing tools soon.

There are some vendor claiming that their load testing tools better fit agile processes, but it looks like in the best case it means that the tool is a little easier to handle (and, unfortunately, often just because there is not much functionality in it). Even if there is something that may be used for automation, like starting by a command line with parameters, it is difficult to find out.

At the moment, I read about few implementation of continuous performance testing – for example OpTier or Betfair (and few about measuring response times during functional testing in single-user mode – like this – which is a good step toward full-scale performance testing automation). Some prototypes, like this, are described– but without many details.

Probably we don’t see it more often because we don’t have much infrastructure for that kind of automation and performance testers may be not the best people to create complex integrated solutions from dozens of not-related pieces (as those who implemented it did). When we get more automation support in tools, we will probably see it changing.

By the way, I am not saying that automation would replace performance testing as we know it. Performance testing of new systems is agile and exploratory in nature and can’t be replaced by automation (well, at least not in the foreseen future). Automation would complement it – together with additional input from development offloading performance engineers from routine tasks not requiring sophisticated research and analysis.

It looks like agile methodologies are somewhat struggling with performance requirements (and non-functional requirements in general). Probably there are several reasons for that. One is that actually even traditional software development methodologies and processes never came with a good approach to handle performance requirements. They are, of course, considered in both literature and practical projects – but are usually handled rather in ad hoc manner. Actually the process of gathering and elaboration of performance requirements is rather agile in itself, and attempts to make it rigorous and formal look unnatural and have never fully succeeded – so it should be easier and more natural to do it as part of agile methods. Still the challenge of handling multidimensional and difficult to formalize performance requirements remains intact and the difference is rather in minor adjustments to agile processes than in the essence of performance requirements.

Another reason is that practical agile development is struggling with performance in general. Theoretically it should be a piece of cake: every iteration you have a working system and know exactly where you stand with the system’s performance. You shouldn’t wait until the end of the waterfall process to figure out where you are – on every iteration you can track your performance against requirements and see the progress (making adjustments on what is already implemented and what is not yet). Clearly it is supposed to make the whole performance engineering process much more straightforward.

Unfortunately, it looks like it doesn’t always work this way in practice. So such notions as “hardening iterations” and “technical debt” get introduced. Although it is probably the same old problem: functionality gets priority over performance (which is somewhat explainable: you need first some functionality before you can talk about its performance). So performance related activities slip toward the end of the project, and the chance is missed to implement a proper performance engineering process built around performance requirements.

Another issue here is that agile methods are oriented toward breaking projects into small tasks, which is quite difficult to do with performance (and many other non-functional requirements) – performance-related activities usually span the whole project.

There is no standard approach to specifying performance requirements in agile methods. Mostly it is suggested to present them as user stories or as constraints. And the difference is not so much in the way the requirements are presented, both ways rather use plain text.

User stories assume using a user voice form. Cohn, for example, suggests to use the “As a , I want , so that ” template for user stories (although he cautions that the user story template should only be used as a thinking tool, it should not be used as a fixed template). For constraints, both traditional expressions and user voice forms may be used.

The difference between user stories and constraints approaches is not in performance requirements per se, but how to address them during the development process. The point of the constraint approach is that user stories should represent finite manageable tasks, while performance-related activities can’t be handled as such because they usually span multiple components and iterations. Those who suggest to use user stories address that concern in another way – for example, separating cost of initial compliance and cost of ongoing compliance.

It looks like testingreflections.com, where I had blogged for a while, doesn’t exist anymore. One more confirmation that you can’t rely on somebody else long-term – who knows what would happen… Well, I still have some drafts of my old posts. Here is what I wrote about agile performance testing in 2009:

Two Views of Agile Unit Testing reminded me (indirectly) the QAForums Agile Performance Testing – Oxymoron? discussion about agile development and performance testing some time ago

Well, if we talk about agile development I don’t see any problem with performance testing at all: you have a working build each iteration – you test it. What is a problem? It should be synonym, not oxymoron.

All issues mentioned in the discussion boil down to one: nobody wants to pay for it (so no testers, no equipment, etc). Don’t see anything especially agile in it. Maybe companies still don’t switch to agile development for major projects – and don’t want to spend money for minor projects. Another explanation may be that in most traditional projects performance testing happens at the very end instead of every iteration (so require less resources) – but it not the way to do it properly.

I see significantly more problems doing performance testing properly during traditional development. I wrote a paper that has the same title, Agile Performance Testing, but completely different content: how to do performance testing in an agile way for any kind of projects (where agile projects look rather like a trivial case for me).

Well, returning to the original question of the discussion, there are a few articles in the Internet about the subject:

Agile Performance Testing by Jamie Dobson

An Explanation of Performance Testing on an Agile Team Part 1 and Part2 by Scott Barber

Chapter 6 of Performance Testing Guidance for Web Applications. Managing an Agile Performance Test Cycle.

Well, I still believe so. I still believe that performance testing in agile projects should be easier conceptually (although, of course, it would be a lot of technical difficulties – but who said that it should be easy?).

There are a few good newer articles addressing some aspects of the subject:

Top Ten Secret Weapons For Agile Performance Testing by Patrick Kua

Agile Performance Testing process AgileLoad whitepaper

Performance Testing in the Agile Enterprise by Scott Barber

Performance By Design – an Agile Approach by Jason Buksh

Still the subject is not covered much (and it looks like not actually practiced much – apart from involvement of traditional performance testing team in agile projects).

I started to prepare my talk for Belgium Testing Days (which promises to be a very interesting global testing conference) and the subject got me to very philosophical questions – will try to share my further thoughts about the subject in following posts.

It is interesting to look how handling performance changed with time. Probably performance went beyond single-user profiling when mainframes started to support multiprogramming. It was mainly batch loads with sophisticated ways to schedule and ration consumed resources as well as pretty powerful OS-level instrumentation allowing to track down performance issues. The cost of mainframe resources was high, so there were capacity planners and performance analysts to optimize mainframe usage.

Then the paradigm changed to client-server and distributed systems. Available operating systems didn’t have almost any instrumentation and workload management capabilities, so load testing became almost only remedy in addition to system-level monitoring to handle multi-user performance. Deploying across multiple machines was more difficult and the cost of rollback was significant, especially for Commercial Of-The-Shelf (COTS) software which may be deployed by thousands of customers. Load testing became probably the main way to ensure performance of distributed systems and performance testing groups became the centers of performance-related activities in many organizations.

While cloud looks quite different from mainframes, there are many similarities between them, especially from the performance point of view. Such as availability of computer resources to be allocated, an easy way to evaluate the cost associated with these resources and implement chargeback, isolation of systems inside a larger pool of resources, easier ways to deploy a system and pull it back if needed without impacting other systems.

However there are notable differences and they make managing performance in cloud more challenging. First of all, there is no instrumentation on the OS level and even resource monitoring becomes less reliable. So all instrumentation should be on the application level. Second, systems are not completely isolated from the performance point of view and they could impact each other. And, of course, we mostly have multi-user interactive workloads which are difficult to predict and manage. That means that such performance risk mitigation approaches as APM, load testing, and capacity management are very important in cloud.

It is interesting that while performance is the result of all design and implementation details, performance engineering area remains very siloed. Those who do capacity planning are usually not involved much in performance testing or software performance engineering. The new and fastest growing group, web performance specialists, remains mainly isolated from other performance-related groups. People and organizations trying to span all performance-related activities together are few and far apart.

I don’t see that the need that need for specific performance-related expertise, such as load testing or capacity planning, is going away. Even in case of web operations, we would probably see load testing coming back as soon as systems become more complex and performance issues start to hurt business. There perhaps would be less need for “performance testers” as it was at the heyday due to better instrumenting, APM tools, continuous integration, resource availability, etc. – but I’d expect more need for performance experts who would be able to see the whole picture using all available tools and techniques.

Updated the list of my publications. Suddenly a lot happened in December.

At the CMG Conference I presented the latest version of my performance requirements view: Performance Requirements: the Backbone of the Performance Engineering Process, paper and presentation.

Also at the CMG Conference I presented Load Testing: See a Bigger Picture, paper and presentation. A magazine version of it, A Bird’s-Eye View of Load Testing, was published in the December issue of Software Test and Quality Assurance Magazine.

My notes on Performance Testing in the Cloud: Look Beyond the Word were published in the December issue of Testing Experience.

My post Performance: See the Whole Picture was included in 2012 Performance Calendar.

My next talk, Agile Aspects of Performance Testing, would be at Belgium Testing Days in Brussels, Belgium on March 1, 2013. Belgium Testing Days is a great global testing conference in spite of its rather local name – just check the program. All areas of testing covered – even performance testing is well represented, that is rather unusual for testing conferences.

One prediction in 13 APM Predictions for 2013, surprised me: 7B. Splitting the Market – Ops vs. Dev Tools by Chris Neal, BlueStripe CEO and Co-Founder (@ctneal).

My understanding of the APM market is that it is not yet a mature market. There are a lot of quite different products on the market whose vendors name them APM or whatever nice buzzword is around (some changing them quickly according to the latest fashion). Some are ‘deep diagnostic’ tools which provide insights into specific environment, some are transactional tracking tools, some are rather traditional monitoring tools, and some are Web end-user monitoring. When I started my collection of links 7 years ago, I classified them into Application Management Tools, Enterprise Monitoring Tools, and Transaction Tracking/Business Activity Monitoring – definitely far from ideal (especially because APM became an umbrella category), but still looking for a better way to do this. But marketing messages of almost all vendors were practically identical: “Buy our solution and you get full insight into your environment with minimal configuration and overheads”. Which was quite far from the truth on all three points.

It became much better (so I am surprised by the posts talking about APM death basically because APM doesn’t deliver on above mentioned promises – well, it was much worse some time ago). And now, I guess, we see some symptoms of maturing. Gartner’s five dimensions of APM did, I guess, a good job of structuring the solutions (at least in what concerns ‘Monitoring’ – it doesn’t look like we get to ‘Management’ yet). So some vendors started to deliver a more meaningful message on what their solution really does instead of saying that it solves all your performance problems. And, in a way, some tools started to be grouped around these two categories – dev-oriented vs. ops-oriented. See, for example, The Third Generation of APM by Bernd Harzog (@Bernd_Harzog) – although Bernd is using rather un-orthodox terminology there.

So I see a point in Chris’ prediction if we talk about maturing market and more concise vendor messages about what their solutions actually do (and maybe grouping them around these two categories). But if we talk about overall long-term trend, we probably expect APM tools to cover all this functionality in one seamless solution (with all five Gartner’s dimensions and, probably, much more). That is the main problem of the APM market – even the companies that formally cover all this APM functionality have actually several separate solutions not well integrated between themselves. And I suspect that the first vendor that comes out with really good integrated solution will get a great competitive advantage. Almost all other APM predictions actually screaming about this market need – having one, comprehensive, end-to-end solution (either by one vendor or by cooperation of several). And DevOps need that from APM: not only ability to register a long transaction, not only ability to see at what component the time was spent, but also tracing it down to specific query or method and its parameters – because otherwise you are actually back to the traditional approach when you try to reproduce operational problems in test or dev environments and thus lose a significant part of the promised APM advantage.