Alex Podelko

Just read Gartner Q&A: Analytics vs. APM with Will Cappelli, Gartner Research VP in Enterprise Management, part one, part two, and part three about his latest report: Will IT Operations Analytics Platforms Replace APM Suites?

The discussion is very interesting and informative, touches areas that interested me for a long time, but the title doesn’t make sense for me. Maybe I just got behind in terminology – but here is as I understand it.

I joined Hyperion in 1999 and, with one break, did, in a way, performance management of performance management applications. I was responsible for application performance of business performance management and business intelligence applications. At this point you probably can guess some questions that came to my mind. Can we use anything from application performance management in business performance management? Or, vise versa, can we use business analytic software for application performance analysis? Well, I haven’t still figured out a meaningful way to do it.

I am also involved in CMG (worldwide organization of IT professionals specializing in performance, capacity and service management), where there was an idea that skills of application performance and capacity management may be used for business performance and capacity. I waited with interest if the best experts in application performance and capacity management (and CMG is the place where you find them) would be able to come with interesting ideas of using their skills for business (beyond the trivial fact that performance impacts business). Still waiting.

While these two areas, application performance and business performance, look very similar and, of course, often use the same underlying approaches and math, the devil is in details. They are just two different areas and not much may be re-used between them (as soon as we get to details).

Reading the discussion, I came to one explanation of what is going on. It also explains one more phenomenon surprising me for a while. The phenomenon was that there are many new companies on the market providing only end-user monitoring (EUM, or real-user monitoring, RUM) and apparently doing pretty well. While EUM is definitely a very important APM area, my understanding was that it is just one part of APM and, without other parts, provides very limited APM value. My understanding was that we should rather move toward integrated APM solutions. Still the impression is that some such EUM companies doing even better than companies providing integrated APM solutions.

Well, the explanation is that web analytics (even with EUM) is just a completely different area from APM (although EUM is an important part of APM too). Web analytics is a business application (and yes, end-user performance is one of components of business analysis now) and APM is an “IT” application. If we agree on this, it explains all these strange titles about death of APM or replacing APM by analytics. End-user performance with web analytics became a business intelligence application – and the business intelligence market is many times larger than IT intelligence market (where we still have all true APM solutions – with all their problems because the APM is, in essence, much more sophisticated area technically than business analytics). It was always this way – recall market size of business intelligence (Hyperion, Cognos, Business Objects, etc.) vs. market size of companies providing monitoring and deep diagnostics solutions.

Business intelligence/analytics moved to a new level: from sales analysis we are getting to user clicks analysis. And there we have another level of data volume (==big data) and we are getting to the point when end-user performance is just one more dimension of business data. Performance impacts business – and business wants to analyze it. Well, in application performance management we worked with “big data” for ages – and have you heard about it as a world problem until it got into the business realm?

But nothing changed in APM – if we find a way to separate these business-related applications, I guess we find that the APM market is slowly growing and that APM applications are slowly becoming better and better. To manage your applications, you need an APM solution, not web analytics and not EUM only (although EUM is important for APM too).

As I highlighted in my performance requirements article (here is the last version just presented at CMG’12), we have two completely different views: business view and “IT” view. Business is interested in one (and only one) performance metric – end-user response time. That’s it, nothing else at all. All other stuff – throughput, resource utilization, bandwidth, latency, etc. – is for IT internal use only (even if “IT” is the core of the company as with many web services). Yes, business people know these words too – they hear them all the time as explanation why we can’t get requested performance or why we need to spend another pile of money on something IT needs – but if “IT” delivered the requested performance they were quite happy not even knowing these words. And that is exactly what we see here: we have solutions for business (EUM – which is much simpler of integrated APM solutions and provides everything business needs) and solutions for “IT” (with all these stuff you needed to manage applications performance – but much more sophisticated and difficult to implement).

One prediction in 13 APM Predictions for 2013, surprised me: 7B. Splitting the Market – Ops vs. Dev Tools by Chris Neal, BlueStripe CEO and Co-Founder (@ctneal).

My understanding of the APM market is that it is not yet a mature market. There are a lot of quite different products on the market whose vendors name them APM or whatever nice buzzword is around (some changing them quickly according to the latest fashion). Some are ‘deep diagnostic’ tools which provide insights into specific environment, some are transactional tracking tools, some are rather traditional monitoring tools, and some are Web end-user monitoring. When I started my collection of links 7 years ago, I classified them into Application Management Tools, Enterprise Monitoring Tools, and Transaction Tracking/Business Activity Monitoring – definitely far from ideal (especially because APM became an umbrella category), but still looking for a better way to do this. But marketing messages of almost all vendors were practically identical: “Buy our solution and you get full insight into your environment with minimal configuration and overheads”. Which was quite far from the truth on all three points.

It became much better (so I am surprised by the posts talking about APM death basically because APM doesn’t deliver on above mentioned promises – well, it was much worse some time ago). And now, I guess, we see some symptoms of maturing. Gartner’s five dimensions of APM did, I guess, a good job of structuring the solutions (at least in what concerns ‘Monitoring’ – it doesn’t look like we get to ‘Management’ yet). So some vendors started to deliver a more meaningful message on what their solution really does instead of saying that it solves all your performance problems. And, in a way, some tools started to be grouped around these two categories – dev-oriented vs. ops-oriented. See, for example, The Third Generation of APM by Bernd Harzog (@Bernd_Harzog) – although Bernd is using rather un-orthodox terminology there.

So I see a point in Chris’ prediction if we talk about maturing market and more concise vendor messages about what their solutions actually do (and maybe grouping them around these two categories). But if we talk about overall long-term trend, we probably expect APM tools to cover all this functionality in one seamless solution (with all five Gartner’s dimensions and, probably, much more). That is the main problem of the APM market – even the companies that formally cover all this APM functionality have actually several separate solutions not well integrated between themselves. And I suspect that the first vendor that comes out with really good integrated solution will get a great competitive advantage. Almost all other APM predictions actually screaming about this market need – having one, comprehensive, end-to-end solution (either by one vendor or by cooperation of several). And DevOps need that from APM: not only ability to register a long transaction, not only ability to see at what component the time was spent, but also tracing it down to specific query or method and its parameters – because otherwise you are actually back to the traditional approach when you try to reproduce operational problems in test or dev environments and thus lose a significant part of the promised APM advantage.

CMG’12 is an annual conference organized by Computer Management Group – a volunteer organization of professionals specialized in performance, capacity, and IT service management. This year it is held in Las Vegas, December 2-7, 2012.

Why I love CMG, spend a lot of my time organizing and promoting it, and coming there every year (sometimes on my own)? Well, because I believe that it is the best (and actually the only) conference on performance and capacity, the main topic of my interest for the last fifteen years. There are many conferences on specific topics. For example, the Velocity conference, devoted to web performance, is significantly larger and more popular – but it is still devoted mainly to single-use web performance, leaving all other performance and capacity questions to CMG. Let me share some of my excitement – of course, from my personal point of view (there is plenty of other highlights, but I am mentioning only the ones that are close to my heart).

This year the conference covers all aspects of performance (well, almost all – performance is so sophisticated subject that there is always much more to learn) from Web Performance Optimization (the conference opens by the keynote by Patrick Meenan, a web performance Google guru and the creator of WebPagetest) to mainframe performance (and everything in between).

The conference starts with a half-day workshops – see here the description. In addition to workshops, there are CMG-T sessions during the whole conference. Each CMG-T class spans 2 or 3 session spots, so it could easily be considered as a workshop or a training class. All led by renown experts with tons of experience, you hardly would get anybody even remotely close if you engage in a typical vendor class (not to mention a unique vendor-neutral or vendor-agnostic perspective you hardly find anywhere else). You have the CMG-T track through the whole conference and every one of them is a gem:

• Capacity Planning by Ray Wicks
• z/OS Basics by Glenn Anderson
• Java Performance Analysis and Tuning by Peter Johnson
• Model and Forecasting Basics by Dr. Michael Salsburg
• Network Performance Management by Manoj Nambiar
• Windows System Performance Management and Analysis by Jeffry Schwartz
• Using SAS to Communicate Your Message by MP Welch

CMG’12 has 4 keynote/plenary session and almost a hundred regular track sessions going on from mid-Monday to mid-Friday. The conference is 5 tracks wide. One track, as I already mentioned, is CMG-T 101– type classes (with 301-depth). Others four tracks shared between five subject areas: Performance Engineering and Testing, Capacity Planning, Application Performance Management, IT Service Management, and Hot Topics. It is difficult to list all highlights – too many. While I know many great presenters and am fascinated by many topics, commenting every single one would take too much time and space. Probably you just need to look at agenda – there are three different views: preliminary agenda (overview, a day on a page), a list of abstracts in a single pdf document and search/scheduler (click on the abstract number to see the abstract).

One track on Wednesday is a Michelson award track. CMG is presenting Michelson award since 1974 (if you wonder, Albert Abraham Michelson was known for his technical accomplishments in measuring the speed of light and for his role as teacher and inspirer of others – and measuring is the key to performance). This year we will see many Michelson winners presenting: Dr. Connie Smith, the founder of Software Performance Engineering, Dr. Daniel Menasce, the author of many great books about performance and capacity planning, Adam Grummit, the author of the great Capacity Management book (ITSM Library) and the CMG president, Dr. Pat Artis, Bruce McNutt, and Dr. Michael Salsburg.

I believe that the main advantage of attending CMG is networking with best world experts in almost all areas of performance and capacity. Nowadays you can find all technical information on the Internet, but there is no substitution to face-to-face conferences to learn how to use it and what were people experiences, and, of course, to see the whole picture. Especially in performance: performance is the result of every design and implementation detail and you need to be learning all the time to keep up with coming challenges.

I am presenting there too: Load Testing: See a Bigger Picture on Thursday and
Performance Requirements: the Backbone of the Performance Engineering Process on Friday. Nothing comparing to other CMG’12 highlights, but I hope to trigger discussions around these two very important topics.

And, of course, it is Las Vegas – and Rio’s rate is $55 per night until November 14th. See you there!

The Computer Measurement Group (CMG) calls for papers and presentations for CMG’s 38th International Conference to be held in Las Vegas, Nevada, December 3rd through 7th, 2012.

The 2012 CMG conference will cover all areas of systems management, including but not limited to: capacity planning, IT service management, application performance management, performance engineering and testing, as well as the latest developments in the overall field of computer performance evaluation. See the Call for Papers and Call for Workshops for details.

CMG is the source of unbiased and objective expert information and practical, real life experiences across all computing platforms in the computer industry for over 30 years. Share your knowledge and experiences: write a paper and submit it for presentation at CMG’12.

Paper are categorized as Introductory, Tutorial, Advanced, or User Experience. I want to especially encourage all of you to consider writing a User Experience paper. Every year, the conference evaluations show a common theme: “More User Experience Papers, please!” You don’t need to be one of the field’s superstars to write one — in fact, they seem to work better from people who are just working in the field, in non-IT companies and government bodies. Just tell us what problem you faced, how you went about figuring out what the cause was, and how you dealt with it. Mentors are available for writing assistance, and may be requested at any point in the writing process, including before the paper is started. Just write mentor@cmg.org and ask.

Please take the time to participate in the CMG’12 program. It will be rewarding for both authors and attendees, and as we all share our knowledge we all become more complete professionals.
Paper submission through the CMG website is now available. For more information go to paper submission and workshop submission.

The deadline for paper submissions is June 8, 2012.

Please send questions to CMG’12 Program Chair, Bill Jouris at cmgpc@cmg.org.

There are no standards on titles and skill sets related to performance dimension of IT. I decided to put together how I understand them (most terms are vague, so it is quite possible that other people understand them differently). Of course, it is a simplification – but the topic is probably too heavy influenced by organization history and politics in every particular organization to be clear cut anyway.

I still think that we can break the whole area into three major categories: design (and development), testing, and production (maybe somewhat matching ITIL terms of Service Design, Service Transition, and Service Operation). The term Performance Engineering may be related to the whole area (or maybe related to the design category – in this case sometimes referred as Software Performance Engineering, SPE).

Performance Design. Talking about the design category (I used the ‘Performance Design’ term to group all performance-related activities during design and development , although it isn’t used this way – probably reflecting that the whole area is not quite existing as a separate discipline), we have specific areas of performance engineering knowledge for each specific technology. Such as Java performance, .Net performance, etc. One relatively new, but large and popular area is Web Performance Optimization, covering end-user Web performance. And, of course, we have Software Performance Engineering (SPE) trying to establish generic approaches – although SPE progress wasn’t too impressive since Dr. Connie Smith published ‘Performance Engineering of Software Systems’ in 1990.

It is definitely supposed to be an important part of the skill set of software architects (on a higher level, SPE, etc.) and software developers (maybe on a lower level, how efficiently design specific component using the chosen technology – but good understanding of high-level performance engineering won’t hurt either).

And while many architects and developers have some understanding of performance, often the main stress is on functionality and deadlines, so performance is left to the very end – where it sometimes may be indeed tuned in (usually when technologies are mature and the team is quite experienced), and sometimes require major changes (and late changes are very expensive).

It looks like the idea to have an explicit person responsible for performance from the beginning (starting from requirements) and working with other architects and developers to build it in makes sense. The title may be ‘performance architect’ or ‘performance champion’. Although such people are rare – rather we could see a proactive person from performance engineering or performance testing groups trying to ask performance questions early.

Performance Testing. Including, of course, all other variations and names, such as load, stress, endurance, etc. testing. ITIL matching term would probably Service Validation and Testing. All ways to apply synthetic load to the system and analyze system’s behavior. In the narrow sense, ‘performance tester’ is responsible for creating and applying such load (test scripting and execution). In a wider sense, it also includes workload characterization (workload modeling), performance analysis and performance troubleshooting – and often such person is referred as ‘performance engineer’. In some cases they are different people: performance tester is responsible for applying the load and performance engineer (maybe performance analyst in this case) is responsible for system analysis and optimization.

I definitely put performance testing in a separate category due to specific set of skills required: workload generation. And, perhaps, techniques to find and fix issues in the system applying an appropriate workload. But definitely not because “testing should go after development before production” as it use to be in the waterfall approach – testing should start as early as possible mostly overlapping with development and may continue in production. Monitoring the system using synthetic workload, for example, I’d rather also put in this testing category – it is actually testing the production system in parallel to production workload.

Performance Management, perhaps, may be a good name for the collection of performance-related activities and skills in production (and around).

It is interesting that ITIL places Capacity Management and Service Level Management processes into Service Design. I see a point here – you definitely need to allocate capacity before deploying the system, and Service Levels should probably come directly from the performance requirements. Still real people working in these areas are usually part of operations. Capacity Planners are responsible for allocating resources, although fewer and fewer people have such title and these responsibilities get spread between other groups (which, unfortunately, often don’t have appropriate skills).

Service Level Management would probably handled by Performance Monitoring (Analysis). ITIL matching term would probably Service Measurement. Title ‘Performance Analysts’ used often in the past – but not very popular anymore. Probably title ‘Performance Engineer’ is more popular now. And, of course, it may be specialized, like Database Monitoring, System Monitoring, Application Server Monitoring. These may be done by respective administrators (DBA, system administrator, etc.).

Application Monitoring – relatively new staff. Usually referred as Application Performance Monitoring. The idea is to measure application-specific metrics (including business-related metric, end-user metrics, etc.) in addition to those system-level metrics that used to be measured earlier. Importance of application monitoring is definitely growing. From one side, system-level metrics becomes less relevant in today’s infrastructure with virtualization, multi-tenancy, cloud, etc. From another side, the system becomes so complicated that trying to figure out what is going on using low-level metrics becomes nightmare. Form the third side, full monitoring from the business point of view becomes a business requirement – and it is where IT can provide unique business advantage.

Probably Application Performance Management (APM) would the right category encompassing most production-related categories such as Performance Monitoring, Capacity Management, Diagnostics (troubleshooting) and Tuning (and Optimization – although this may somewhat get into re-design category). We probably not there yet and Application Performance Management is rather a vague vision than reality. Gartner, for example, stresses that APM is Application Performance Monitoring, not Management. And I am not sure what would be a title of the person doing this. Management is a favorite word for an area of expertise (as in Performance Management or Capacity Management), but Manager (at least in the US) still means a person who manages other people. So the title, I guess, would be the same ubiquitous ‘performance engineer’.

Performance Troubleshooting or Diagnostics is definitely important part of Performance Management and is an application of performance engineering to existing performance issues. While it is probably the most typical performance-related activity at many corporations, very few have anything formal around it and usually all other performance-related groups get involved. And we need performance engineering kind of skills to investigate and fix performance problems in production.

It looks like that in the new generation of Web companies monitoring and capacity planning often included into ‘Site Reliability’, adding, I guess, some confusion to the already existing mess of terms and notions.

P.S. By the way, the only conference covering almost all topics mentioned above is CMG. Call for papers and workshops is opened now.

Dennis Drogseth’s post The Many Dimensions of User Experience Management (UEM) is very indicative of the main problem we have in performance: people thinking about many small specific performances, but we have just one PERFORMANCE. It depends on many different components and manifests itself in many different ways, but any attempt to decompose it results in silos and losing some important parts of the whole.

From the post: When we asked “What is your primary driver?” Better application performance and triage came in fifth, with only 13% of the votes. Employee productivity topped the list at 23%, followed by business competitiveness and/or revenue at 20%. Better support for services delivered over the network came in third, and brand protection and customer satisfaction came in fourth.

Well. Ask , for example, business users about JVM performance and it probably won’t get into the first hundred of issues they care about. Does it prove anything? No. They care a lot about it if they use J2EE systems, but just don’t know about it (except maybe a few most curious).

“Employee productivity” heavily depends on application performance. ” Business competitiveness and/or revenue” is related to application performance. “Better support for services delivered over the network” – not sure what it means, but performance also comes to mind. “Customer satisfaction” – performance is a pretty major component. And even with “brand” quite may be impacted by bad performance. Probably business users (and not only business) don’t care much about performance when it is good, but as soon as performance degrades, it immediately jumps on the top of everybody’s priority list.

I, of course, don’t want to say that performance is the main thing in business – if you don’t have any business, you may not be concerned with performance. But as soon as you do, application performance would impact all parts of your business. But you notice it only when it is bad (and usually it will happen soon if you don’t take care).

Then the post says: Similarly, when we wanted to understand which organizations or groups within IT and the business were behind UEM or QoE, the Help Desk/End User Support came in first, Customer Experience Management came in second, and Applications Management and Network Operations were tied at third and fourth place.

And when asked which organization is likely to DRIVE the overall QOE/UEM initiative, the first five groups were: Line of Business, Customer Experience Management, Process Management and Compliance Professional, Help Desk, and Service Management.
Applications Management came in seventh, one percentage point after Infrastructure Management!

Yeah, exactly proves the point: there is no organization/group responsible for performance today. Not sure what “Application Management” is (I don’t recall seeing such group – app admins?). And it is not surprising that people don’t put such group to drive such initiative – I guess perception is that such groups are groups of IT geeks doing something with computers, not caring about business, and starting to do something only when would be told by CIO to fix it (that, unfortunately, often is close to the truth).

How it relates to concept of Application Performance Management (which is rather concept for the moment)? It just proves that it doesn’t exist in practice (at least in its ideal form). Usually there is no organization responsible for it (as holistic concept, in conjunction with business).

What are end-user response times (what EUM monitors)? They are external symptoms of application performance. The only part of application performance end users care about. The tip of the iceberg. If we are saying that we want to manage application performance, would end-user response times part of it? I have no doubt it would. Otherwise the whole concept doesn’t make sense.

The post states: User Experience Management also has strong business impact, governance, service level and user productivity implications that transcend performance management. Yes, performance has “business impact, governance, service level and user productivity implications”.

So the data provided in the post, by my opinion, proves two things: business cares about performance a lot, but there is no any reliable structure in place to care about end-to-end performance.

Actually I am rather confused by the term User Experience Management. I understand what it is User Experience Monitoring or End User Experience (which usually used in the context of measuring response times). But how would you manage it? You may manage your application/systems which would improve response times. Unless you just saying that you want to use the name User Experience Management as an umbrella name covering all related to performance (including APM, Capacity Planning, etc., etc.) – which maybe an option, but it doesn’t look like it is used this way. Or maybe User Experience Management is used as a wider term including usability, UX (User eXperience), etc., which usually relate to UI design? If yes, then it indeed includes important factors not related to performance and only partially overlaps with APM – but then I am not sure why we compare EUM with APM.

Ian Molyneaux’s post The Case for the CPO brought the topic of a person responsible for performance to its extreme. Great idea, but… How far are we from there? Forget CPO, but just having a person (or persons) responsible for end-to-end performance and building up the process assuring such performance? See job posting – have somebody seen any position saying that we need a person to drive performance in our organization (and meant it)? I haven’t. All positions are for a specific silo team or for consulting. So it looks like it would be awhile until we see a more holistic approach to performance (whatever name would be used for it).

It looks like everything related to performance has multiple dimensions. Reading recently excellent posts A non-geeky guide to understanding performance measurement terms by Joshua Bixby and Building a High Performance Website by Phil Stanhope, I realized how many dimensions even a relatively simple term “response time” has. And, moreover, it looks like we don’t have a reliable way to measure the response time that would matter to end user (I guess something between “time to display” and “time to interactivity” depending on the site design, if follow the posts terminology). Both authors look at this rather from the front end / Web Performance Optimization (WPO) point of view.

Spending most of my time in performance testing, I’d guess that “response time” comes from load testing / active monitoring tools that are the main source of performance information (the “waterfall” approach of the WPO community quickly becomes popular – but I am not sure how many monitoring services use it). And in this case, “response time” is what the tool reports. What “response time” means in such case is heavily depends on the tool and its settings – and in many cases, I guess, it won’t be any of the metrics provided in the aforementioned posts (which, I guess, are standard in the WPO community – but they may be not easy to measure by load testing and enterprise monitoring tools). For protocol-based tools it would be probably the time of receiving all requests without any client-side activities (with many additional details of browser emulation- like caching, threading, keep-alive, compressing, etc.). For GUI-based tools it probably depends on what underlying mechanism the tool uses and how the script is designed. Quite often if you don’t set any specific checks it may report a success without full downloading and rendering (and when somebody say that a modern sophisticated site will load for 0.169 sec over the Internet it would be my first guess). Although, if scripted properly, it perhaps may measure the performance metric that matters (when the page would “be almost fully interactive”) by checking that the parts that matter are downloaded and rendered (that probably can’t be done without manual scripting / analysis).

That brings an interesting question about Application Performance Management (APM): what End-User Experience Monitoring (EUM) a.k.a. Real-User Monitoring (RUM) measures? EUM/RUM is considered as an integral part of APM (and definitely should be), but may measure pretty different things depending on the approach to measure it. And as I mentioned above, it probably won’t be the actual end-user experience – but only its approximation by another metric (different for different tools).

Only thing that often saves us from all this complexness is , as often happens in performance, that in many cases it doesn’t matter. All of the metrics are just close enough from the practical point of view. In old good times of plain html the main part was getting response from the server, the client-side part was fixed and usually small. So it wasn’t said much about different kinds of response times in the past. The situation is changing now: the front-end time becomes significant (see the Performance Golden Rule by Steve Souders, keeping in mind that it is based on front pages mainly) and now it looks like we can’t ignore the differences between response times anymore.

My guest post A Few Thoughts about APM (Application Performance Management) and Its Future on the Monitis blog.

When I created my site as a collection of performance-related links and documents in 2004, I grouped links somewhat arbitrary, just to avoid “analysis paralysis”, hoping to get back soon and polish as needed. It is interesting that I haven’t changed much in grouping for these seven years (definitely many things changed, many changes are long time due, but with main grouping of information I wasn’t able to improve much). Whatever links I added, they mainly fit one (or few) existing category. And just now I realized that we have a new information category – Application Performance Management – which doesn’t fit in any existing category. I had a category for APM tools from the beginning – they were around for a while – but not for generic APM information (something beyond talking about just tool features). And finally I put together a list of great information sources into a new group, Application Performance Management:

Application Performance Engineering Hub

Application Performance, Scalability, and Architecture blog from Dynatrace

The Performance Management section of The Virtualization Practice

APM Digest

Correlsense blog

App Signal blog from AppDynamics

Catchpoint’s Blog

Seriti Consulting Blog the Web Operations and Management Specialists, by Stephen Thair

Many of them existed for a while, but it looks like the quantity finally got into a new quality and we see a new discipline emerging (instead of a marketing term to promote a special kind of tools). It is definitely related that with new technologies, such as virtualization and cloud computing, traditional resource monitoring is not enough anymore and there is a need monitor on application and service levels. Some mentioned above blogs are from tool vendors, but they provide great content far beyond discussing the tools.

Bernd Harzog’s post Why is Application Performance Management so Screwed Up? started a lot of discussions on the Internet. The post is a very good list of existing issues you may face when you try to use APM tools. I’d add one more – overheads. At least for the first generation, the claim that you may use APM in production worked only if you did very selective monitoring.

My view of APM is that first generation of APM tools so well described by Bernd was very immature. Not that something was explicitly wrong with the APM in general – really wrong was the drastic contrast between what the tools actually could do and marketing promises of tool vendors. The vendors talked more about the APM vision and how the APM tools are supposed to work – but not about the exact things these tools are able to do. Which you figured out in the best case after you spent a few days evaluating the product.

If check Garter Magic Quadrant for Application Performance Monitoring or my list of tools, it is clear that the market is very crowded and not well defined. There is no good criteria you can compare tools and different tools may actually do pretty different things, although it may be difficult to understand reading about them on vendor’s sites.

However I’d say that now we are getting the second generation of APM tools which are much closer to the APM promise for some technologies. I don’t want to list names here and separate “first” and “second” generations. I’d guess that some “first” generation tools might advance to the “second” generation if kept progress – but, as I said, it is difficult to say without actual evaluation of the tools. So I am hearing a lot of stories that people were able to successfully implement APM for system X using tool Y without many problems.

Still you doesn’t have a product which will do APM across all platforms and system if you have a full zoo of different technologies some of which are older than most of your IT employees (as many large corporations do). And don’t believe to anybody who tells you that they can do that. Still it looks like you can do it now for more systems with fewer problems – and start reaping the benefits of APM. Actually I don’t see any other alternative to APM in the long run – although it is a topic for a separate post. But be aware of all points mentioned in Bernd’s post – and check if the product you are going to use doing what you need in the way you want.

P.S. Just before posting noticed another Bernd Harzog post where he shares his view of next generation APM products.