Alex Podelko

CMG’12 is an annual conference organized by Computer Management Group – a volunteer organization of professionals specialized in performance, capacity, and IT service management. This year it is held in Las Vegas, December 2-7, 2012.

Why I love CMG, spend a lot of my time organizing and promoting it, and coming there every year (sometimes on my own)? Well, because I believe that it is the best (and actually the only) conference on performance and capacity, the main topic of my interest for the last fifteen years. There are many conferences on specific topics. For example, the Velocity conference, devoted to web performance, is significantly larger and more popular – but it is still devoted mainly to single-use web performance, leaving all other performance and capacity questions to CMG. Let me share some of my excitement – of course, from my personal point of view (there is plenty of other highlights, but I am mentioning only the ones that are close to my heart).

This year the conference covers all aspects of performance (well, almost all – performance is so sophisticated subject that there is always much more to learn) from Web Performance Optimization (the conference opens by the keynote by Patrick Meenan, a web performance Google guru and the creator of WebPagetest) to mainframe performance (and everything in between).

The conference starts with a half-day workshops – see here the description. In addition to workshops, there are CMG-T sessions during the whole conference. Each CMG-T class spans 2 or 3 session spots, so it could easily be considered as a workshop or a training class. All led by renown experts with tons of experience, you hardly would get anybody even remotely close if you engage in a typical vendor class (not to mention a unique vendor-neutral or vendor-agnostic perspective you hardly find anywhere else). You have the CMG-T track through the whole conference and every one of them is a gem:

• Capacity Planning by Ray Wicks
• z/OS Basics by Glenn Anderson
• Java Performance Analysis and Tuning by Peter Johnson
• Model and Forecasting Basics by Dr. Michael Salsburg
• Network Performance Management by Manoj Nambiar
• Windows System Performance Management and Analysis by Jeffry Schwartz
• Using SAS to Communicate Your Message by MP Welch

CMG’12 has 4 keynote/plenary session and almost a hundred regular track sessions going on from mid-Monday to mid-Friday. The conference is 5 tracks wide. One track, as I already mentioned, is CMG-T 101– type classes (with 301-depth). Others four tracks shared between five subject areas: Performance Engineering and Testing, Capacity Planning, Application Performance Management, IT Service Management, and Hot Topics. It is difficult to list all highlights – too many. While I know many great presenters and am fascinated by many topics, commenting every single one would take too much time and space. Probably you just need to look at agenda – there are three different views: preliminary agenda (overview, a day on a page), a list of abstracts in a single pdf document and search/scheduler (click on the abstract number to see the abstract).

One track on Wednesday is a Michelson award track. CMG is presenting Michelson award since 1974 (if you wonder, Albert Abraham Michelson was known for his technical accomplishments in measuring the speed of light and for his role as teacher and inspirer of others – and measuring is the key to performance). This year we will see many Michelson winners presenting: Dr. Connie Smith, the founder of Software Performance Engineering, Dr. Daniel Menasce, the author of many great books about performance and capacity planning, Adam Grummit, the author of the great Capacity Management book (ITSM Library) and the CMG president, Dr. Pat Artis, Bruce McNutt, and Dr. Michael Salsburg.

I believe that the main advantage of attending CMG is networking with best world experts in almost all areas of performance and capacity. Nowadays you can find all technical information on the Internet, but there is no substitution to face-to-face conferences to learn how to use it and what were people experiences, and, of course, to see the whole picture. Especially in performance: performance is the result of every design and implementation detail and you need to be learning all the time to keep up with coming challenges.

I am presenting there too: Load Testing: See a Bigger Picture on Thursday and
Performance Requirements: the Backbone of the Performance Engineering Process on Friday. Nothing comparing to other CMG’12 highlights, but I hope to trigger discussions around these two very important topics.

And, of course, it is Las Vegas – and Rio’s rate is $55 per night until November 14th. See you there!

Reading about performance modeling / simulation and system sizing, you often see two completely opposite views of the subject. Either authors describe in detail how you can model performance using some math and you may feel that as soon as you comprehend that math, you won’t have any problem with modeling. Or authors say that it is a black magic and you’d better stay away from it or do it in a minimal way with simple trending (while you probably won’t see that view in serious books, it is often can be seen in Internet discussions).

The truth, as usual, is in the middle. Modeling is a very helpful and works well if you use it properly and understand its limitations. And there are two main challenges here that rarely get highlighted – while everybody who wants to approach the topic should understand them clearly.

The first challenge is that modeling works well for known resource limitations. You should know these limitations in advance (and how your system uses that limited resource – which is also a challenge, but more technical one). For example, if your system is processor-bound and you know how much cpu it takes per transaction, you may build a rather simple and pretty accurate model (using queuing theory or even something pretty simple – for example, if you stay away from heavy cpu utilization, linear model may work well with multi-processor systems).

But that model would never tell you when you run out of another resource and run into another kind of bottleneck until you build it into the model. And beyond a few common resources (processor, memory, disk, network) and explicitly introduced throttling, you usually don’t know about bottlenecks until you run into them. This is the primary reason that results of your model (which may be perfect from the mathematical point of view) are not reliable if you model significantly higher load than you tested / validated – as far as there is a high probability that you run into another bottleneck you are not taking in consideration now. However, the model would provide the best possible case (which turns true when you fix all other bottlenecks you didn’t take in consideration at the moment of modeling), which is important information by itself. A model would also be very useful to see if the system behaves up to expectations – or there are internal issues degrading performance and preventing scalability (that may be not so trivial to catch in complex systems).

Another challenge is a lack of performance-related metrics of hardware to use in modeling. You can find detailed hardware specifications, but they won’t tell you how fast your systems would work on this hardware. As far as I understand, the only relatively objective approach (without testing the real system on the real hardware – which is, of course, the best) is to use existing benchmark results to compare performance (keeping in mind that they represent results of this specific benchmark, not your systems). Most serious commercial modeling tools come with a library of hardware configurations and their performance metrics, allowing what-if performance analysis. It looks like keeping such libraries is a pretty time-consuming task and their quality may differ. Such a library is usually a major advantage of commercial modeling tools in comparison with free or inexpensive modeling tools (which may be quite good from the mathematical point of view, but you need to provide all numbers yourself).

IDC made an interesting move here introducing QPI (Qualified Performance Indicator) as a part of IDC’s Server Decision Suite Metrics (free 30-days trial available). A kind of independent performance library that may be used for proper performance modeling / sizing (and, as far as I understand, going well beyond performance, integrating this information with other IT-related metrics such as price, power, and size – it should be a very interesting optimization task to find the best hardware configuration based on all these metrics).

Solving Enterprise Applications Performance Puzzles : Queuing Models to the Rescue by Leonid Grinshpan is a pretty interesting book about application of queuing models to solving enterprise performance and I believe the book fills a few gaps in practical application of queuing theory. Another good name for this book could be “Building queuing models by example”.

I spent a lot of time trying to use queuing models to solve practical performance issues and would testify that it is pretty challenging. There are a few areas where it was developed a little further (for example, around capacity planning of existing systems), but if you trying to do something else – you won’t find much help. You have a lot of books about systems performance, you have a lot of books about queuing theory with simple examples, but not much in between to solve practical tasks. And here Leonid’s book may help, especially if you are new in this area.

Chapter 1, Queuing Networks as Applications Models, is an introduction into the topic. It discusses how queuing theory may be used to model enterprise applications. A lot of analogues are used to introduce the subject.

Chapter 2, Building and Solving Application Models, is an overview of the whole process, including short discussions about essentials of queuing theory and using of tools to solve models.

Chapter 3, Workload Characterization and Transaction Profiling, discusses what input data for models are and how to gather them.

Chapter 4, Servers, CPUs, and Other Building Blocks of Application Scalability, discusses scalability, bottlenecks, how to identify bottlenecks and ways to fix them (mostly on CPU and I/O examples).

Chapter 5, Operating System Overheads, discusses main components of operating systems, where overheads come from, how to measure them, and their impact on transaction time.

Chapter 6, Software Bottlenecks, is devoted to software bottlenecks, which are rarely discussed in application to queuing models – while in practice software bottlenecks happen all the time. Memory bottlenecks and thread optimizations and their modeling are discussed in details. Multiple other software bottlenecks are also reviewed.

Chapter 7, Performance and Capacity of Virtual Systems, is an overview of performance issues related to virtualization , their explanation with queuing theory, and a methodology of virtual machine sizing.

Chapter 8, Model-Based Application Sizing: Say Good-Bye to guessing, explains why to use model-based sizing and discusses it step-by-step from gathering input data to model deliverables and what-if scenarios.

Chapter 9, Modeling Different Application Configurations, discusses several specials cases including geographical distribution of users, cross-platform modeling, remote terminal services, load balancing, and parallelization of transactions.

The book covers a lot of topics. However, to avoid disappointments, I’d like to point out what this book is not:

- It is not a textbook about queuing theory. The section 2.2 Essentials of Queuing Networks Theory has 5 pages in it.

- It is not a book about tools to solve queuing models. Available tools are listed and there are references, but they are just mentioned as a way to solve models (with one tool used as an illustration of the process). You don’t need to know any tool to read the book (but you will need one when you try to solve your own models).

- It is not a comprehensive book about enterprise application performance. There is plenty of important information and practical recommendations about enterprise application performance in the book, but it is shared as needed to build models and analyze their results.

So the book is exactly what the title says: a practical book about building queuing models to investigate enterprise applications performance issues.

I worked with Hyperformix products for rather a short period of time, but followed the company for a long time. Hyperformix tried to implement a lot of very interesting things in performance engineering.

The original Hyperformix product, Hyperformix Optimizer, has a modeling language and you may build very sophisticated models with it, including models of new systems (after, of course, you learn the language). Optimizer uses simulation – so while you may build as complex models as you want, it may take a while to run such models. Although Optimizer was rather weak on data collection. It had interfaces to many monitoring tools, but it was a challenge to collect information from a large zoo of servers.

Many other modeling tools, like Teamquest, came from the enterprise capacity planning point of view. You set agents on all servers, collect and report information, and build analytical models based on the collected information. It is not easy to build more sophisticated models or models for new systems. But if the goal is to monitor a zoo of servers, report results, and build some predictions for the existing systems, such tools are a good choice.

Teamquest, for example, didn’t have any meaningful methodology for modeling non-existing systems until my colleague Leonid Grinshpan came up with Multi-tiered Applications Sizing Methodology Based on Load Testing and Queuing Network Models (a CMG’08 paper).

I guess that Hyperformix did the best job to integrate modeling and load testing. For example, see Moving Beyond Test and Guess – Using Modeling with Load Testing to Improve Web Application Readiness by Richard Gimarc, Amy Spellmann, and Jim Reynolds (as well as other Richard Gimarc’s papers – just do a search through CMG proceedings, papers up to 2007 are opened to public, free registration required). For some time, Mercury even re-sold Hyperformix Optimizer as Mercury Capacity Planning.

A group of Hyperformix authors published a very good practical book on performance engineering Fundamentals of Performance Engineering; You can’t spell firefighter without IT. For those without deep math background it may be a good book to start learning modeling / performance engineering from (it has some Hyperformix-inclined stuff, but not much).
And at one moment Hyperformix provided a performance engineering certification – nothing is available in this area yet until now. But they did it too Hyperformix-inclined to interest somebody outside the rather narrow circle of Hyperformix clients. I guess such certification should be vendor-independent to succeed (at least until consolidation starts in this area).

Later Hyperformix created another product, Capacity Manager, which looks similar to other capacity management products (like Teamquest) in goals and methods (somewhat admitting that Optimizer, as it is, is not the best tool for enterprise-level monitoring and capacity planning). And looks like Capacity Manager became the main Hyperformix product even before CA acquisition (so many performance engineering initiatives were forgotten) and Capacity Manager was the main goal of the CA acquisition. Although, of course, it is just as I see it from outside.