Alex Podelko

Reading Who should be on your WPO team? reminded me so many things I read before like, for example, creating a SPE team or creating Performance Center of Excellence.

Looks like we got a completely new area of performance engineering – Web Performance Optimization (WPO), with its own terminology, approaches, experts , Web Performance meeting groups, Velocity conference, and, perhaps, even new load testing tools like CloudTest (according to my impression, it is more beneficial for WPO projects). WPO actually was around for a while (looks like the first Velocity conference was in 2008), but only recently, after attending a couple of New York Web Perf events, I realized that it became a separate discipline. I guess the appearance of this new movement concentrated on the web performance means that we get a pretty mature industry of very scalable web sites delivering sophisticated content.

Well , the history of performance engineering looks like a series of waves (for me, although my knowledge of its history is limited, especially for the period before I got involved). Computer Measurement Group (CMG) was organized in 1975 as an organization of performance analysts and capacity planners. Dr. Connie Smith book “Performance Engineering of Software Systems” book was published in 1990 created the Software Performance Engineering movement.

Distributed systems brought new wave of performance engineering based around load testing. Perhaps because there was not much instrumentation available and only way to make sure that the system performs was to apply load. It looks like the first version of LoadRunner was shipped in 1989. But when I first time got involved into load testing in 1997 with SQL Bench (SilkPerformer’s ancestor), it was still far from what we expect from load testing tools now. The latest wave was probably Application Performance Management with a large array of tools promising application instrumentation (visibility in what is going on inside applications).
It is interesting that all these overlapping areas never completely merged. This is probably the reason why we have such discrepancy in performance terminology because every group often started terminology from a scratch (while others still used old terminology).

And now we get Web Performance Optimization (looks like the term was coined by Steve Souders). While WPO looks like a separate discipline, I’d rather placed it as a part of overall performance engineering. You still have a back end in most cases – and while the back end is mentioned in the WPO presentations, it sometimes looks like authors mention something trivial. Well, it is not, even for most web sites, not to mention large banks and insurance companies with many tiers of sophisticated systems in the back – and for the end-user performance you need to consider all together. Downplaying “back end” is probably as wrong as downplaying “front end” (which, working mostly with business applications, I am definitely guilty – well, historically load testing concentrated on the server performance). The importance of each component depends on the system. In my opinion, performance principles are much more generic that the details of specific technologies. Most of performance engineering experience may be applied to any technology (you, of course, still need to learn something about this new technology too).

So, while it is very promising and exciting that we get a new wave of people dedicated to performance, it is a little sad that it looks like it often gets started from a scratch inventing new terminology and ignoring what existed before. For me it would be better if we get all these waves together to enrich each other with the area of performance engineering they specialize in. Of course, there are some interaction – well, you need to work together in a way to ensure systems’ performance – but it still looks like every wave tend to stay somewhat separate, cultivating their own terminology, approaches, and events.

The Computer Measurement Group (CMG) calls for papers and presentations for CMG’s 37th International Conference to be held in Washington, DC, December 5th through 9th, 2011. The 2011 CMG conference will cover all areas of systems management, including but not limited to: measurement and tuning, modeling and statistics, capacity planning, performance engineering and load testing, management and reporting, as well as the latest developments in the overall field of computer performance evaluation.

Of special interest for CMG’11 will be presentations relating to this year’s theme of Optimizing IT Infrastructures: Management, Virtualization, and Cloud Computing.

CMG is the source of unbiased and objective expert information and practical, real life experiences across all computing platforms in the computer industry for over 30 years. Share your knowledge and experiences: write a paper and submit it for presentation at CMG’11.

Submissions at introductory through advanced levels are welcomed and encouraged. We especially look for papers based on user experiences. These often provide value that can be immediately applied in one’s own environment.

Abstracts should be submitted by May 15, 2011. Papers are due by June 10, 2011.

All paper and presentation submissions will be evaluated through a blind peer-referee process and will be categorized as Introductory, Tutorial, Advanced, or User Experience. Mentors are available for writing assistance and should be requested early in the writing process. Editorial assistance is provided for all accepted papers.

For more information on the paper submission process please go to:
http://www.cmg.org/conference/paper-system.html

Please send questions to the CMG’11 Program Chair at cmgpc@cmg.org

It looks like the competition on the load testing tools market heats up again. And, of course, new players need to differentiate themselves. I started to see statements that zero-scripting tools is a new word here. See, for example, the Agile Thinking: A New Approach to Performance Testing paper by Graham Parsons.

While I completely agree with the first part (compare with my Agile Performance Testing article), I am rather confused with the second part. I don’t see why we can’t use well-established tools like LoadRunner or JMeter (if we speak about free tools) in agile development. When you know the tool, scripting is usually not the most difficult part of your performance project.

The zero-scripting approach (if I understand it correctly) looks for me rather limited to simple web sites (and if you have such sites – you don’t need to do much in LoadRunner either). How do you handle complex correlation and parametrization often required by today’s rich web clients? And as you start to introduce ways to do it through, for example, graphical interface – you start to create a proprietary way to do this.

Also there is very interesting Fred Beringer’s post. I completely agree the main idea of the post. Performance engineering is much more than just a gate-keeping function. The terminology is somewhat vague here, but I usually use ‘performance engineering’ when I want to stress that it is more than “gate-keeping”. ‘Wisdom’ sounds a little bit too pathetic to me and the definitions of knowledge and wisdom may be argued about, but the idea is well formulated and definitely valid.

But I still believe that you may be in the ‘wisdom’ business with whatever tools are available to you. I, of course, don’t know all the features of CloudTest (the materials I read were too marketing-oriented to figure out what are real advantages). Tools help you a lot, but still it is people who are in knowledge / wisdom business, not tools. Even the best tool won’t do the work for you – you need to know how to use it, how to interpret it, etc.

And I still don’t see that drastic difference with “traditional” load testing tools. I guess CloudTest has some advantages. Listening to a webinar I realized one – they can quickly and automatically deploy agents across as many cloud servers as needed. Well, this is a significant advantage when you work in the cloud or want to run a really large-scale test (and an interesting variable here is how many users you can simulate from one server). Probably there are other advantages. But the basic principles still look the same.

Going to how Fred describes the difference:

1. Scale. Maybe CloudTest has some advantages on the high-end. It usually is a problem when you want to simulate many thousands users. But why “traditional tools limit themselves to testing behind a firewall”? Nothing, up to my knowledge, prevents you from installing them outside firewall. Yes, most performance testing in large corporations is done in a lab inside firewall – but, I guess, it is because most performance testing they do is for internal applications. And nothing actually prevents you from generating traffic from different geographies. I often did it myself by installing LoadRunner agents in remote locations.

I am not sure that I got Fred’s point in the statement: Most traditional tools, including Loadrunner, use manual dynamic session or hard-coded rules. You end up with significant repetitive work and long support delays. . At least what I saw about templates during a CloudTest webinar, looks pretty similar to what is used in “traditional” tools.

2. Speed. Why “proprietary scripting language”? LoadRunner uses ANSI C for Web protocol (and other standard languages for other protocols), Oracle Application Testing Suite uses Java, etc. Sure, they use a lot of proprietary functions there, but languages are standards and you can add your code or use external functions when you need to extend script functionality. Yes, it requires some skills – but this is required only in really complicated cases. In simple cases, you don’t care much about what language is used in the script – just record it and play it back. Anyway, looks like SOASTA uses JavaScript as a scripting language.

3. Affordability. Well, LoadRunner is not the cheapest product on the market for sure. But whatever licensing model is used by CloudTest, it is probably still a noticeable sum. While we have multiple free products around. Some, like JMeter, are mature enough.

To summarize, it quite could be that new tools have some advantages over old players. Still I don’t see a revolution yet. Some progress for sure (especially after a long period of stagnation in this market), but not a revolution. Well, maybe I just not good in marketing – but I wish to see some webinars / papers for technical audience explaining real difference and advantages.