Posted How Do We Measure Computer Resources? on Application Performance Engineering Hub. It looks like an important issue for the high-tech industry for me – it is a pity that it continues to be unnoticed.
See my How Response Times Impact Business? post in 2011 Performance Calendar.
When I created my site as a collection of performance-related links and documents in 2004, I grouped links somewhat arbitrary, just to avoid “analysis paralysis”, hoping to get back soon and polish as needed. It is interesting that I haven’t changed much in grouping for these seven years (definitely many things changed, many changes are long time due, but with main grouping of information I wasn’t able to improve much). Whatever links I added, they mainly fit one (or few) existing category. And just now I realized that we have a new information category – Application Performance Management – which doesn’t fit in any existing category. I had a category for APM tools from the beginning – they were around for a while – but not for generic APM information (something beyond talking about just tool features). And finally I put together a list of great information sources into a new group, Application Performance Management:
Application Performance Engineering Hub
Application Performance, Scalability, and Architecture blog from Dynatrace
The Performance Management section of The Virtualization Practice
App Signal blog from AppDynamics
Seriti Consulting Blog the Web Operations and Management Specialists, by Stephen Thair
Many of them existed for a while, but it looks like the quantity finally got into a new quality and we see a new discipline emerging (instead of a marketing term to promote a special kind of tools). It is definitely related that with new technologies, such as virtualization and cloud computing, traditional resource monitoring is not enough anymore and there is a need monitor on application and service levels. Some mentioned above blogs are from tool vendors, but they provide great content far beyond discussing the tools.
See my Performance Requirements – Do we need a better word? post on Application Performance Engineering Hub
Join your peers to discuss the dynamics of performance testing, planning and experimentation in today’s fast-paced, impatient world. At the Computer Measurement Group conference (CMG’11) in Gaylord National, Washington DC area, on December 7th, Wednesday, at 6:30pm in the Annapolis 1/2 room we will have an open Birds of a Feather session “New Approaches to Performance Testing: An Open Discussion on Plans, Experiments and Points In-Between”.
Everybody is welcome!
The idea to discuss the topic was suggested by James Pulley during a Linkedin discussion in the LoadRunner group. It turned out that there are quite different views of the role of experimenting in performance testing.
We hope to get a lot of CMG attendees as well as many performance testers from the area there. Please e-mail me to apodelko at yahoo dot com if you have any question or going to attend – just to have an idea how many people will attend.
Bernd Harzog’s post Why is Application Performance Management so Screwed Up? started a lot of discussions on the Internet. The post is a very good list of existing issues you may face when you try to use APM tools. I’d add one more – overheads. At least for the first generation, the claim that you may use APM in production worked only if you did very selective monitoring.
My view of APM is that first generation of APM tools so well described by Bernd was very immature. Not that something was explicitly wrong with the APM in general – really wrong was the drastic contrast between what the tools actually could do and marketing promises of tool vendors. The vendors talked more about the APM vision and how the APM tools are supposed to work – but not about the exact things these tools are able to do. Which you figured out in the best case after you spent a few days evaluating the product.
If check Garter Magic Quadrant for Application Performance Monitoring or my list of tools, it is clear that the market is very crowded and not well defined. There is no good criteria you can compare tools and different tools may actually do pretty different things, although it may be difficult to understand reading about them on vendor’s sites.
However I’d say that now we are getting the second generation of APM tools which are much closer to the APM promise for some technologies. I don’t want to list names here and separate “first” and “second” generations. I’d guess that some “first” generation tools might advance to the “second” generation if kept progress – but, as I said, it is difficult to say without actual evaluation of the tools. So I am hearing a lot of stories that people were able to successfully implement APM for system X using tool Y without many problems.
Still you doesn’t have a product which will do APM across all platforms and system if you have a full zoo of different technologies some of which are older than most of your IT employees (as many large corporations do). And don’t believe to anybody who tells you that they can do that. Still it looks like you can do it now for more systems with fewer problems – and start reaping the benefits of APM. Actually I don’t see any other alternative to APM in the long run – although it is a topic for a separate post. But be aware of all points mentioned in Bernd’s post – and check if the product you are going to use doing what you need in the way you want.
P.S. Just before posting noticed another Bernd Harzog post where he shares his view of next generation APM products.
Oracle Application Testing Suite 9.3 was released some time ago. It is available for download (subject to OTN License Agreement). Some new features and updates in this release are described in the press release.
At last I have updated my site a little. It is, of course, long time overdue for re-design – but, with all things going on, it doesn’t look like I get to this soon. Meanwhile I decided to get more mileage from the update and list links I added to the site here separately. These links may be not new, they are just links I stumbled upon relatively recently and decided that I need to add them to my collection of performance links.
Application Performance Engineering Hub
Performance By Design by Mark Friedman
Book of Speed by Stoyan Stefanov
Thinking Clearly about Performance by Cary Millsap
Performance Engineering Parables by Chris Papineau
Seriti Consulting Blog the Web Operations and Management Specialists, by Stephen Thair
High performance Web Sites site and blog by Steve Souders
Best Practices for Speeding Up Your Web Site by Yahoo
Performance Testing portal
Testing Experience Magazine the publisher, Díaz & Hilterscheid, organized multiple conferences across Europe
Gartner Magic Quadrant for Application Performance Monitoring, 2010
De-Mystifying Software Performance Optimization by Paul Del Vecchio
1202 Performance Performance by Design, Andrew Lee’s site
Performance Testing Professional
New York Web Performance Group and links to other Web Performance Groups
WPO Links (thanks to Alon’s write-up):
Yslow analyzes web pages and suggests ways to improve their performance
Smush.it “smushes” images to make them smaller without losing any noticeable quality.
Show Slow is an open source tool that helps monitor various website performance metrics over time. It captures the results of YSlow, Page Speed and dynaTrace AJAX Edition rankings and graphs them, to help you understand how various changes to your site affect its performance
Mobile Perf Bookmarklet – According to Steve Souders, “Web development on mobile devices is especially challenging. The debuggers and profilers we use on the desktop aren’t available. Bookmarklets are a good alternative.”
Page Speed is an open-source project started at Google to help developers optimize their web pages by applying web performance best practices
HTTParchive.org a permanent repository of web performance information such as size of pages, failed requests, and technologies utilized.
Whichloadsfaster.com Pages compete head-to-head in your browser to see who’s fastest
Webpagetest.org tests website performance
Mod_PageSpeed is an open-source Apache module that automatically optimizes web pages and resources on them
SOASTA launched CloudTest Lite – a free edition of their performance testing solution. Basically, they give it free for up to 100 users. A serious move for sure. It should heat up the load testing tool market. It may work indeed – I guess they don’t have many paid customers in that range anyway, looks like CloudTest’s sweet spot is when you need a very large number of users. I am very interested to see how it will turn out.
Several rosy reviews were posted, for example, CloudTest Lite – A Game Changer in the Performance Tool Market by Scott Barber and SOASTA CloudTest Lite Hands-On by Bernard Golden.
As I already mentioned, it indeed is pretty interesting. However, I’d say that we need add some skepticism to be more realistic.
First, it is not the first and absolutely unique move in load testing tools. I recall a few somewhat similar moves before which then quietly disappeared. Well, I don’t remember what limitations were (maybe a little bit more restrictive). And the companies were not the leaders of the market. Moreover, there is a list of 50 open source load testing tools on opensourcetesting.com and some, like JMeter and OpenSTA, are pretty mature. Yes, open source in load testing area was not so successful as in other areas. Especially analysis is weak in most of these tools (if existent at all).
Second, releasing is just the first step. The challenge for SOASTA would be how they support a large number of non-paying users (although, of course, for a promising start-up the number of customers may be important by itself). The community maybe can help with “how-to” questions, but implementing, let’s say, enhancement requests is up to the SOASTA team. And the number of such requests may be pretty high as people start to use it with different applications.
For example, it looks like we can’t specify transactions during recording in CloudTest for the moment. Well, what I am supposed to do with a script with a few hundred identical requests in it (AJAX type, differ by incomprehensible http body content)? Track delays in the scripts and try to correlate them with recording steps? Not exactly my understanding of quick and easy.
Scott writes in his review “it is free from now until the sun explodes” Hmm… I’d rather prefer to hear this from the SOASTA team. Well, even if the SOASTA team is completely devoted to this edition, nobody can guarantee that SOASTA won’t be acquired and who knows what acquirer decides to do with the freemium edition…
Yes, Scott is not easily getting excited. Last time, as far as I remember, Scott got excited about a load testing tool when Microsoft released their tool as part of Visual Studio back in 2005. See, for example, the discussions around my old posts VisualStudio 2005 and Load Testing and Scripting Language in Performance Tools. Well, Microsoft didn’t live up to its promises and I haven’t heard about their load testing tools for a while (my understanding is that it is not dead, but doesn’t play any noticeable role). But who knew that Microsoft was losing its grip?
By the way, returning to this old post about scripting languages, I don’t object the idea of GUI-based load testing tools (I mean when script is represented by a kind of graphical tree or something like this) if we can extend it with code (as in CloudTest with JavaScript) or switch between tree and script view (as in Oracle Application Testing Suite). My concern was (and is) if you can’t extend you recording with code at all – I still believe that it limits the area of application significantly.
Anyway, it looks like we have several interesting developments in the load testing tool market that may be beneficial to the community. CloudTest and its Lite version are definitely on the list. LoadRunner AJAX True client may be introducing a new paradigm in load testing (or promoting it if follow e-Valid blog). Oracle Application Testing Suite (former Empirix) is practically a new product and is getting traction [at least in the Oracle Universe].
