Comments for Alex Podelko http://alexanderpodelko.com/blog Alex Podelko's blog about Performance Engineering and other subjects Fri, 11 May 2012 08:12:02 +0000 hourly 1 http://wordpress.org/?v=3.3.2 Comment on Load Testing: What Tool to Choose? by Richard Bishop http://alexanderpodelko.com/blog/2012/05/10/load-testing-what-tool-to-chose/comment-page-1/#comment-3887 Richard Bishop Fri, 11 May 2012 08:12:02 +0000 http://alexanderpodelko.com/blog/?p=257#comment-3887 Great summary Alex! I've been a LoadRunner user / evangelist for 12-13 years and this is an excellent article which compares it nicely with the competition. If you only want to test web/HTTP, you're spoilt for choice. As soon as you want the more "exotic" protocols you fall back on one or two alternatives. Greg Moore's comments about reporting and analysis are spot on as well. People assume that because you can generate load on an application, you're testing it. Unless you can produce well-informed, timely reports about your tests it often isn't worth testing. Thanks for sharing this. All the best, Richard Great summary Alex!

I’ve been a LoadRunner user / evangelist for 12-13 years and this is an excellent article which compares it nicely with the competition. If you only want to test web/HTTP, you’re spoilt for choice. As soon as you want the more “exotic” protocols you fall back on one or two alternatives.

Greg Moore’s comments about reporting and analysis are spot on as well. People assume that because you can generate load on an application, you’re testing it. Unless you can produce well-informed, timely reports about your tests it often isn’t worth testing.

Thanks for sharing this.
All the best,
Richard

]]> Comment on Load Testing: What Tool to Choose? by Greg Moore http://alexanderpodelko.com/blog/2012/05/10/load-testing-what-tool-to-chose/comment-page-1/#comment-3885 Greg Moore Thu, 10 May 2012 20:05:27 +0000 http://alexanderpodelko.com/blog/?p=257#comment-3885 Alex, Thank you for at least mentioning monitoring and results analysis. There is a third component that is all too often overlooked or simply just not there. I think its part of the reason that Loadrunner with all its faults, excessive costs, and learning cliff is still being used. That is reporting and producing meaningful reports and charts that can be given to management that dont require and engineer to explain them. SOASTA, imho, is one of the few aside from HP that has a good monitoring, analysis and reporting capability. Jmeter, Selenium, and several other free or open source solutions just don't have even rudimentary report production ability. They might produce something I'd hand another engineer but not a Senior manager or VP. Alex,
Thank you for at least mentioning monitoring and results analysis. There is a third component that is all too often overlooked or simply just not there. I think its part of the reason that Loadrunner with all its faults, excessive costs, and learning cliff is still being used. That is reporting and producing meaningful reports and charts that can be given to management that dont require and engineer to explain them. SOASTA, imho, is one of the few aside from HP that has a good monitoring, analysis and reporting capability. Jmeter, Selenium, and several other free or open source solutions just don’t have even rudimentary report production ability. They might produce something I’d hand another engineer but not a Senior manager or VP.

]]> Comment on Load Testing: What Tool to Choose? by Jeff Lucas http://alexanderpodelko.com/blog/2012/05/10/load-testing-what-tool-to-chose/comment-page-1/#comment-3883 Jeff Lucas Thu, 10 May 2012 16:47:21 +0000 http://alexanderpodelko.com/blog/?p=257#comment-3883 Alex - Thanks for the blog, a good summary. Under the "Programming" category, we successfully used the "grinder" application (http://grinder.sourceforge.net/) to emulate a Java thick-client application. It uses Jython (Java for Python) and interacts with the client Jar files to provide a top-level client emulation. Most discussions of performance testing only discuss the thin-client web solutions which leaves those with thick client applications in the cold. Alex – Thanks for the blog, a good summary. Under the “Programming” category, we successfully used the “grinder” application (http://grinder.sourceforge.net/) to emulate a Java thick-client application. It uses Jython (Java for Python) and interacts with the client Jar files to provide a top-level client emulation. Most discussions of performance testing only discuss the thin-client web solutions which leaves those with thick client applications in the cold.

]]> Comment on Performance Dimension of Information Technology by Software Testing Novice http://alexanderpodelko.com/blog/2012/04/16/performance-dimension-of-information-technology/comment-page-1/#comment-3873 Software Testing Novice Wed, 02 May 2012 01:44:46 +0000 http://alexanderpodelko.com/blog/?p=246#comment-3873 I don't think the need for load testing is going away, but it is changing and becoming more complicated. Surely, with so many components now involved in the architecture of a modern application, considering not only software architecture with different mash-up components and network architecture, considering the cloud (and placing faith in the cloud to handle load dynamically), not to mention other growing variables, has in a way taken away a single point of vulnerability. Without that, risk is spread out, and points of failure are spread out. But then when there is a performance problem, it is also harder to troubleshoot and solve. I don’t think the need for load testing is going away, but it is changing and becoming more complicated. Surely, with so many components now involved in the architecture of a modern application, considering not only software architecture with different mash-up components and network architecture, considering the cloud (and placing faith in the cloud to handle load dynamically), not to mention other growing variables, has in a way taken away a single point of vulnerability. Without that, risk is spread out, and points of failure are spread out. But then when there is a performance problem, it is also harder to troubleshoot and solve.

]]> Comment on Performance Testing and Optimization for the Cloud by QA Thought Leaders http://alexanderpodelko.com/blog/2012/03/29/performance-testing-and-optimization-for-the-cloud/comment-page-1/#comment-3869 QA Thought Leaders Wed, 25 Apr 2012 08:07:49 +0000 http://alexanderpodelko.com/blog/?p=243#comment-3869 Exceptional post on performance testing and optimizing cloud. I would like to know more if we can leverage the Cloud and Open Source Technologies in Performance Testing. Look forward to your next post. Exceptional post on performance testing and optimizing cloud. I would like to know more if we can leverage the Cloud and Open Source Technologies in Performance Testing. Look forward to your next post.

]]> Comment on Performance Testing and Optimization for the Cloud by Duval http://alexanderpodelko.com/blog/2012/03/29/performance-testing-and-optimization-for-the-cloud/comment-page-1/#comment-3866 Duval Wed, 18 Apr 2012 08:44:15 +0000 http://alexanderpodelko.com/blog/?p=243#comment-3866 Hi Alexander, I agree with your point of view: Performing load testing from the Cloud is not a rigorous scientific method. But a lot of our customers ask for it in order to determine if SLA are met under very high load. We take the margin so that even if the "real" number was worse, it would be marginal. Often I recommend to use an hybrid approach when it's possible: start to test from the lab (nearest point from the servers) and finish to test from the cloud (furthest point from the servers). This method allow to optimize the architecture first and to validate the SLA under very high load at the end. All the application’s delivery chain is tested. I obtain good successes with the hybrid approach with accurate recommendations for my customers. In the last version of NeoLoad, we simplify the monitoring configuration (even remotely), a way to improve again the recommendations to our customers because it's where the important value is for them. Best regards, Bruno Duval http://www.neotys.com/product/neoload-cloud-testing.html Hi Alexander,

I agree with your point of view: Performing load testing from the Cloud is not a rigorous scientific method. But a lot of our customers ask for it in order to determine if SLA are met under very high load.
We take the margin so that even if the “real” number was worse, it would be marginal.

Often I recommend to use an hybrid approach when it’s possible: start to test from the lab (nearest point from the servers) and finish to test from the cloud (furthest point from the servers). This method allow to optimize the architecture first and to validate the SLA under very high load at the end. All the application’s delivery chain is tested.
I obtain good successes with the hybrid approach with accurate recommendations for my customers. In the last version of NeoLoad, we simplify the monitoring configuration (even remotely), a way to improve again the recommendations to our customers because it’s where the important value is for them.

Best regards,

Bruno Duval
http://www.neotys.com/product/neoload-cloud-testing.html

]]> Comment on Performance Testing and Optimization for the Cloud by Community News: Tech tips from YouTube, Bitly and Heroku | New Relic blog http://alexanderpodelko.com/blog/2012/03/29/performance-testing-and-optimization-for-the-cloud/comment-page-1/#comment-3860 Community News: Tech tips from YouTube, Bitly and Heroku | New Relic blog Fri, 30 Mar 2012 19:06:41 +0000 http://alexanderpodelko.com/blog/?p=243#comment-3860 [...] attention to in real time.* Gartner predicts the future of the SaaS market.* Alexander Podelko discusses performance testing and optimization for the cloud.* Forbes magazine features an interview with GitHub co-founder and [...] [...] attention to in real time.* Gartner predicts the future of the SaaS market.* Alexander Podelko discusses performance testing and optimization for the cloud.* Forbes magazine features an interview with GitHub co-founder and [...]

]]> Comment on The Main Performance Problem by Tyson http://alexanderpodelko.com/blog/2012/03/22/the-main-performance-problem/comment-page-1/#comment-3855 Tyson Fri, 23 Mar 2012 16:28:47 +0000 http://alexanderpodelko.com/blog/?p=237#comment-3855 Most of the time, application performance has to do with end-user experience as Mr. Drogseth says. But not all the time! I was speaking with one of our ExtraHop customers who runs IT operations at a leading SaaS provider. He said he wasn't just concerned about speed or response times, but also about driving down the cost of IT in general. He said, "We have to be able to complete the same amount of work tomorrow with less resources than we used today." This is the case where IT is intrinsically part of the service or product provided. If they can identify and eliminate bad talkers on the network, inefficiencies in the database tier, and move more workloads to the data cache, then that translates directly to their profit margin or ability to undercut their competitors in terms of price. I love your captcha, btw! Most of the time, application performance has to do with end-user experience as Mr. Drogseth says. But not all the time! I was speaking with one of our ExtraHop customers who runs IT operations at a leading SaaS provider. He said he wasn’t just concerned about speed or response times, but also about driving down the cost of IT in general. He said, “We have to be able to complete the same amount of work tomorrow with less resources than we used today.” This is the case where IT is intrinsically part of the service or product provided. If they can identify and eliminate bad talkers on the network, inefficiencies in the database tier, and move more workloads to the data cache, then that translates directly to their profit margin or ability to undercut their competitors in terms of price.

I love your captcha, btw!

]]> Comment on The Main Performance Problem by William Louth http://alexanderpodelko.com/blog/2012/03/22/the-main-performance-problem/comment-page-1/#comment-3854 William Louth Fri, 23 Mar 2012 12:12:14 +0000 http://alexanderpodelko.com/blog/?p=237#comment-3854 What worries me more about this is that the focus seems to shifting away from monitoring (and managing to some extent) what is under our control to attempting to manage (cough monitor) what is not under our control (to an large extent) but has much more "meaning" and "visibility" internally to the ever growing segment in the performance management industry that have "don't have me think" printed on their forehead if not t-shirt. Performance is very much like supply chain management and yes getting these goods to the customer on time is important but this should drive engineering efforts along the supply chain and not be just at one end point because its easy to comprehend for some and makes everything look like a black (or green or red) box (which is gooooddddd for don't have me think kind of people). We need to take a holistic approach because efficiencies in one point in the chain might appear to allow greater flow when in fact it can have the complete opposite impact because of implicit feedback within the entire chain. Its the outliers I worry so much about and these tend to originate in some form or another from the backend and such feedbacks (and signals) kick in. Does that mean we need end to end tracing? NO at least not as prescribed by many of the BTM/APM vendors today. We need to first observe, model, understand and manage each point in the chain in terms of in and out flows and how the behavior is impacted by changes in these and how behavior changes internally impact these transgress points. What worries me more about this is that the focus seems to shifting away from monitoring (and managing to some extent) what is under our control to attempting to manage (cough monitor) what is not under our control (to an large extent) but has much more “meaning” and “visibility” internally to the ever growing segment in the performance management industry that have “don’t have me think” printed on their forehead if not t-shirt. Performance is very much like supply chain management and yes getting these goods to the customer on time is important but this should drive engineering efforts along the supply chain and not be just at one end point because its easy to comprehend for some and makes everything look like a black (or green or red) box (which is gooooddddd for don’t have me think kind of people).

We need to take a holistic approach because efficiencies in one point in the chain might appear to allow greater flow when in fact it can have the complete opposite impact because of implicit feedback within the entire chain. Its the outliers I worry so much about and these tend to originate in some form or another from the backend and such feedbacks (and signals) kick in.

Does that mean we need end to end tracing? NO at least not as prescribed by many of the BTM/APM vendors today. We need to first observe, model, understand and manage each point in the chain in terms of in and out flows and how the behavior is impacted by changes in these and how behavior changes internally impact these transgress points.

]]> Comment on The Main Performance Problem by Andy Hawkes http://alexanderpodelko.com/blog/2012/03/22/the-main-performance-problem/comment-page-1/#comment-3853 Andy Hawkes Fri, 23 Mar 2012 05:47:39 +0000 http://alexanderpodelko.com/blog/?p=237#comment-3853 Great points and I totally agree. I think the impact of this is hugely underestimated and generally ignored. Looking back over my own career, I figure my productivity at big companies averages maybe 15-25% what it is at small companies, and the shoddy performance (and poor fit) of so many "enterprise" applications are one of the main reasons. We've all been there. The aggregate cost in lost productivity must be huge. The inertia in those big IT silos is so bad though, I doubt anything short of a "chief" something-or-other would be able to crack the whip hard enough to bring about change. Great points and I totally agree. I think the impact of this is hugely underestimated and generally ignored. Looking back over my own career, I figure my productivity at big companies averages maybe 15-25% what it is at small companies, and the shoddy performance (and poor fit) of so many “enterprise” applications are one of the main reasons. We’ve all been there. The aggregate cost in lost productivity must be huge.

The inertia in those big IT silos is so bad though, I doubt anything short of a “chief” something-or-other would be able to crack the whip hard enough to bring about change.

]]>