Alex Podelko

Just finished reading The Lean Startup by Eric Ries. It is definitely a great book. And probably it says as much about software development as about entrepreneurship (most examples are from software development). It should be a must reading about agile software development.

However, reading The Lean Startup, the discussion about “best practices” popped into my head again and again. I have a more moderate position towards “best practices”. Maybe “best practice” and “requirement” are not the best terms if we consider them from the original meaning of English words, but the meaning of these notions deviated from the original word meaning and “best practice”, as far as I understand, means a good practice in an appropriate context – not “the best” “practice” (as “hot dog” is not “hot” “dog” anymore). And it looks like attempts to replace “best practice” and “requirement” with something else rather confuse people – at least I haven’t seen a good replacement yet. So I don’t think that words “best practices” are especially dangerous, it is just any good idea / practice / framework applied outside of the proper context may hurt.

No words “best practices” are in the book, all practices described are good in the specified context [of entrepreneurship], the author himself warns many times against blind application, but I clearly see how people would start to use some of the lean startup principles out of the appropriate context.

Let’s look at testing. Testing (and performance testing) is a way to mitigate risks. My interpretation of Eric Ries’ message is that don’t be abscessed by possible risks when you don’t have anything to lose yet. Nothing to say against it. However at some point you are getting something to loose, and you need to introduce some practices to mitigate the risks. How much – depends on your business. Perhaps the main example of “The Lean Startup”, IMVU, has higher tolerance to risk than most other business (single-site, not critical business, low cost of losing a user). I, of course, understand that you can’t cover everything in a single book, but I am still concerned that the topic of introducing of risk-mitigation was not covered. Actually testing was mentioned at least once: p.189 “we had an extensive set of automated tests that assured that after every change our product still worked as designed”. So even in IMVU, with their relatively high risk tolerance and early stage, they had “an extensive set of automated tests”. I hope that other readers of the book won’t miss that, but I am afraid that many will.

By the way, it is interesting what would be opposite to lean? Probably not fat (although it is what using the word “lean” hints) – you probably don’t want to have fat in whatever approach you use. Maybe “Build to Grow”? Lean means that you do absolute minimum (accumulating technical debt) – so the opposite would be if you build infrastructure / framework to grow from the beginning. So, I guess, the dimension we are talking here about is Build to Grow – Lean. Maybe it would be a next book explaining when you need to be very lean and when you need to start building up muscles?

And now I continue my post Can Processes Be Agile? into its application to testing (I broke it into two to avoid complete mess).

In a way I think that Exploratory Testing is an application of pure Agile principles to testing. And it looks like adherence to pure agile principles makes it rather difficult to explain – I guess that What Exploratory Testing is not series is an evidence how had is to apply pure agile principles without bringing it down to just another described in details process.

I have a small stake here: in 2008 I wrote a paper titled Agile Performance Testing. I shared a few ideas on how to do performance testing in a more agile way there (using the word agile according to my reading of Agile Manifesto). By the way, I never meant that it is a new way of performance testing – rather a collection of several practices that, I believe, good performance engineers use (consciously or unconsciously). The paper didn’t touched at all performance testing in agile software development, so some attendees were disappointed (well, in its latest reincarnation I presented at Agile Testing Days I added slide 47 about it). Should admit that the title was chosen to provoke interest to the topic and it fulfilled this purpose well. Still I was periodically wondered if the name was correct – and still think that it does.

Moreover, considering that Exploratory Testing is, in my current understanding, is application of agile principles to testing, I wonder if I should name the next reincarnation of this presentation “Exploratory Performance Testing”. Indeed the point I am trying to convey is that simultaneous learning, test design and test execution is even more important in performance testing. In functional testing you may have something (like use cases) describing how the system should work, but you don’t have anything that would describe how the system would behave from the performance and resource utilization point of view (performance requirements saying what are the limits stakeholders wants the system to achieve, but that’s the best you can hope to get).

At the CMG’12 conference we had an open discussion “New Approaches to Performance Testing: An Open Discussion on Plans, Experiments and Points In-Between”. The point I was trying to convey was that you need to experiment in performance testing to learn how the system behaves; just following a formal plan is less effective and often leads to missing performance problems altogether or to prolonged agony of performance troubleshooting. I, of course, mean a system of experiments based on the information we have about the system and results we are getting – it looks like some people believe that to experiment means to do random things. Well, it looks like I wasn’t too successful in this. Probably I should learn more how people advocate Exploratory Testing – but it looks like it is not easy to advocate something that doesn’t look a straightforward process.

With great interest I read through Agile Culture, Adoption, & Transformation Reading Guide by Michael Sahota. It makes a lot of sense for me. Along with many other interesting statements, Michael states that Kanban is not Agile.

Reading Michael’s posts encouraged me to share my old heretic thought. The Agile manifesto states we have come to value … Individuals and interactions over processes and tools. And here we have processes explained in excruciating details (like how to stand and how to chart), which usually are referred to as Agile. Do we have here a contradiction or I am missing something? I am not discussing their value here at all – I just saying that if I read The Agile manifesto correctly, it looks like agile process is an oxymoron, isn’t it? Sure, it still says there is value in the items on the right, but nevertheless.

I guess that we have multiple dimensions here and looks like they all get messed up. One dimension would be Waterfall – Iterative (or maybe Large Batch – Small Batch following “The Lean Startup” would be even more exact – waterfall may be considered iterative, but with very large iterations).

Another dimension may be Process Oriented – Agile.

It is interesting what would be opposite to lean? Probably not fat (although it is what using the word “lean” hints) – you probably don’t want to have fat in whatever approach you use. Maybe “Build to Grow”? Lean means that you do absolute minimum (accumulating technical debt) – so the opposite would be if you build infrastructure / framework to grow from the beginning. So Build to Grow – Lean would be probably another dimension.

So, I guess, every software development methodology may be described by its position on these (and/or other) dimensions and a set of specific practices (which may be useful in some context). However, except Michael Sahota’s analysis, I haven’t seen much in this direction yet.

I have a small stake here: in 2008 I wrote a paper titled Agile Performance Testing. I shared a few ideas on how to do performance testing in a more agile way there (using the word agile according to my reading of Agile Manifesto). By the way, I never meant that it is a new way of performance testing – rather a collection of several practices that, I believe, good performance engineers use (consciously or unconsciously). The paper didn’t touched at all performance testing in agile software development, so some attendees were disappointed (well, in its latest reincarnation I presented at Agile Testing Days I added slide 47 about it). Should admit that the title was chosen to provoke interest to the topic and it fulfilled this purpose well. Still I was periodically wondered if the name was correct – and still think that it does.

Application of agile principles to testing is another interesting topic.

Posted How Do We Measure Computer Resources? on Application Performance Engineering Hub. It looks like an important issue for the high-tech industry for me – it is a pity that it continues to be unnoticed.

See my reponse Is the Current Model of Load/Performance Testing Broken? on Application Performance Engineering Hub to Stephen Thair’s post.

See my How Response Times Impact Business? post in 2011 Performance Calendar.

When I created my site as a collection of performance-related links and documents in 2004, I grouped links somewhat arbitrary, just to avoid “analysis paralysis”, hoping to get back soon and polish as needed. It is interesting that I haven’t changed much in grouping for these seven years (definitely many things changed, many changes are long time due, but with main grouping of information I wasn’t able to improve much). Whatever links I added, they mainly fit one (or few) existing category. And just now I realized that we have a new information category – Application Performance Management – which doesn’t fit in any existing category. I had a category for APM tools from the beginning – they were around for a while – but not for generic APM information (something beyond talking about just tool features). And finally I put together a list of great information sources into a new group, Application Performance Management:

Application Performance Engineering Hub

Application Performance, Scalability, and Architecture blog from Dynatrace

The Performance Management section of The Virtualization Practice

APM Digest

Correlsense blog

App Signal blog from AppDynamics

Catchpoint’s Blog

Seriti Consulting Blog the Web Operations and Management Specialists, by Stephen Thair

Many of them existed for a while, but it looks like the quantity finally got into a new quality and we see a new discipline emerging (instead of a marketing term to promote a special kind of tools). It is definitely related that with new technologies, such as virtualization and cloud computing, traditional resource monitoring is not enough anymore and there is a need monitor on application and service levels. Some mentioned above blogs are from tool vendors, but they provide great content far beyond discussing the tools.

See my Performance Requirements – Do we need a better word? post on Application Performance Engineering Hub

Join your peers to discuss the dynamics of performance testing, planning and experimentation in today’s fast-paced, impatient world. At the Computer Measurement Group conference (CMG’11) in Gaylord National, Washington DC area, on December 7th, Wednesday, at 6:30pm in the Annapolis 1/2 room we will have an open Birds of a Feather session “New Approaches to Performance Testing: An Open Discussion on Plans, Experiments and Points In-Between”.

Everybody is welcome!

The idea to discuss the topic was suggested by James Pulley during a Linkedin discussion in the LoadRunner group. It turned out that there are quite different views of the role of experimenting in performance testing.

We hope to get a lot of CMG attendees as well as many performance testers from the area there. Please e-mail me to apodelko at yahoo dot com if you have any question or going to attend – just to have an idea how many people will attend.

Bernd Harzog’s post Why is Application Performance Management so Screwed Up? started a lot of discussions on the Internet. The post is a very good list of existing issues you may face when you try to use APM tools. I’d add one more – overheads. At least for the first generation, the claim that you may use APM in production worked only if you did very selective monitoring.

My view of APM is that first generation of APM tools so well described by Bernd was very immature. Not that something was explicitly wrong with the APM in general – really wrong was the drastic contrast between what the tools actually could do and marketing promises of tool vendors. The vendors talked more about the APM vision and how the APM tools are supposed to work – but not about the exact things these tools are able to do. Which you figured out in the best case after you spent a few days evaluating the product.

If check Garter Magic Quadrant for Application Performance Monitoring or my list of tools, it is clear that the market is very crowded and not well defined. There is no good criteria you can compare tools and different tools may actually do pretty different things, although it may be difficult to understand reading about them on vendor’s sites.

However I’d say that now we are getting the second generation of APM tools which are much closer to the APM promise for some technologies. I don’t want to list names here and separate “first” and “second” generations. I’d guess that some “first” generation tools might advance to the “second” generation if kept progress – but, as I said, it is difficult to say without actual evaluation of the tools. So I am hearing a lot of stories that people were able to successfully implement APM for system X using tool Y without many problems.

Still you doesn’t have a product which will do APM across all platforms and system if you have a full zoo of different technologies some of which are older than most of your IT employees (as many large corporations do). And don’t believe to anybody who tells you that they can do that. Still it looks like you can do it now for more systems with fewer problems – and start reaping the benefits of APM. Actually I don’t see any other alternative to APM in the long run – although it is a topic for a separate post. But be aware of all points mentioned in Bernd’s post – and check if the product you are going to use doing what you need in the way you want.

P.S. Just before posting noticed another Bernd Harzog post where he shares his view of next generation APM products.