Alex Podelko

And now, after my posts about agile performance testing and performance requirements in agile projects, we are getting to another fundamental issue behind performance testing issues in agile projects.

There are two very important assumptions for what I am saying below: stakeholders understand the need in performance engineering and the team knows the basics. How to get there is a very interesting topic, but it is outside of this post. For those, who are not there yet, the issues described below may sound abstract and they probably have more urgent problems to fight – but it is something you may want to be aware about while you are getting there.

The fundamental issue is, as I see it, that performance engineering teams don’t scale well, even assuming that they are competent and effective. At least not in their traditional form. They work well in traditional corporate environments where they check products for performance before release, but they face challenges as soon as we start to expand the scope of performance engineering (early involvement, more products/configurations/scenarios, etc.). And agile projects, when we need to test the product each iteration or build, expose the problem through the increased volume of work to do.

Just to avoid misunderstandings, I am a strong supporter of having performance teams and I believe that it is the best approach to building performance culture. Performance is a special area and performance specialists should have an opportunity to work together to grow professionally. The details of organization may vary (Scott Barber, for example, specified three models: “on demand”, “on retainer”, “full immersion”), but a center of performance expertise should exist. Only thing I am saying here is that while the approach works fine in traditional environment, it needs major changes in organization, tools, and skills when the scope of performance engineering should be extended (as in the case of agile projects).

Actually remedies are well known: automation, making performance everyone jobs (full immersion), etc. However they are not wide-spread yet.

Historically performance testing automation was almost non-existent (at least in traditional environments). Performance testing automation is much more difficult than, for example, functional testing automation (I use “automation” here as what should be done for “continuous testing”, i.e. process when you run test and get a report automatically for a new build without human intervention – not in its old way, when it meant using a tool; in performance we almost always use a tool). Setups are much more complicated. A list of possible issues is long. Results are complex (not just pass/fail). It is not easy to compare two result sets. So it is definitely much more difficult and would probably require much more human intervention, but it isn’t impossible.

However, the cost of performance testing automation is high. You need to know system well enough to make meaningful automation. Automation for a new system doesn’t make much sense – overheads are too high. So there was almost no automation in traditional environment [with testing in the end with a record/playback tool]. When you test the system once in a while before next major release, chances to re-use your artifacts are low.

It is opposite when the same system is tested again and again (as it should be in agile projects). It makes sense to invest in setting up automation. It rarely happened in traditional environments – even if you test each build, they are far apart and the difference between the builds prevents re-using the artifacts (especially with recorded scripts – API, for example, is usually more stable). So demand for automation was rather low and tool vendors didn’t pay much attention to it. Well, the situation is changing – we may see more automation-related features in load testing tools soon.

There are some vendor claiming that their load testing tools better fit agile processes, but it looks like in the best case it means that the tool is a little easier to handle (and, unfortunately, often just because there is not much functionality in it). Even if there is something that may be used for automation, like starting by a command line with parameters, it is difficult to find out.

At the moment, I read about few implementation of continuous performance testing – for example OpTier or Betfair (and few about measuring response times during functional testing in single-user mode – like this – which is a good step toward full-scale performance testing automation). Some prototypes, like this, are described– but without many details.

Probably we don’t see it more often because we don’t have much infrastructure for that kind of automation and performance testers may be not the best people to create complex integrated solutions from dozens of not-related pieces (as those who implemented it did). When we get more automation support in tools, we will probably see it changing.

By the way, I am not saying that automation would replace performance testing as we know it. Performance testing of new systems is agile and exploratory in nature and can’t be replaced by automation (well, at least not in the foreseen future). Automation would complement it – together with additional input from development offloading performance engineers from routine tasks not requiring sophisticated research and analysis.

It looks like agile methodologies are somewhat struggling with performance requirements (and non-functional requirements in general). Probably there are several reasons for that. One is that actually even traditional software development methodologies and processes never came with a good approach to handle performance requirements. They are, of course, considered in both literature and practical projects – but are usually handled rather in ad hoc manner. Actually the process of gathering and elaboration of performance requirements is rather agile in itself, and attempts to make it rigorous and formal look unnatural and have never fully succeeded – so it should be easier and more natural to do it as part of agile methods. Still the challenge of handling multidimensional and difficult to formalize performance requirements remains intact and the difference is rather in minor adjustments to agile processes than in the essence of performance requirements.

Another reason is that practical agile development is struggling with performance in general. Theoretically it should be a piece of cake: every iteration you have a working system and know exactly where you stand with the system’s performance. You shouldn’t wait until the end of the waterfall process to figure out where you are – on every iteration you can track your performance against requirements and see the progress (making adjustments on what is already implemented and what is not yet). Clearly it is supposed to make the whole performance engineering process much more straightforward.

Unfortunately, it looks like it doesn’t always work this way in practice. So such notions as “hardening iterations” and “technical debt” get introduced. Although it is probably the same old problem: functionality gets priority over performance (which is somewhat explainable: you need first some functionality before you can talk about its performance). So performance related activities slip toward the end of the project, and the chance is missed to implement a proper performance engineering process built around performance requirements.

Another issue here is that agile methods are oriented toward breaking projects into small tasks, which is quite difficult to do with performance (and many other non-functional requirements) – performance-related activities usually span the whole project.

There is no standard approach to specifying performance requirements in agile methods. Mostly it is suggested to present them as user stories or as constraints. And the difference is not so much in the way the requirements are presented, both ways rather use plain text.

User stories assume using a user voice form. Cohn, for example, suggests to use the “As a , I want , so that ” template for user stories (although he cautions that the user story template should only be used as a thinking tool, it should not be used as a fixed template). For constraints, both traditional expressions and user voice forms may be used.

The difference between user stories and constraints approaches is not in performance requirements per se, but how to address them during the development process. The point of the constraint approach is that user stories should represent finite manageable tasks, while performance-related activities can’t be handled as such because they usually span multiple components and iterations. Those who suggest to use user stories address that concern in another way – for example, separating cost of initial compliance and cost of ongoing compliance.

It looks like testingreflections.com, where I had blogged for a while, doesn’t exist anymore. One more confirmation that you can’t rely on somebody else long-term – who knows what would happen… Well, I still have some drafts of my old posts. Here is what I wrote about agile performance testing in 2009:

Two Views of Agile Unit Testing reminded me (indirectly) the QAForums Agile Performance Testing – Oxymoron? discussion about agile development and performance testing some time ago

Well, if we talk about agile development I don’t see any problem with performance testing at all: you have a working build each iteration – you test it. What is a problem? It should be synonym, not oxymoron.

All issues mentioned in the discussion boil down to one: nobody wants to pay for it (so no testers, no equipment, etc). Don’t see anything especially agile in it. Maybe companies still don’t switch to agile development for major projects – and don’t want to spend money for minor projects. Another explanation may be that in most traditional projects performance testing happens at the very end instead of every iteration (so require less resources) – but it not the way to do it properly.

I see significantly more problems doing performance testing properly during traditional development. I wrote a paper that has the same title, Agile Performance Testing, but completely different content: how to do performance testing in an agile way for any kind of projects (where agile projects look rather like a trivial case for me).

Well, returning to the original question of the discussion, there are a few articles in the Internet about the subject:

Agile Performance Testing by Jamie Dobson

An Explanation of Performance Testing on an Agile Team Part 1 and Part2 by Scott Barber

Chapter 6 of Performance Testing Guidance for Web Applications. Managing an Agile Performance Test Cycle.

Well, I still believe so. I still believe that performance testing in agile projects should be easier conceptually (although, of course, it would be a lot of technical difficulties – but who said that it should be easy?).

There are a few good newer articles addressing some aspects of the subject:

Top Ten Secret Weapons For Agile Performance Testing by Patrick Kua

Agile Performance Testing process AgileLoad whitepaper

Performance Testing in the Agile Enterprise by Scott Barber

Performance By Design – an Agile Approach by Jason Buksh

Still the subject is not covered much (and it looks like not actually practiced much – apart from involvement of traditional performance testing team in agile projects).

I started to prepare my talk for Belgium Testing Days (which promises to be a very interesting global testing conference) and the subject got me to very philosophical questions – will try to share my further thoughts about the subject in following posts.

And now I continue my post Can Processes Be Agile? into its application to testing (I broke it into two to avoid complete mess).

In a way I think that Exploratory Testing is an application of pure Agile principles to testing. And it looks like adherence to pure agile principles makes it rather difficult to explain – I guess that What Exploratory Testing is not series is an evidence how had is to apply pure agile principles without bringing it down to just another described in details process.

I have a small stake here: in 2008 I wrote a paper titled Agile Performance Testing. I shared a few ideas on how to do performance testing in a more agile way there (using the word agile according to my reading of Agile Manifesto). By the way, I never meant that it is a new way of performance testing – rather a collection of several practices that, I believe, good performance engineers use (consciously or unconsciously). The paper didn’t touched at all performance testing in agile software development, so some attendees were disappointed (well, in its latest reincarnation I presented at Agile Testing Days I added slide 47 about it). Should admit that the title was chosen to provoke interest to the topic and it fulfilled this purpose well. Still I was periodically wondered if the name was correct – and still think that it does.

Moreover, considering that Exploratory Testing is, in my current understanding, is application of agile principles to testing, I wonder if I should name the next reincarnation of this presentation “Exploratory Performance Testing”. Indeed the point I am trying to convey is that simultaneous learning, test design and test execution is even more important in performance testing. In functional testing you may have something (like use cases) describing how the system should work, but you don’t have anything that would describe how the system would behave from the performance and resource utilization point of view (performance requirements saying what are the limits stakeholders wants the system to achieve, but that’s the best you can hope to get).

At the CMG’12 conference we had an open discussion “New Approaches to Performance Testing: An Open Discussion on Plans, Experiments and Points In-Between”. The point I was trying to convey was that you need to experiment in performance testing to learn how the system behaves; just following a formal plan is less effective and often leads to missing performance problems altogether or to prolonged agony of performance troubleshooting. I, of course, mean a system of experiments based on the information we have about the system and results we are getting – it looks like some people believe that to experiment means to do random things. Well, it looks like I wasn’t too successful in this. Probably I should learn more how people advocate Exploratory Testing – but it looks like it is not easy to advocate something that doesn’t look a straightforward process.

With great interest I read through Agile Culture, Adoption, & Transformation Reading Guide by Michael Sahota. It makes a lot of sense for me. Along with many other interesting statements, Michael states that Kanban is not Agile.

Reading Michael’s posts encouraged me to share my old heretic thought. The Agile manifesto states we have come to value … Individuals and interactions over processes and tools. And here we have processes explained in excruciating details (like how to stand and how to chart), which usually are referred to as Agile. Do we have here a contradiction or I am missing something? I am not discussing their value here at all – I just saying that if I read The Agile manifesto correctly, it looks like agile process is an oxymoron, isn’t it? Sure, it still says there is value in the items on the right, but nevertheless.

I guess that we have multiple dimensions here and looks like they all get messed up. One dimension would be Waterfall – Iterative (or maybe Large Batch – Small Batch following “The Lean Startup” would be even more exact – waterfall may be considered iterative, but with very large iterations).

Another dimension may be Process Oriented – Agile.

It is interesting what would be opposite to lean? Probably not fat (although it is what using the word “lean” hints) – you probably don’t want to have fat in whatever approach you use. Maybe “Build to Grow”? Lean means that you do absolute minimum (accumulating technical debt) – so the opposite would be if you build infrastructure / framework to grow from the beginning. So Build to Grow – Lean would be probably another dimension.

So, I guess, every software development methodology may be described by its position on these (and/or other) dimensions and a set of specific practices (which may be useful in some context). However, except Michael Sahota’s analysis, I haven’t seen much in this direction yet.

I have a small stake here: in 2008 I wrote a paper titled Agile Performance Testing. I shared a few ideas on how to do performance testing in a more agile way there (using the word agile according to my reading of Agile Manifesto). By the way, I never meant that it is a new way of performance testing – rather a collection of several practices that, I believe, good performance engineers use (consciously or unconsciously). The paper didn’t touched at all performance testing in agile software development, so some attendees were disappointed (well, in its latest reincarnation I presented at Agile Testing Days I added slide 47 about it). Should admit that the title was chosen to provoke interest to the topic and it fulfilled this purpose well. Still I was periodically wondered if the name was correct – and still think that it does.

Application of agile principles to testing is another interesting topic.