Alex Podelko

And now, after my posts about agile performance testing and performance requirements in agile projects, we are getting to another fundamental issue behind performance testing issues in agile projects.

There are two very important assumptions for what I am saying below: stakeholders understand the need in performance engineering and the team knows the basics. How to get there is a very interesting topic, but it is outside of this post. For those, who are not there yet, the issues described below may sound abstract and they probably have more urgent problems to fight – but it is something you may want to be aware about while you are getting there.

The fundamental issue is, as I see it, that performance engineering teams don’t scale well, even assuming that they are competent and effective. At least not in their traditional form. They work well in traditional corporate environments where they check products for performance before release, but they face challenges as soon as we start to expand the scope of performance engineering (early involvement, more products/configurations/scenarios, etc.). And agile projects, when we need to test the product each iteration or build, expose the problem through the increased volume of work to do.

Just to avoid misunderstandings, I am a strong supporter of having performance teams and I believe that it is the best approach to building performance culture. Performance is a special area and performance specialists should have an opportunity to work together to grow professionally. The details of organization may vary (Scott Barber, for example, specified three models: “on demand”, “on retainer”, “full immersion”), but a center of performance expertise should exist. Only thing I am saying here is that while the approach works fine in traditional environment, it needs major changes in organization, tools, and skills when the scope of performance engineering should be extended (as in the case of agile projects).

Actually remedies are well known: automation, making performance everyone jobs (full immersion), etc. However they are not wide-spread yet.

Historically performance testing automation was almost non-existent (at least in traditional environments). Performance testing automation is much more difficult than, for example, functional testing automation (I use “automation” here as what should be done for “continuous testing”, i.e. process when you run test and get a report automatically for a new build without human intervention – not in its old way, when it meant using a tool; in performance we almost always use a tool). Setups are much more complicated. A list of possible issues is long. Results are complex (not just pass/fail). It is not easy to compare two result sets. So it is definitely much more difficult and would probably require much more human intervention, but it isn’t impossible.

However, the cost of performance testing automation is high. You need to know system well enough to make meaningful automation. Automation for a new system doesn’t make much sense – overheads are too high. So there was almost no automation in traditional environment [with testing in the end with a record/playback tool]. When you test the system once in a while before next major release, chances to re-use your artifacts are low.

It is opposite when the same system is tested again and again (as it should be in agile projects). It makes sense to invest in setting up automation. It rarely happened in traditional environments – even if you test each build, they are far apart and the difference between the builds prevents re-using the artifacts (especially with recorded scripts – API, for example, is usually more stable). So demand for automation was rather low and tool vendors didn’t pay much attention to it. Well, the situation is changing – we may see more automation-related features in load testing tools soon.

There are some vendor claiming that their load testing tools better fit agile processes, but it looks like in the best case it means that the tool is a little easier to handle (and, unfortunately, often just because there is not much functionality in it). Even if there is something that may be used for automation, like starting by a command line with parameters, it is difficult to find out.

At the moment, I read about few implementation of continuous performance testing – for example OpTier or Betfair (and few about measuring response times during functional testing in single-user mode – like this – which is a good step toward full-scale performance testing automation). Some prototypes, like this, are described– but without many details.

Probably we don’t see it more often because we don’t have much infrastructure for that kind of automation and performance testers may be not the best people to create complex integrated solutions from dozens of not-related pieces (as those who implemented it did). When we get more automation support in tools, we will probably see it changing.

By the way, I am not saying that automation would replace performance testing as we know it. Performance testing of new systems is agile and exploratory in nature and can’t be replaced by automation (well, at least not in the foreseen future). Automation would complement it – together with additional input from development offloading performance engineers from routine tasks not requiring sophisticated research and analysis.

It looks like agile methodologies are somewhat struggling with performance requirements (and non-functional requirements in general). Probably there are several reasons for that. One is that actually even traditional software development methodologies and processes never came with a good approach to handle performance requirements. They are, of course, considered in both literature and practical projects – but are usually handled rather in ad hoc manner. Actually the process of gathering and elaboration of performance requirements is rather agile in itself, and attempts to make it rigorous and formal look unnatural and have never fully succeeded – so it should be easier and more natural to do it as part of agile methods. Still the challenge of handling multidimensional and difficult to formalize performance requirements remains intact and the difference is rather in minor adjustments to agile processes than in the essence of performance requirements.

Another reason is that practical agile development is struggling with performance in general. Theoretically it should be a piece of cake: every iteration you have a working system and know exactly where you stand with the system’s performance. You shouldn’t wait until the end of the waterfall process to figure out where you are – on every iteration you can track your performance against requirements and see the progress (making adjustments on what is already implemented and what is not yet). Clearly it is supposed to make the whole performance engineering process much more straightforward.

Unfortunately, it looks like it doesn’t always work this way in practice. So such notions as “hardening iterations” and “technical debt” get introduced. Although it is probably the same old problem: functionality gets priority over performance (which is somewhat explainable: you need first some functionality before you can talk about its performance). So performance related activities slip toward the end of the project, and the chance is missed to implement a proper performance engineering process built around performance requirements.

Another issue here is that agile methods are oriented toward breaking projects into small tasks, which is quite difficult to do with performance (and many other non-functional requirements) – performance-related activities usually span the whole project.

There is no standard approach to specifying performance requirements in agile methods. Mostly it is suggested to present them as user stories or as constraints. And the difference is not so much in the way the requirements are presented, both ways rather use plain text.

User stories assume using a user voice form. Cohn, for example, suggests to use the “As a , I want , so that ” template for user stories (although he cautions that the user story template should only be used as a thinking tool, it should not be used as a fixed template). For constraints, both traditional expressions and user voice forms may be used.

The difference between user stories and constraints approaches is not in performance requirements per se, but how to address them during the development process. The point of the constraint approach is that user stories should represent finite manageable tasks, while performance-related activities can’t be handled as such because they usually span multiple components and iterations. Those who suggest to use user stories address that concern in another way – for example, separating cost of initial compliance and cost of ongoing compliance.

It looks like testingreflections.com, where I had blogged for a while, doesn’t exist anymore. One more confirmation that you can’t rely on somebody else long-term – who knows what would happen… Well, I still have some drafts of my old posts. Here is what I wrote about agile performance testing in 2009:

Two Views of Agile Unit Testing reminded me (indirectly) the QAForums Agile Performance Testing – Oxymoron? discussion about agile development and performance testing some time ago

Well, if we talk about agile development I don’t see any problem with performance testing at all: you have a working build each iteration – you test it. What is a problem? It should be synonym, not oxymoron.

All issues mentioned in the discussion boil down to one: nobody wants to pay for it (so no testers, no equipment, etc). Don’t see anything especially agile in it. Maybe companies still don’t switch to agile development for major projects – and don’t want to spend money for minor projects. Another explanation may be that in most traditional projects performance testing happens at the very end instead of every iteration (so require less resources) – but it not the way to do it properly.

I see significantly more problems doing performance testing properly during traditional development. I wrote a paper that has the same title, Agile Performance Testing, but completely different content: how to do performance testing in an agile way for any kind of projects (where agile projects look rather like a trivial case for me).

Well, returning to the original question of the discussion, there are a few articles in the Internet about the subject:

Agile Performance Testing by Jamie Dobson

An Explanation of Performance Testing on an Agile Team Part 1 and Part2 by Scott Barber

Chapter 6 of Performance Testing Guidance for Web Applications. Managing an Agile Performance Test Cycle.

Well, I still believe so. I still believe that performance testing in agile projects should be easier conceptually (although, of course, it would be a lot of technical difficulties – but who said that it should be easy?).

There are a few good newer articles addressing some aspects of the subject:

Top Ten Secret Weapons For Agile Performance Testing by Patrick Kua

Agile Performance Testing process AgileLoad whitepaper

Performance Testing in the Agile Enterprise by Scott Barber

Performance By Design – an Agile Approach by Jason Buksh

Still the subject is not covered much (and it looks like not actually practiced much – apart from involvement of traditional performance testing team in agile projects).

I started to prepare my talk for Belgium Testing Days (which promises to be a very interesting global testing conference) and the subject got me to very philosophical questions – will try to share my further thoughts about the subject in following posts.

Recent trends of agile development, DevOps, Web and Social Media sites somewhat question importance of load testing. Some (not many) openly saying that they don’t need load testing, some still paying lip service to it – but just never get to it. In more traditional corporate world we still see performance testing groups and important systems usually get load tested before deployment.

Let’s first define load testing as far as terminology is rather vague here. I use it here as anything that requires applying multi-user synthetic load – in contrast with single-user performance (which is a subset of performance engineering and may include, for example, profiling or Web Performance Optimization as it is defined now). And I use it here as an umbrella term including all other variations of multi-user testing, such as performance, concurrency, stress, endurance, longevity, scalability, etc. – but you may replace it with any other term if you prefer.

Yes, it looks like some Web and Social Media sites managed to survive without load testing. However, it looks like many such companies match the following profile:
-Business is built around a single Web site, so everybody in the company follows what is going on in production.
-Overall architecture is still clear and relatively simple. Changes (however frequent) are rather minor and evolutional.
-There is decent instrumentation providing performance information.
-There is a possibility to remove changes relatively easy.
-Site downtime/a period of slow performance (until the problem would be noticed and fixed) is not extremely painful or dangerous to the business.

Load testing is a way to mitigate load- and performance-related risks. There are other approaches and techniques that also alleviate some performance risks:
-Good single-user performance engineering practices (single-user requests performance are constantly tracked).
-Good instrumentation/Application Performance Management providing insights in what is going on inside the system.
-[Auto] scalable architecture.
-Continuous integration allowing quickly deploy and remove changes.

Still all of these don’t completely replace load testing, but rather complement it. They definitely decrease performance risk comparing with situation when nothing was done about performance at all until the last moment before rolling out the system in production without any instrumentation at all, but it still leaves risks of crashing and performance degradation under multi-user load. And if the cost of it is high, you should do load testing (what exactly and how is another large topic – there is much more here than the stereotypical waterfall-like last-moment record-and-replay approach).

There is always a risk of crashing or performance issues under heavy load – and the only way to mitigate it is actually test it. Even stellar performance in production and highly scalable architecture don’t guarantee that it won’t crash with a slightly higher load. Truly speaking, even load testing doesn’t completely guarantee it (real-life workload may be different from what you have tested), but it drastically decreases the risk.

Another important value of load testing is making sure that changes don’t degrade multi-user performance. Unfortunately, better single-user performance doesn’t guarantee better multi-user performance. In many cases it improves multi-user performance too, but definitely not always. And the more complex system, the more probable exotic multi-user performance issues no one even thought of. And a way to ensure that you don’t have such issues is load testing.

When you do performance optimization, you need a reproducible way to evaluate the impact of changes on multi-user performance. The impact on multi-user performance probably won’t be proportional to what you see with single-user performance (even if it still would be somewhat correlated). Without multi-user testing the actual effect is difficult to quantify. The same with the issues happening only in specific cases that are difficult to troubleshoot and verify in production – using load testing can significantly simplify the process.

Summarizing, I don’t see that the need in load testing is going away. Even in case of Web and Social Media sites we would probably see load testing coming back as soon as systems become more complex and performance issues start to hurt business. Maybe it would be less need for “performance testers” as it was at the heyday due to better instrumenting, APM tools, continuous integration, etc. – but I’d expect more need for performance experts that would be able to see the whole picture using all available tools and techniques (although I don’t see it yet).

Just finished reading The Lean Startup by Eric Ries. It is definitely a great book. And probably it says as much about software development as about entrepreneurship (most examples are from software development). It should be a must reading about agile software development.

However, reading The Lean Startup, the discussion about “best practices” popped into my head again and again. I have a more moderate position towards “best practices”. Maybe “best practice” and “requirement” are not the best terms if we consider them from the original meaning of English words, but the meaning of these notions deviated from the original word meaning and “best practice”, as far as I understand, means a good practice in an appropriate context – not “the best” “practice” (as “hot dog” is not “hot” “dog” anymore). And it looks like attempts to replace “best practice” and “requirement” with something else rather confuse people – at least I haven’t seen a good replacement yet. So I don’t think that words “best practices” are especially dangerous, it is just any good idea / practice / framework applied outside of the proper context may hurt.

No words “best practices” are in the book, all practices described are good in the specified context [of entrepreneurship], the author himself warns many times against blind application, but I clearly see how people would start to use some of the lean startup principles out of the appropriate context.

Let’s look at testing. Testing (and performance testing) is a way to mitigate risks. My interpretation of Eric Ries’ message is that don’t be abscessed by possible risks when you don’t have anything to lose yet. Nothing to say against it. However at some point you are getting something to loose, and you need to introduce some practices to mitigate the risks. How much – depends on your business. Perhaps the main example of “The Lean Startup”, IMVU, has higher tolerance to risk than most other business (single-site, not critical business, low cost of losing a user). I, of course, understand that you can’t cover everything in a single book, but I am still concerned that the topic of introducing of risk-mitigation was not covered. Actually testing was mentioned at least once: p.189 “we had an extensive set of automated tests that assured that after every change our product still worked as designed”. So even in IMVU, with their relatively high risk tolerance and early stage, they had “an extensive set of automated tests”. I hope that other readers of the book won’t miss that, but I am afraid that many will.

By the way, it is interesting what would be opposite to lean? Probably not fat (although it is what using the word “lean” hints) – you probably don’t want to have fat in whatever approach you use. Maybe “Build to Grow”? Lean means that you do absolute minimum (accumulating technical debt) – so the opposite would be if you build infrastructure / framework to grow from the beginning. So, I guess, the dimension we are talking here about is Build to Grow – Lean. Maybe it would be a next book explaining when you need to be very lean and when you need to start building up muscles?

With great interest I read through Agile Culture, Adoption, & Transformation Reading Guide by Michael Sahota. It makes a lot of sense for me. Along with many other interesting statements, Michael states that Kanban is not Agile.

Reading Michael’s posts encouraged me to share my old heretic thought. The Agile manifesto states we have come to value … Individuals and interactions over processes and tools. And here we have processes explained in excruciating details (like how to stand and how to chart), which usually are referred to as Agile. Do we have here a contradiction or I am missing something? I am not discussing their value here at all – I just saying that if I read The Agile manifesto correctly, it looks like agile process is an oxymoron, isn’t it? Sure, it still says there is value in the items on the right, but nevertheless.

I guess that we have multiple dimensions here and looks like they all get messed up. One dimension would be Waterfall – Iterative (or maybe Large Batch – Small Batch following “The Lean Startup” would be even more exact – waterfall may be considered iterative, but with very large iterations).

Another dimension may be Process Oriented – Agile.

It is interesting what would be opposite to lean? Probably not fat (although it is what using the word “lean” hints) – you probably don’t want to have fat in whatever approach you use. Maybe “Build to Grow”? Lean means that you do absolute minimum (accumulating technical debt) – so the opposite would be if you build infrastructure / framework to grow from the beginning. So Build to Grow – Lean would be probably another dimension.

So, I guess, every software development methodology may be described by its position on these (and/or other) dimensions and a set of specific practices (which may be useful in some context). However, except Michael Sahota’s analysis, I haven’t seen much in this direction yet.

I have a small stake here: in 2008 I wrote a paper titled Agile Performance Testing. I shared a few ideas on how to do performance testing in a more agile way there (using the word agile according to my reading of Agile Manifesto). By the way, I never meant that it is a new way of performance testing – rather a collection of several practices that, I believe, good performance engineers use (consciously or unconsciously). The paper didn’t touched at all performance testing in agile software development, so some attendees were disappointed (well, in its latest reincarnation I presented at Agile Testing Days I added slide 47 about it). Should admit that the title was chosen to provoke interest to the topic and it fulfilled this purpose well. Still I was periodically wondered if the name was correct – and still think that it does.

Application of agile principles to testing is another interesting topic.