Alex Podelko

It looks like agile methodologies are somewhat struggling with performance requirements (and non-functional requirements in general). Probably there are several reasons for that. One is that actually even traditional software development methodologies and processes never came with a good approach to handle performance requirements. They are, of course, considered in both literature and practical projects – but are usually handled rather in ad hoc manner. Actually the process of gathering and elaboration of performance requirements is rather agile in itself, and attempts to make it rigorous and formal look unnatural and have never fully succeeded – so it should be easier and more natural to do it as part of agile methods. Still the challenge of handling multidimensional and difficult to formalize performance requirements remains intact and the difference is rather in minor adjustments to agile processes than in the essence of performance requirements.

Another reason is that practical agile development is struggling with performance in general. Theoretically it should be a piece of cake: every iteration you have a working system and know exactly where you stand with the system’s performance. You shouldn’t wait until the end of the waterfall process to figure out where you are – on every iteration you can track your performance against requirements and see the progress (making adjustments on what is already implemented and what is not yet). Clearly it is supposed to make the whole performance engineering process much more straightforward.

Unfortunately, it looks like it doesn’t always work this way in practice. So such notions as “hardening iterations” and “technical debt” get introduced. Although it is probably the same old problem: functionality gets priority over performance (which is somewhat explainable: you need first some functionality before you can talk about its performance). So performance related activities slip toward the end of the project, and the chance is missed to implement a proper performance engineering process built around performance requirements.

Another issue here is that agile methods are oriented toward breaking projects into small tasks, which is quite difficult to do with performance (and many other non-functional requirements) – performance-related activities usually span the whole project.

There is no standard approach to specifying performance requirements in agile methods. Mostly it is suggested to present them as user stories or as constraints. And the difference is not so much in the way the requirements are presented, both ways rather use plain text.

User stories assume using a user voice form. Cohn, for example, suggests to use the “As a , I want , so that ” template for user stories (although he cautions that the user story template should only be used as a thinking tool, it should not be used as a fixed template). For constraints, both traditional expressions and user voice forms may be used.

The difference between user stories and constraints approaches is not in performance requirements per se, but how to address them during the development process. The point of the constraint approach is that user stories should represent finite manageable tasks, while performance-related activities can’t be handled as such because they usually span multiple components and iterations. Those who suggest to use user stories address that concern in another way – for example, separating cost of initial compliance and cost of ongoing compliance.

Updated the list of my publications. Suddenly a lot happened in December.

At the CMG Conference I presented the latest version of my performance requirements view: Performance Requirements: the Backbone of the Performance Engineering Process, paper and presentation.

Also at the CMG Conference I presented Load Testing: See a Bigger Picture, paper and presentation. A magazine version of it, A Bird’s-Eye View of Load Testing, was published in the December issue of Software Test and Quality Assurance Magazine.

My notes on Performance Testing in the Cloud: Look Beyond the Word were published in the December issue of Testing Experience.

My post Performance: See the Whole Picture was included in 2012 Performance Calendar.

My next talk, Agile Aspects of Performance Testing, would be at Belgium Testing Days in Brussels, Belgium on March 1, 2013. Belgium Testing Days is a great global testing conference in spite of its rather local name – just check the program. All areas of testing covered – even performance testing is well represented, that is rather unusual for testing conferences.

Returning to the discussion around my post How Response Times Impact Business? and recent publications about the topic, like For Impatient Web Users, an Eye Blink Is Just Too Long to Wait.

From one side, we see more and more statement the response times should be shorter and shorter. For example, both Scott Barber and Harry Shum (“a computer scientist and speed specialist at Microsoft” according to the New York Times article) state 250 milliseconds as the magic number for response times (although I am not sure where these 250 milliseconds came from).

From another side, the three psychological thresholds and other considerations I referred to in my post were based on multiple researches and definitely make sense.

Well, I definitely prefer to see a recent research about the subject. It is strange that there were a lot of research since 1968 – but none recent. And it is when really big money gets involved. Or there are some, but they just don’t get released?

Meanwhile one explanation may be that perception about [at least] simple web navigation is changing. Web response times were defined by the second threshold: users feel they are interacting freely with the information (1-5 seconds). They notice the delay, but feel that the computer is “working” on the command. Well, maybe users don’t feel anymore that the computer should “work” [at least] for simple web navigation. Maybe they now perceive it as described by the first threshold: instantaneous (0.1-0.2 second). Users feel that they directly manipulate objects in the user interface. So while these psychological thresholds are still correct, perception of [at least] simple web navigation is changing and it gets defined by another threshold. Just a speculation, of course – it would be interesting to see any research to prove or disprove it.

In a similar classification, Steven Seow in Designing and Engineering Time: The Psychology of Time Perception in Software defines four classes of responsiveness bases on user expectancy:

• Instantaneous: 0.1 to 0.2 seconds
• Immediate: 0.5 to one second
• Continuous: two to five seconds
• Captive: seven to ten seconds

So in a way he breaks the middle threshold into two classes: immediate and continues. If accept this division, we perhaps may say that user expectations for [at least] for simple web navigation are moving from continuous class to immediate class. That maybe makes more sense for me: I am still rather skeptical that we indeed need 250 ms end-user response time (of course, if we talk about server response time, it would be another story).

Posted How Do We Measure Computer Resources? on Application Performance Engineering Hub. It looks like an important issue for the high-tech industry for me – it is a pity that it continues to be unnoticed.

See my How Response Times Impact Business? post in 2011 Performance Calendar.

See my Performance Requirements – Do we need a better word? post on Application Performance Engineering Hub

My performance requirements slides from yesterday’s CT CMG meeting (significantly updated) are on the CT CMG site.