I needed to check if setting processor affinity would be able to minimize mutual performance impact of server programs running inside a Linux VM. Setting processor affinity in Linux with taskset looks pretty straightforward:

taskset -c -p <cpu#> <pid>

However, when I ran a test, I noticed that the program still use all CPUs. So I started investigation (truly speaking, suspecting that it may be a virtualization effect) – but I was pointed out that to set processor affinity for a multithreaded application (such as the one I worked with) I need to set affinity for all tasks in /proc/<pid>/task with something like:

for p in `ls /proc/<pid>/task`; do taskset -p -c <cpu#> $p; done

as far as “threads are essentially just processes with a shared address space on Linux”.

As soon as I ran the code above for the pid, the process was limited to one processor and affinity worked fine.

So the problem was that I set affinity only for the parent process, but not for the tasks in /proc/<pid>/task (child processes inherit affinity when they are started – but in this case they all were started before I set affinity for the main process).

The process may be started with affinity from the beginning. The following, for example, works fine too:
taskset -c <cpu#> <start_command>

But if you need to set affinity for already running processes, don’t forget to set it for all tasks in /proc/<pid>/task too.

A lot of discussions are going on around MOOCs (Massive Open Online Courses) and how they would revolutionize college education. Of course, it would be interesting to see what would be an outcome – although I don’t expect to see drastic changes in traditional education soon, probably it will trigger some evolutional changes (and MOOCs don’t sound as something completely new for me – rather a next step in slow evolution of online / remote education).

I found rather interesting their application in another market – in vendor training. It was, in my understanding, a very stable market with expensive classes without much checking of results (as you pay for the class, you get a certificate). Sometimes complemented by a certification program – where for some money your knowledge is checked and confirmed by a nice title and certificate. All variations I saw so far included doing the same classes online or selling training materials (for the price comparable to the cost of actual class).

I was surprised to find that 10gen provides free online classes for MongoDB – a leading NoSQL database. The classes look well-designed and include meaningful knowledge check (only those who get up to some level get a certificate). I was also surprised to learn that they have thousands of people enrolled for a specific class – considering that MongoDB is rather a specialized product. Probably introducing these free online classes contributed to popularity of MongoDB, which looks like the most popular NoSQL database for the moment.

While I don’t know if other vendors follow the example, it looks revolutionary for me. Not from the technological point of view – it looks in line with other MOOC technology allowing a huge number of students with relatively moderate efforts to manage it. But from the business point of view, when a vendor voluntarily gives up training revenue for spreading a word about their technology and building up a community of loyal users. It was even more surprising because MongoDB is open source and 10gen doesn’t have a stream of revenue from software licenses (at least from the base product) – so giving up one more stream of potential revenue doesn’t sounds self-evident. But indeed the benefits they get with such approach are probably much more significant that the stream of revenue they could get (and maybe they are still getting it doing more traditional training for those who wants it – and their share may be larger due to a larger number of people familiar with the technology).

It is definitely a very important precedent which potentially may overturn the whole vendor training market. It would be interesting to see if other follow the pattern – and, I guess, those who follow may see a noticeable competitive advantage. Of course, MongoDB is an open-source product and it may be not so straightforward for proprietary commercial products. Although many vendors have some kind of limited licenses (evaluation, developer, limited edition, etc.) anyway that potentially allows to have that kind of online courses.

Another side effect may be the destruction of third-party basic training for specific products. If free training is available, you can’t just live on teaching basics to people – you need add value to have people paying (whatever this value is).

I see some hints that it may become a trend. It looks like free online tutorials become a standard. BlazeMeter putting together JMeter educational videos. It looks like the Practical Performance Analyst site gave up on the idea to charge for online performance engineering classes – so they are available now for free. Evan large companies share a lot of free educational video. See, for example, Oracle Learning Library. Of course, free videos is not a full class with checking knowledge and providing a certificate – so it may take some time until we see it as mainstream (although I may miss other important developments as far as the subject itself is not my primary interest).

Recently I continue to stumble upon ISTQB (International Software Testing Qualification Board) certification.

First, after a few discussion about need of standardization (and potentially certifications) for performance engineering, I decided to check available [functional] testing certifications and started to ask people what they think about existing certification and which is the most popular. It is interesting that everybody referred to ISTQB – no others were mentioned seriously (while there are some around). While feedback differs depending on people’s position to certification in general, nobody mentioned anything particularly negative (beyond discussion about need in certifications in general). Not a scientific research, of course – but often such informal discussions help to understand what is going on.

I looked through ISTQB syllabi and got impression of pretty decent attempt to formalize a not very structured area. I understand how difficult it should be to formalize it on the concept level – it is much more difficult than to make a syllabus / exam for a specific product or a well established discipline. And, of course, it is changing before our eyes with trends of agility (including exploratory testing) and continuous integration. Of course, it is not ideal (I won’t even comment on the performance testing part) – but my impression is that it is a very good step in the right direction.

My opinion is that reasonable good certifications are beneficial for both people and the industry. You can read why on any certification provider site. Of course, you need to take it with a grain of salt – actually a very large grain – but overall they probably bring more benefits than issues. Issues, of course, exist too.

If we speak about people, it probably brings more benefits in the beginning of the career. When you get to the expert level, certifications won’t probably add much to your credentials. So here I’d agree with Anne-Marie Charrett’s post, but it is probably not even certification-specific. If you see that your potential employer practices something that contradicts your believes – you’d better look for another one (if you have an option). However I don’t agree that certification is needed only for screening – for example, certified people would probably know and use common terminology and may get some understanding of the areas they never touch otherwise. So certification may improve the bottom line, but it doesn’t add much to very experienced people. It even put them in somewhat awkward situation – they need either to go and prove that they know the basic stuff or prove periodically that they are so good that they don’t need any certification. And it is indeed an issue – but, unfortunately, such people don’t represent the majority – so it may be unfair to evaluate the benefits of certifications on the example of experts.

And I don’t agree with this Scott Barber’s post: it would be great in the ideal world, but in the real world we have certifications everywhere and I don’t recall any certification body in any area that gets financially accountable for certified people. The word certification in the real world has a pretty standard meaning and has no such high-level connotations as listed by Scott.

Then I stumbled on Pradeep Soundararajan’s post about ISTQB. It struck me as illogical and unprofessional. While I am definitely a side observer and perhaps missing something important – but if you accuse somebody in scam (which in my understanding is a crime), you’d better put better reasons behind it.

“these training for the cheap certification that is really expensive”. And what? Training is usually expensive (there are some interesting trends now, but they didn’t get to testing yet). And as soon as it is not required for the certification, I am fine with it. You don’t need to take training for ISTQB, do you? As far as I understand, there are books, which are not very expensive – buy a book and get prepared. Am I missed something? If we speak about scam, I am more concerned when the training is required to get certification – and I know a lot of vendor certifications that do that (and at least one testing certification too). Why Pradeep doesn’t complain about those?

“I have a pre final year college graduate who wanted to try her luck, try ISTQB, cleared it and does not have a clue on how to test” –I met many people who got their degrees, and didn’t have a clue on how to do real work. There is always some danger in every training/certification that somebody would just memorize the facts. I’d rather consider foundation certification as evidence that the person knows terminology / main notions. Anyway it is fine to say that the certification doesn’t verify that or this enough – but saying that it is a scam and shame? In my understanding you need to provide something more to prove the point.

“ISTQB as an organization was built to show they are not for profit but those who say it is not for profit” Well, not for profit is, as far as I understand, is a well defined legal term and there are government organization (such as IRS in the US) that keep an eye that such organizations do what they are supposed to do. And this status doesn’t mean that the organization shouldn’t earn money.

“use their own company names to make the profit out of ISTQB” And what is bad here? I guess that they get a small marketing advantage saying that they were involved in its development – but as soon as they don’t distribute, for example, exam questions, I don’t see crime here. People use their credentials to earn money, what’s wrong with this?

“The bigger shame is – when I talked about this – the ISTQB fellas said, “We don’t ask them to do it but it is not our business to get them to change it”.” And what you want them to say? If a company believes that you need a certification (for whatever reason they have) – you either get it, or go to another employer. It looks pretty straightforward for me. While I don’t think that it was a good thing for them to do, I don’t see any reason why they can’t. I am not a big fun, for example, pre-employment drug testing or sexual harassment trainings, but many companies require them – and you just do them if you want employment.

“I have been employed in services companies that have asked me to take ISTQB and my appraisal would depend on that.” Hmm, and what is the problem with that? I’d really love if I was asked to get certified. I rather consider as a problem when employers don’t care at all about training / people development.

“Also, the marketing and advertising they put up in magazines, conferences and other places (I haven’t checked any bill boards) is misleading. It ranges from ROI on this certification to how better testers can be after this. ” Actually in the couple of your examples you yourself proved that the certification does provide ROI to the certified – the employers do care about it and want to get certified people! And as for better testers – well, I’d assume that the certified people would know and use common terminology and may get some understanding of the areas they never touch otherwise. That’s it – and if it gets relatively popular, it works. Certifications never were a way to revolutionize the process – rather an evidence of maturity. In that sense, if there are significant breaks through in the area, certification may need to be updated – but that is a separate story.

Then there is a petition, it is definitely more to the point and is very intriguing. Wow, there is “an exam reliability coefficient reviewed by your exam consultants Kryterion” – it is indeed pretty interesting how it is calculated and what it shows. Kryterion turned out to be the ISTQB exam provider – so I don’t think you can name it third-party, it rather looks like internal feedback on exams to ISTQB (which, I hope, ISTQB will use to improve them) – and the petition is asking to reveal this information. Well, I’d guess the petitioners have a right to ask – and it is up to ISTQB to decide what to do with this information. I don’t think that any organization ever revealed such information before (at least my Google search didn’t return anything), especially if there are some concerns – but it is, of course, up to ISTQB to decide. By the way, I won’t be surprised if some coefficients would be worse for the ISTQB exams comparing with specific product exams – the area is definitely more difficult to formalize and evaluate.

But I am somewhat confused what the goal of the petition is. Concerns that the exam doesn’t test the syllabus properly? Apply pressure to ISTQB to make the exam more rigorous? Real concerns about improving the quality of the ISTQB certification? Doesn’t sound so at least when you read Pradeep’s post.

Well, testing as a discipline is not in the best shape right now. Perhaps we may say that it is in a crisis. Status of testers may definitely be improved. There are several groups of people working to improve status of testing and testers – but their efforts are far from full success. Certifications, with all their shortcoming, improve the maturity of the area. They don’t help top-level people much (and probably rather annoy them), but they move up the bottom line. So, in my opinion, it would be much more benefits for testing in general if all testing thought leaders would at least be professional and constructive in their discussions – for example, suggest ways to improve certifications if they believe they are not good enough. While we probably won’t need certifications in the ideal world when everybody would be knowledgeable and passionate about their work – but, unfortunately, the real world is far from that. So we probably need to live with certifications for a while – and the question is if we have a relatively decent one (as much as it could be) or they would be indeed close to a scam (as soon as developing decent one requires a lot of efforts).

In a strange association Scott Barber’s video Why Executives See Software Testing as #EpicFail comes to mind. It is a very sobering view and perhaps all testers should watch it from time to time. There are quite different opinions almost about every subject – but, in my opinion, it would be better to see the whole picture and at least keep the discussions constructive and professional.

Performance testing is an interesting discipline: while you may see a lot of people doing it, not much is written about it. And what is written is often either introduction-level or tool-specific (other large groups of topics including performance analysis and performance troubleshooting are wider topics and not testing-specific). But not much about performance testing specific challenges beyond of what is covered by tool manuals.

My group specializes in performance testing of Hyperion / Enterprise Performance Management and Business Intelligence products at Oracle. A few of scripting challenges exist for almost every product. Nothing exceptional – you should resolve them if you have some experience and would be attentive enough, but time after time we are called to save performance testing projects to find out that there are serious problems with scripts and scenarios which make test results meaningless. Even very experienced testers stumble, but problems could be avoided if more time was spent analyzing what is going on.

Let’s consider an example – probably typical for challenges you can face with modern Web-based applications. Some operations, like financial consolidation, can take a long time. The client starts the operation on the server and then waits until it will finish, a progress bar is shown in meanwhile. When recorded, the script looks like (in LoadRunner pseudo-code):

web_custom_request(“XMLDataGrid.asp_7″,
“URL={URL}/Data/XMLDataGrid.asp?Action=EXECUTE&TaskID=1024&RowStart=1&ColStart=2&RowEnd=1&ColEnd=2&SelType=0&Format=JavaScript”,
LAST);

web_custom_request(“XMLDataGrid.asp_8″,
“URL={URL}/Data/XMLDataGrid.asp?Action=GETCONSOLSTATUS”,
LAST);

web_custom_request(“XMLDataGrid.asp_9″,
“URL={URL}/Data/XMLDataGrid.asp?Action=GETCONSOLSTATUS”,
LAST);

web_custom_request(“XMLDataGrid.asp_10″,
“URL={URL}/Data/XMLDataGrid.asp?Action=GETCONSOLSTATUS”,
LAST);

What each request is doing is defined by the ?Action= part. The number of GETCONSOLSTATUS requests recorded depends on the processing time. In the example above it was recorded three times; it means that the consolidation was done by the moment the third GETCONSOLSTATUS request was sent to the server. If playback this script, it will work in the following way: the script submits the consolidation in the EXECUTE request and then calls GETCONSOLSTATUS three times. If we have a timer around these requests, the response time will be almost instantaneous. While in reality the consolidation may take many minutes or even an hour (yes, this is a good example when sometimes people may be happy having one hour response time in a Web application). If we have several iterations in the script, we will submit several consolidations, which continue to work in background competing for the same data, while we report sub-second response times.

Consolidation scripts require creating an explicit loop around GETCONSOLSTATUS to catch the end of consolidation:

web_custom_request(“XMLDataGrid.asp_7″,
“URL={URL}/Data/XMLDataGrid.asp?Action=EXECUTE&TaskID=1024&RowStart=1&ColStart=2&RowEnd=1&ColEnd=2&SelType=0&Format=JavaScript”,
LAST);

do {

sleep(3000);

web_reg_find(“Text=1″,”SaveCount=abc_count”,LAST);

web_custom_request(“XMLDataGrid.asp_8″,
“URL={URL}/Data/XMLDataGrid.asp?Action=GETCONSOLSTATUS”,
LAST);

} while (strcmp(lr_eval_string(“{abc_count}”),”1″)==0);

Here the loop simulates the internal logic of the system sending GETCONSOLSTATUS requests each 3 sec until the consolidation is done. Without such loop the script just send requests and ends the iteration while the consolidation runs for a long time after that.

Some other examples can be found in my presentation A Load Testing Story at CMG’10 – a story about one my project which had a good share of testing challenges.

Many technical details about testing Hyperion products with LoadRunner are included in Oracle support document 1339743.1. A lot of this knowledge is now included in the Hyperion module of Oracle Application Testing Suite. But that is just about one family of products – which is a small sample of challenges you would see around.

According to my experience, creating a meaningful and realistic workload is the main performance testing challenge – and you probably face it in most non-trivial systems (including most of corporate applications). It is surprising that very few people talk about it: reading about performance testing you may get impression that the scripting is trivial and mundane (and it looks like many “performance testers” truly believe in it – and when they face any challenge, they start to talk about the death of performance testing).

Only few names come to my mind who write about specific challeneges. For example, see Northway (@northwaysg) / Scott Moore (@loadtester) performance testing blog and white papers or Mentora (@MentoraGroup ) / Dan Downing presentations and white papers. Maybe few more – but definitely they are few and far apart.

Performance testing terminology is not well defined and one of the most ambiguous terms is user concurrency. Re-reading Load Testing: Concurrent Users verses Simultaneous Users by Scott Moore (@loadtester) and LoadRunner Concurrency video by Mark Tomlinson (@mtomlins) inspired me to post this comment. Here is what I wrote (and still believe in it) in my old CMG paper about performance requirements in 2007 ( the latest version of this paper was presented again at CMG’12):

Concurrency is the number of simultaneous users or threads. It is important too: connected, but inactive users still hold some resources. For example, the requirement may be to support up to 300 active users.

The number of online users (the number of parallel session) looks like the best metric for concurrency (complementing throughput and response time requirements).

To summarize my comment, I believe that the number of “really concurrent” users is not an appropriate input metric for performance engineering and performance testing. It perhaps may be an output metric characterizing system’s load if we find a way to measure it (in a way, it is the number of users in the system if we use queuing theory terminology).

After attending Sergey Chernyshev’s (@sergeyche) Scalability vs. Performance presentation at NY Web Performance Meetup and reading Scalability: it’s the question that drives us by Robert David Graham (@ErrataRob) and Scalability vs. Performance: it isn’t a battle by Theo Schlossnagle (@postwait) I would like to share my understanding. While I agree in general with everything said, I would rather word it differently. The topic became loaded, so accents are important. Robert, for example, states that “performance” and “scalability” are orthogonal problems. Well, no, they are not. They are different, but correlated notions. Even leaving aside that performance and scalability are somewhat vague terms.

If we speak about web systems now, it looks like we can roughly separate two main components in response time (which is the main performance metric): backend (server-side) time and frontend (network and client-side time). There are subtleties and grey areas, but I’d ignore them here. The frontend time, the subject of Web Performance Optimization (WPO), doesn’t relate to scalability as far as it is not involving server processing (again ignoring subtleties).

The proportion of frontend time vs. backend time may be any at all. According to Steve Souders (@souders), the founder of WPO, 80-90% of the end-user response time is spent on the frontend. But even for major web sites the backend time for requests involving database processing (such as submitting orders or querying order status) may be more noticeable. And there are plenty of corporate web applications where the share of frontend time is rather small. Of course, the starting point of any performance troubleshooting is to find where time is spent. And there is absolutely no sense to optimize parts where time is not spent.

However, there is one important “but”. While front-end time supposed to be constant (another simplification, but again ignoring subtleties), the backend (server) time depends on load. The heavier is load, the larger may be server time. And at some point it may skyrocket making the system practically unusable. So thinking about what you need to optimize, you need to check where time is spend under maximal load. Unless you don’t care about downtime and user experience, the way to do it is load testing.

And here we get to scalability. The frontend performance indeed doesn’t matter here and is independent of scalability. But the backend performance is directly related to scalability. The relationship, of course, may be non-linear and quite sophisticated – but it does exist.

To illustrate it, let’s consider one simple (but still typical) example. The backend processing takes X ms, the time is mainly spent in CPU and we don’t have any other bottlenecks. In this case the server response time would be mainly CPU processing time – and every request would take X ms of CPU time (if we don’t have parallelism here). As soon as we take most of available CPU time, server response time would skyrocket (that situation may be modeled using queuing theory). So there is a load when the system becomes practically unusable – and the question is just when we get to this load. We, of course, may get to problems sooner if we have any scalability problem inside the system or run out of another kind of resources.

Generally speaking, you can increase scalability by either optimizing server processing (using less resources) or providing more resources. Of course, if your architecture allows using these additional resources – so mainly scalability boils down to ability to parallelize your processing (and often limited by what you can’t fully parallelize – like a centralized database).

We do have two parts of response time – frontend and backend – which behaves differently and may need different approaches and tools to optimize. But the end user experience is the sum of these two parts – where the backend time is a function of load. You can’t say much about your end-to-end performance and its backend part until you check it under load – and load testing is the safe way to do so.

Historically performance engineering concentrated on backend – where main performance and scalability issues were – and practically ignored frontend (which indeed was usually pretty straightforward then). Several sub-disciplines were formed including performance analysis, capacity planning, and load testing. Later, when sophistication of frontend skyrocketed, a whole new discipline was established by Steve Souders and quickly grew around Velocity and Web Performance meetups. Unfortunately, it practically dismissed performance engineering developments of the last 40 years (maybe even more – the Computer Measurement Group (CMG) was founded in 1975). While frontend WPO definitely has its own specific, I’d still expect to see a holistic approach to performance engineering, taking in account all aspects of performance and scalability end-to-end.

Just read Gartner Q&A: Analytics vs. APM with Will Cappelli, Gartner Research VP in Enterprise Management, part one, part two, and part three about his latest report: Will IT Operations Analytics Platforms Replace APM Suites?

The discussion is very interesting and informative, touches areas that interested me for a long time, but the title doesn’t make sense for me. Maybe I just got behind in terminology – but here is as I understand it.

I joined Hyperion in 1999 and, with one break, did, in a way, performance management of performance management applications. I was responsible for application performance of business performance management and business intelligence applications. At this point you probably can guess some questions that came to my mind. Can we use anything from application performance management in business performance management? Or, vise versa, can we use business analytic software for application performance analysis? Well, I haven’t still figured out a meaningful way to do it.

I am also involved in CMG (worldwide organization of IT professionals specializing in performance, capacity and service management), where there was an idea that skills of application performance and capacity management may be used for business performance and capacity. I waited with interest if the best experts in application performance and capacity management (and CMG is the place where you find them) would be able to come with interesting ideas of using their skills for business (beyond the trivial fact that performance impacts business). Still waiting.

While these two areas, application performance and business performance, look very similar and, of course, often use the same underlying approaches and math, the devil is in details. They are just two different areas and not much may be re-used between them (as soon as we get to details).

Reading the discussion, I came to one explanation of what is going on. It also explains one more phenomenon surprising me for a while. The phenomenon was that there are many new companies on the market providing only end-user monitoring (EUM, or real-user monitoring, RUM) and apparently doing pretty well. While EUM is definitely a very important APM area, my understanding was that it is just one part of APM and, without other parts, provides very limited APM value. My understanding was that we should rather move toward integrated APM solutions. Still the impression is that some such EUM companies doing even better than companies providing integrated APM solutions.

Well, the explanation is that web analytics (even with EUM) is just a completely different area from APM (although EUM is an important part of APM too). Web analytics is a business application (and yes, end-user performance is one of components of business analysis now) and APM is an “IT” application. If we agree on this, it explains all these strange titles about death of APM or replacing APM by analytics. End-user performance with web analytics became a business intelligence application – and the business intelligence market is many times larger than IT intelligence market (where we still have all true APM solutions – with all their problems because the APM is, in essence, much more sophisticated area technically than business analytics). It was always this way – recall market size of business intelligence (Hyperion, Cognos, Business Objects, etc.) vs. market size of companies providing monitoring and deep diagnostics solutions.

Business intelligence/analytics moved to a new level: from sales analysis we are getting to user clicks analysis. And there we have another level of data volume (==big data) and we are getting to the point when end-user performance is just one more dimension of business data. Performance impacts business – and business wants to analyze it. Well, in application performance management we worked with “big data” for ages – and have you heard about it as a world problem until it got into the business realm?

But nothing changed in APM – if we find a way to separate these business-related applications, I guess we find that the APM market is slowly growing and that APM applications are slowly becoming better and better. To manage your applications, you need an APM solution, not web analytics and not EUM only (although EUM is important for APM too).

As I highlighted in my performance requirements article (here is the last version just presented at CMG’12), we have two completely different views: business view and “IT” view. Business is interested in one (and only one) performance metric – end-user response time. That’s it, nothing else at all. All other stuff – throughput, resource utilization, bandwidth, latency, etc. – is for IT internal use only (even if “IT” is the core of the company as with many web services). Yes, business people know these words too – they hear them all the time as explanation why we can’t get requested performance or why we need to spend another pile of money on something IT needs – but if “IT” delivered the requested performance they were quite happy not even knowing these words. And that is exactly what we see here: we have solutions for business (EUM – which is much simpler of integrated APM solutions and provides everything business needs) and solutions for “IT” (with all these stuff you needed to manage applications performance – but much more sophisticated and difficult to implement).

And now, after my posts about agile performance testing and performance requirements in agile projects, we are getting to another fundamental issue behind performance testing issues in agile projects.

There are two very important assumptions for what I am saying below: stakeholders understand the need in performance engineering and the team knows the basics. How to get there is a very interesting topic, but it is outside of this post. For those, who are not there yet, the issues described below may sound abstract and they probably have more urgent problems to fight – but it is something you may want to be aware about while you are getting there.

The fundamental issue is, as I see it, that performance engineering teams don’t scale well, even assuming that they are competent and effective. At least not in their traditional form. They work well in traditional corporate environments where they check products for performance before release, but they face challenges as soon as we start to expand the scope of performance engineering (early involvement, more products/configurations/scenarios, etc.). And agile projects, when we need to test the product each iteration or build, expose the problem through the increased volume of work to do.

Just to avoid misunderstandings, I am a strong supporter of having performance teams and I believe that it is the best approach to building performance culture. Performance is a special area and performance specialists should have an opportunity to work together to grow professionally. The details of organization may vary (Scott Barber, for example, specified three models: “on demand”, “on retainer”, “full immersion”), but a center of performance expertise should exist. Only thing I am saying here is that while the approach works fine in traditional environment, it needs major changes in organization, tools, and skills when the scope of performance engineering should be extended (as in the case of agile projects).

Actually remedies are well known: automation, making performance everyone jobs (full immersion), etc. However they are not wide-spread yet.

Historically performance testing automation was almost non-existent (at least in traditional environments). Performance testing automation is much more difficult than, for example, functional testing automation (I use “automation” here as what should be done for “continuous testing”, i.e. process when you run test and get a report automatically for a new build without human intervention – not in its old way, when it meant using a tool; in performance we almost always use a tool). Setups are much more complicated. A list of possible issues is long. Results are complex (not just pass/fail). It is not easy to compare two result sets. So it is definitely much more difficult and would probably require much more human intervention, but it isn’t impossible.

However, the cost of performance testing automation is high. You need to know system well enough to make meaningful automation. Automation for a new system doesn’t make much sense – overheads are too high. So there was almost no automation in traditional environment [with testing in the end with a record/playback tool]. When you test the system once in a while before next major release, chances to re-use your artifacts are low.

It is opposite when the same system is tested again and again (as it should be in agile projects). It makes sense to invest in setting up automation. It rarely happened in traditional environments – even if you test each build, they are far apart and the difference between the builds prevents re-using the artifacts (especially with recorded scripts – API, for example, is usually more stable). So demand for automation was rather low and tool vendors didn’t pay much attention to it. Well, the situation is changing – we may see more automation-related features in load testing tools soon.

There are some vendor claiming that their load testing tools better fit agile processes, but it looks like in the best case it means that the tool is a little easier to handle (and, unfortunately, often just because there is not much functionality in it). Even if there is something that may be used for automation, like starting by a command line with parameters, it is difficult to find out.

At the moment, I read about few implementation of continuous performance testing – for example OpTier or Betfair (and few about measuring response times during functional testing in single-user mode – like this – which is a good step toward full-scale performance testing automation). Some prototypes, like this, are described– but without many details.

Probably we don’t see it more often because we don’t have much infrastructure for that kind of automation and performance testers may be not the best people to create complex integrated solutions from dozens of not-related pieces (as those who implemented it did). When we get more automation support in tools, we will probably see it changing.

By the way, I am not saying that automation would replace performance testing as we know it. Performance testing of new systems is agile and exploratory in nature and can’t be replaced by automation (well, at least not in the foreseen future). Automation would complement it – together with additional input from development offloading performance engineers from routine tasks not requiring sophisticated research and analysis.

It looks like agile methodologies are somewhat struggling with performance requirements (and non-functional requirements in general). Probably there are several reasons for that. One is that actually even traditional software development methodologies and processes never came with a good approach to handle performance requirements. They are, of course, considered in both literature and practical projects – but are usually handled rather in ad hoc manner. Actually the process of gathering and elaboration of performance requirements is rather agile in itself, and attempts to make it rigorous and formal look unnatural and have never fully succeeded – so it should be easier and more natural to do it as part of agile methods. Still the challenge of handling multidimensional and difficult to formalize performance requirements remains intact and the difference is rather in minor adjustments to agile processes than in the essence of performance requirements.

Another reason is that practical agile development is struggling with performance in general. Theoretically it should be a piece of cake: every iteration you have a working system and know exactly where you stand with the system’s performance. You shouldn’t wait until the end of the waterfall process to figure out where you are – on every iteration you can track your performance against requirements and see the progress (making adjustments on what is already implemented and what is not yet). Clearly it is supposed to make the whole performance engineering process much more straightforward.

Unfortunately, it looks like it doesn’t always work this way in practice. So such notions as “hardening iterations” and “technical debt” get introduced. Although it is probably the same old problem: functionality gets priority over performance (which is somewhat explainable: you need first some functionality before you can talk about its performance). So performance related activities slip toward the end of the project, and the chance is missed to implement a proper performance engineering process built around performance requirements.

Another issue here is that agile methods are oriented toward breaking projects into small tasks, which is quite difficult to do with performance (and many other non-functional requirements) – performance-related activities usually span the whole project.

There is no standard approach to specifying performance requirements in agile methods. Mostly it is suggested to present them as user stories or as constraints. And the difference is not so much in the way the requirements are presented, both ways rather use plain text.

User stories assume using a user voice form. Cohn, for example, suggests to use the “As a , I want , so that ” template for user stories (although he cautions that the user story template should only be used as a thinking tool, it should not be used as a fixed template). For constraints, both traditional expressions and user voice forms may be used.

The difference between user stories and constraints approaches is not in performance requirements per se, but how to address them during the development process. The point of the constraint approach is that user stories should represent finite manageable tasks, while performance-related activities can’t be handled as such because they usually span multiple components and iterations. Those who suggest to use user stories address that concern in another way – for example, separating cost of initial compliance and cost of ongoing compliance.

It looks like testingreflections.com, where I had blogged for a while, doesn’t exist anymore. One more confirmation that you can’t rely on somebody else long-term – who knows what would happen… Well, I still have some drafts of my old posts. Here is what I wrote about agile performance testing in 2009:

Two Views of Agile Unit Testing reminded me (indirectly) the QAForums Agile Performance Testing – Oxymoron? discussion about agile development and performance testing some time ago

Well, if we talk about agile development I don’t see any problem with performance testing at all: you have a working build each iteration – you test it. What is a problem? It should be synonym, not oxymoron.

All issues mentioned in the discussion boil down to one: nobody wants to pay for it (so no testers, no equipment, etc). Don’t see anything especially agile in it. Maybe companies still don’t switch to agile development for major projects – and don’t want to spend money for minor projects. Another explanation may be that in most traditional projects performance testing happens at the very end instead of every iteration (so require less resources) – but it not the way to do it properly.

I see significantly more problems doing performance testing properly during traditional development. I wrote a paper that has the same title, Agile Performance Testing, but completely different content: how to do performance testing in an agile way for any kind of projects (where agile projects look rather like a trivial case for me).

Well, returning to the original question of the discussion, there are a few articles in the Internet about the subject:

Agile Performance Testing by Jamie Dobson

An Explanation of Performance Testing on an Agile Team Part 1 and Part2 by Scott Barber

Chapter 6 of Performance Testing Guidance for Web Applications. Managing an Agile Performance Test Cycle.

Well, I still believe so. I still believe that performance testing in agile projects should be easier conceptually (although, of course, it would be a lot of technical difficulties – but who said that it should be easy?).

There are a few good newer articles addressing some aspects of the subject:

Top Ten Secret Weapons For Agile Performance Testing by Patrick Kua

Agile Performance Testing process AgileLoad whitepaper

Performance Testing in the Agile Enterprise by Scott Barber

Performance By Design – an Agile Approach by Jason Buksh

Still the subject is not covered much (and it looks like not actually practiced much – apart from involvement of traditional performance testing team in agile projects).

I started to prepare my talk for Belgium Testing Days (which promises to be a very interesting global testing conference) and the subject got me to very philosophical questions – will try to share my further thoughts about the subject in following posts.