The Intersection of Performance and Functional Testing

The theme for WOPR 16 was announced: The Intersection of Performance and Functional Testing.

One my colleague commented it in the following way:

The fact of establishing that WOPR16 session, from my point of view, just answers on the main question about correlation in between those, IMHO absolutely different areas, as functional and performance testing. If it wasn’t so obvious, that those areas have very limited overlap, then we would see a LOT of evidences in real life ineternet. But, it doesn’t happen.

While, I’m not so firm about areas in general, both of them are testing: functional and performance. But I’m rather solid that people are working in those areas must have different sets of skills. I haven’t seen many examples (if only seen at all) of people coming into performance analysis from functional testing. At the same time all really experienced performance analysts I’ve met came either from development or admin roles (Unix admin, network admin, dba, …). People just need to be willing go deeper into analysis of perf. issues, working with a wide set of profiling tools and very often in cooperation with development, and not just report resultsproblems by making script and running a test. The last part – preparing script and executing a test – is just 20% of responsibilities of a really good perf. engineer.

From one side, I definitely agree with that. I even once upon a time wrote an article highlighting differences between functional and performance testing (it is in a slightly adjusted form as far as the original topic didn’t get much interest ).

From another side, I see some areas where synergy can be achieved. Synergy, not merge – I still believe that these are different activities handled by [usually] different people. For example:

1) Probably functional testers should measure transaction performance during their functional tests. They cover much more functionality, do it end-to-end, and are much more systematic. They easily can track one-user performance numbers and raise alarm early in case of degradation. Yeah, I haven’t seen it much. Some reasons: automated functional testing isn’t as wide spread as it looks; functional testing tools don’t have good functionality to measure and track performance; both areas are often quite separated; many functional testers still need a little bit of performance education; multiple people often using the same environment.

2) Adding functional / GUI / end-user scripts to load testing. Can be useful in many ways, including validation of protocol-based scripts, seeing real end-user performance, covering more functionality, catching more subtle issues related to timing/order of downloading. The script shouldn’t be the same (as it was in old Empirix – although it would be nice when technically possible), but it should be running smoothly inside load testing to make it practical. I did it a few times with LoadRunner/WinRunner long time. If somebody gives me such scripts – I don’t see why not to use them (considering that they are integrated in the Load Testing tool).

3) When we get to the point when you start to test familiar systems again and again (new builds of the same products in different configurations and with different sets of data), a kind of routine performance regression testing, the process becomes somewhat more similar to functional testing. And functional testing has developed some interesting methods of choosing what tests to run – perhaps something may be used in more routine performance testing.

4 Responses to "The Intersection of Performance and Functional Testing"

Greg McNelly says:

February 7, 2011 at 9:36 pm

Thanks for the mention and the feedback, Alexander.

On one hand, I concur with your colleague’s comments – that there seems to be little overlap because we just don’t see it. On the other hand, I’ve had some experiences, faced some challenges, and am seeing some changes in my own shop that have me wondering…

On one recent project, it was the performance testing of a data replication process that uncovered more functional bugs than the ‘functional’ testing. Increasingly aggressive project schedules are making it harder for me to delay performance testing until functional testing stabilizes. Some of our performance testing teams are solving data generation and environment setup problems in ways that would also benefit our functional testing teams, but they aren’t always talking. And the grey-box testing skills of some of our functional testers has begun to resemble the nature and degree of the skill set we see in our performance testers.

Your first two numbered points in your blog post make sense to me. Most functional automation I work with now automatically captures execution times. I would be interested to hear if you have an experience running functional scripts during a performance test. I’ve seen it done, though mostly just as a sanity check. Your third point – on repetitive testing – pains me a bit as I usually question the value of this type of testing, but your musing about whether some of the functional test selection techniques could be used for performance is interesting.

In your paper, your assertion that load testing is inherently more complex than functional testing intrigues me – I don’t know that I agree with you or not – but it makes for an interesting angle of approach to this topic. Workload verification might be another compelling intersection of performance and functional testing. I hear what you’re saying about load testing taking longer than functional testing and being more difficult to approach asynchronously, but I wonder why we accept this – why not run functional testing as a precursor or concurrently to load testing in the same environment? Why do environments have to be so separate and sterile (production is not)? Why not approach functional bug-fixing synchronously? I would like to hear from colleagues who have tried, whether they succeeded or not.

Your blog post and paper have helped me to continue thinking about the topic. Again, thanks for the feedback. I hope you can formulate a submission for the workshop.
Tweets that mention Alex Podelko » The Intersection of Performance and Functional Testing -- Topsy.com says:

February 7, 2011 at 10:12 pm

[…] This post was mentioned on Twitter by Greg McNelly, Joris Meerts. Joris Meerts said: #testing news: The Intersection of Performance and Functional Testing http://bit.ly/fmoTMK […]
Alex Podelko says:

February 10, 2011 at 10:23 pm

Greg,

Yes, I also found a lot of functional errors, but I can’t say that it was during performance tests per se. Rather during performance test setups: when I am trying to create a distributed install or working with realistic data. Unfortunately functional testing is often done on a single-machine setup and with simple data sets. But it rather means that functional testing was not quite adequate.

Actually in my paper I didn’t [intend to] say that performance testing is more complex than functionality testing. My current view is that both are complex, but in their own way. The complexness of functional testing is in test design: how to choose a meaningful set of tests to ensure adequate product quality in the given timeframe. And this looks pretty difficult if done properly (although I suspect that quite often it is done in a very simplistic way without many thoughts). The complexness of performance testing is in system analysis. I guess that it is the reason why quite different skills are required. In my paper I stressed what is different / more complex in performance testing, not talking much about what is more complex in functional testing; but there are challenges on the functional testing side too for sure.

The synchronous work of developers and testers is very expensive and rarely could be seen in large development groups. You need to get immediate attention of qualified developers when thousands of important things are going on. Asynchronous work is just much more efficient when possible. The problem is that it is often impossible in performance testing.

The main advantage of performance testing is that you apply reproducible workloads in a controlled environment. So you can clearly see the performance impact of every change you apply to the system and may track down every issue. Tracking down issues and tuning the system in production are major challenges just because the load is very diverse and not reproducible. Doing something else in the same environment with performance testing kills this advantage: now when you see performance degradation you don’t know either it caused by your change or by impact of another user.

Alex
Kiran Badi says:

May 16, 2011 at 12:32 pm

I think Alex performance tester requires the the skills of the functional testing for the reason that performance testing is all about assuring the functionality of the application under load.Though they require different skillsets, their goal is same,assuring that the functionality is up at all times.Assuring the functionality is common in both cases.
Sometimes bugs which look functional are infact not a functional bugs, they are triggered to appear under load, and sometimes bugs which looks performance isnt performance bugs ,they are functional bugs.In almost all the cases,if functional testing is done with out of box thinking,it reduces considerable effort in performance testing and Security testing.Its just that person doing functional testing needs to have out of the box strategy.

Again there are some performance bugs which needs to be reproduced with single users,here again skillsets of functional testing is required,sometimes people dont believe screenshots,they need live desktop sharing.So Performance testers needs to know as how to reproduce the bug,and I believe all these skills are part of functional testing domain.

Functional testing scripts can be leveraged for performance testing depending on the objectives of the test,if we are measuring the GUI response time, then probably I agree we can use functional testing scripts but again using them for measuring server capacity would be wrong step.

My 2 cents on this.

Leave a Reply