Application of Agile Principles to Testing
And now I continue my post Can Processes Be Agile? into its application to testing (I broke it into two to avoid complete mess).
In a way I think that Exploratory Testing is an application of pure Agile principles to testing. And it looks like adherence to pure agile principles makes it rather difficult to explain – I guess that What Exploratory Testing is not series is an evidence how had is to apply pure agile principles without bringing it down to just another described in details process.
I have a small stake here: in 2008 I wrote a paper titled Agile Performance Testing. I shared a few ideas on how to do performance testing in a more agile way there (using the word agile according to my reading of Agile Manifesto). By the way, I never meant that it is a new way of performance testing – rather a collection of several practices that, I believe, good performance engineers use (consciously or unconsciously). The paper didn’t touched at all performance testing in agile software development, so some attendees were disappointed (well, in its latest reincarnation I presented at Agile Testing Days I added slide 47 about it). Should admit that the title was chosen to provoke interest to the topic and it fulfilled this purpose well. Still I was periodically wondered if the name was correct – and still think that it does.
Moreover, considering that Exploratory Testing is, in my current understanding, is application of agile principles to testing, I wonder if I should name the next reincarnation of this presentation “Exploratory Performance Testing”. Indeed the point I am trying to convey is that simultaneous learning, test design and test execution is even more important in performance testing. In functional testing you may have something (like use cases) describing how the system should work, but you don’t have anything that would describe how the system would behave from the performance and resource utilization point of view (performance requirements saying what are the limits stakeholders wants the system to achieve, but that’s the best you can hope to get).
At the CMG’12 conference we had an open discussion “New Approaches to Performance Testing: An Open Discussion on Plans, Experiments and Points In-Between”. The point I was trying to convey was that you need to experiment in performance testing to learn how the system behaves; just following a formal plan is less effective and often leads to missing performance problems altogether or to prolonged agony of performance troubleshooting. I, of course, mean a system of experiments based on the information we have about the system and results we are getting – it looks like some people believe that to experiment means to do random things. Well, it looks like I wasn’t too successful in this. Probably I should learn more how people advocate Exploratory Testing – but it looks like it is not easy to advocate something that doesn’t look a straightforward process.
Very good point regarding Performance testing being Exploratory, I hadn’t thought of it in that way before.
But on reflection, we’re given some testing objectives and goals to achieve, we have a set of testing sessions in which to test and achieve the objectives / prove the goals. We start testing, perhaps doing a baseline to see roughly where performance sits, then we move on to more directed testing based on what we learn.
Great insight!
Mark.
Mark,
To clarify my point, here are a few points why learning and flexibility is even more important in performance testing than in functional testing:
-You may need to do an unknown number of tuning and performance troubleshooting iterations before you get system to a proper shape. The results you got before full tuning and fixing all performance issues have very limited value.
-Your first results are input to tuning and performance troubleshooting: the more you know about the system, the better you may interpret them and provide input for tuning and performance troubleshooting. Just throwing “it doesn’t fit requirements” or “it failed” over the wall to development has very low efficiency.
-In many cases to narrow down the issues you need to modify your tests based on what you learned about the system.
-Often found issue blocks you from further testing (either because it doesn’t work at all, or because it is already clear that results would be quite different after the fix, so you’d need to throw away results anyway). If you know enough about the system, you can try to figure out tests that still would make sense in this situation.
-Performance testing is always only an approximation of the real work even if you doing it in the production environment and with production data (that, unfortunately, isn’t always the case). So it is always good to know how sensible results are to, for example, load and data. You don’t want to report that the system meets performance requirements just to find out next month than after increasing the number of users and the amount of data by 5% the system slows down to a crawl.
-Performance testing results and observations usually are the main input for capacity planning and production. In the ideal case you want to provide meaningful information to allow proper capacity planning (for now and the future) and set expectation for what you should see in production (so you continue to track system’s performance and verify that what you have tested indeed make sense and represents production in the way you thought it does).
Alex