Testing in Production

I found Seth Eliot’s presentation A to Z Testing in Production: Industry Leading Techniques to Leverage Big Data for Quality at was very thought-provoking. Definitely a non-orthodox view on what is going on and what it is.

Seth listed four categories of testing in production:
-Passive Monitoring with Real Data
-Active Monitoring with Synthetic Transactions
-Experimentation on Real Users
-System Stress of the Service and Environment

Should admit that I expected to see only the last item – seeing first three was rather a surprise for me. I definitely agree that active monitoring and experimentation are testing – and thinking about them as testing is important and provides another view of these activities.

Not sure about passive monitoring. Probably the question is how we define testing. Just realized that I don’t recall a clear definition of testing. My understanding was that testing is exercising the system to check how it works and implies actions by testers. However Wikipedia says Software testing is an investigation conducted to provide stakeholders with information about the quality of the product or service under test. So if follow Wikipedia’s definition, passive monitoring may be testing (analysis of passive monitoring results is an investigation), but it is not how I understand it. If we speak about performance, performance analysis was always a way to investigate system’s behavior, but nobody ever named it testing (on my memory). In my opinion testing is when you apply load. Any thoughts?