Archive

Posts Tagged ‘testing’

Testing in Production

December 20th, 2012 1 comment
Share

I found Seth Eliot’s presentation A to Z Testing in Production: Industry Leading Techniques to Leverage Big Data for Quality at STP Online Summit was very thought-provoking. Definitely a non-orthodox view on what is going on and what it is.

Seth listed four categories of testing in production:
-Passive Monitoring with Real Data
-Active Monitoring with Synthetic Transactions
-Experimentation on Real Users
-System Stress of the Service and Environment

Should admit that I expected to see only the last item – seeing first three was rather a surprise for me. I definitely agree that active monitoring and experimentation are testing – and thinking about them as testing is important and provides another view of these activities.

Not sure about passive monitoring. Probably the question is how we define testing. Just realized that I don’t recall a clear definition of testing. My understanding was that testing is exercising the system to check how it works and implies actions by testers. However Wikipedia says Software testing is an investigation conducted to provide stakeholders with information about the quality of the product or service under test. So if follow Wikipedia’s definition, passive monitoring may be testing (analysis of passive monitoring results is an investigation), but it is not how I understand it. If we speak about performance, performance analysis was always a way to investigate system’s behavior, but nobody ever named it testing (on my memory). In my opinion testing is when you apply load. Any thoughts?

Categories: Testing Tags:

Best Practices and The Lean Startup

January 16th, 2012 No comments
Share

Just finished reading The Lean Startup by Eric Ries. It is definitely a great book. And probably it says as much about software development as about entrepreneurship (most examples are from software development). It should be a must reading about agile software development.

However, reading The Lean Startup, the discussion about “best practices” popped into my head again and again. I have a more moderate position towards “best practices”. Maybe “best practice” and “requirement” are not the best terms if we consider them from the original meaning of English words, but the meaning of these notions deviated from the original word meaning and “best practice”, as far as I understand, means a good practice in an appropriate context – not “the best” “practice” (as “hot dog” is not “hot” “dog” anymore). And it looks like attempts to replace “best practice” and “requirement” with something else rather confuse people – at least I haven’t seen a good replacement yet. So I don’t think that words “best practices” are especially dangerous, it is just any good idea / practice / framework applied outside of the proper context may hurt.

No words “best practices” are in the book, all practices described are good in the specified context [of entrepreneurship], the author himself warns many times against blind application, but I clearly see how people would start to use some of the lean startup principles out of the appropriate context.

Let’s look at testing. Testing (and performance testing) is a way to mitigate risks. My interpretation of Eric Ries’ message is that don’t be abscessed by possible risks when you don’t have anything to lose yet. Nothing to say against it. However at some point you are getting something to loose, and you need to introduce some practices to mitigate the risks. How much – depends on your business. Perhaps the main example of “The Lean Startup”, IMVU, has higher tolerance to risk than most other business (single-site, not critical business, low cost of losing a user). I, of course, understand that you can’t cover everything in a single book, but I am still concerned that the topic of introducing of risk-mitigation was not covered. Actually testing was mentioned at least once: p.189 “we had an extensive set of automated tests that assured that after every change our product still worked as designed”. So even in IMVU, with their relatively high risk tolerance and early stage, they had “an extensive set of automated tests”. I hope that other readers of the book won’t miss that, but I am afraid that many will.

By the way, it is interesting what would be opposite to lean? Probably not fat (although it is what using the word “lean” hints) – you probably don’t want to have fat in whatever approach you use. Maybe “Build to Grow”? Lean means that you do absolute minimum (accumulating technical debt) – so the opposite would be if you build infrastructure / framework to grow from the beginning. So, I guess, the dimension we are talking here about is Build to Grow – Lean. Maybe it would be a next book explaining when you need to be very lean and when you need to start building up muscles?

Application of Agile Principles to Testing

January 12th, 2012 2 comments
Share

And now I continue my post Can Processes Be Agile? into its application to testing (I broke it into two to avoid complete mess).

In a way I think that Exploratory Testing is an application of pure Agile principles to testing. And it looks like adherence to pure agile principles makes it rather difficult to explain – I guess that What Exploratory Testing is not series is an evidence how had is to apply pure agile principles without bringing it down to just another described in details process.

I have a small stake here: in 2008 I wrote a paper titled Agile Performance Testing. I shared a few ideas on how to do performance testing in a more agile way there (using the word agile according to my reading of Agile Manifesto). By the way, I never meant that it is a new way of performance testing – rather a collection of several practices that, I believe, good performance engineers use (consciously or unconsciously). The paper didn’t touched at all performance testing in agile software development, so some attendees were disappointed (well, in its latest reincarnation I presented at Agile Testing Days I added slide 47 about it). Should admit that the title was chosen to provoke interest to the topic and it fulfilled this purpose well. Still I was periodically wondered if the name was correct – and still think that it does.

Moreover, considering that Exploratory Testing is, in my current understanding, is application of agile principles to testing, I wonder if I should name the next reincarnation of this presentation “Exploratory Performance Testing”. Indeed the point I am trying to convey is that simultaneous learning, test design and test execution is even more important in performance testing. In functional testing you may have something (like use cases) describing how the system should work, but you don’t have anything that would describe how the system would behave from the performance and resource utilization point of view (performance requirements saying what are the limits stakeholders wants the system to achieve, but that’s the best you can hope to get).

At the CMG’12 conference we had an open discussion “New Approaches to Performance Testing: An Open Discussion on Plans, Experiments and Points In-Between”. The point I was trying to convey was that you need to experiment in performance testing to learn how the system behaves; just following a formal plan is less effective and often leads to missing performance problems altogether or to prolonged agony of performance troubleshooting. I, of course, mean a system of experiments based on the information we have about the system and results we are getting – it looks like some people believe that to experiment means to do random things. Well, it looks like I wasn’t too successful in this. Probably I should learn more how people advocate Exploratory Testing – but it looks like it is not easy to advocate something that doesn’t look a straightforward process.

New version of Oracle Application Testing Suite (former Emprix)

April 29th, 2010 No comments
Share

New version of Oracle Application Testing Suite 9.10 (former Emprix – now part of Oracle Enterprise Manager) was released recently. It is available for download (subject to OTN License). A lot of additional materials available (including webinars, white papers, etc.).

This version includes maintenance improvements to the existing features as well as new features outlined below (from readme):

  • Oracle Fusion/ADF Functional Testing Accelerator – Enables enhanced support for automated functional testing of Fusion/ADF applications. Extends ATS’s Web functional testing capabilities by adding native support for ADF GUI components for object identification, automation, and validation. Oracle ADF is a key technology in Fusion Applications and can be used to build custom applications.
  • Oracle Fusion/ADF Load Testing Accelerator – Enables enhanced support for automated load testing of Fusion/ADF applications. Extends ATS’s Web/HTTP load testing capabilities by adding a custom load testing correlation library and related scripting enhancements for Fusion/ADF load testing.
  • Script Assets Manager – Standardizes management of scripts and related assets like Data Bank files, Object Libraries, and Script Functions.
  • Enhanced Script Functions Support – Provides an easier way for users to create shared functions through the OpenScript user interface and call those functions from any test script to create more modular, reusable script components.
  • Oracle Real User Experience Insight (RUEI) Script Import – Allows users to generate Oracle Application Testing Suite load test scripts from the Oracle Real User Experience Insight product by first exporting real user session transaction data from RUEI and then importing it into OpenScript to generate a load test script.
  • Oracle E-Business Suite R12 Test Starter Kit / Sample Scripts – Provides a test starter kit with sample test scripts for Oracle EBS R12 applications based on the sample EBS “Vision” database.
  • OpenScript Tree View Debugger – Provides debugging capabilities from the OpenScript Tree View including the ability to set breakpoints, step through script nodes, step over, pause, and watch variables. Previously, script debugging was only available from OpenScript’s Java Code View.
  • Download Manager – Simplifies Oracle Application Testing Suite load test scripts by filtering out embedded HTTP URL requests for Web page resources (i.e. images, js, css, etc.) during script recording in OpenScript and then giving users the option of whether to automatically find and request those resources on playback in OpenScript and Oracle Load Testing.
  • Result Code Verification – Makes it easier to adjust the behavior of an OpenScript test script based on the outcome of script commands by providing a result code that can be queried after command executes.
  • Oracle Load Testing Reporting Enhancements – Provides drill-down capabilities that link OLT’s performance summary report to corresponding load test graphs for both real-time and post-run reporting.
  • Enterprise Manager J2EE Middleware Diagnostics Integration – Provides integration between Oracle Load Testing and Enterprise Manager that enables users to view J2EE middleware performance diagnostics during load tests for identifying bottlenecks under load.
  • Oracle Database ServerStats Profile – New Oracle Load Testing ServerStats monitoring profile for Oracle Database to capture key performance metrics during a load test.

CMG’10 – Call for Papers

April 29th, 2010 No comments
Share


The Computer Measurement Group (CMG) calls for papers and presentations
for for CMG’s 36th International Conference to be held in Kissimmee, Florida, December 6th through 10th, 2010. The 2010 CMG conference will cover all areas of systems management, including but not limited to: capacity planning, management and reporting, modeling and statistics, measurement, tuning, performance engineering and load testing, as well as the latest developments in the overall field of computer performance evaluation.

Since 1975 CMG is a volunteer organization of performance professionals and the CMG conference is the best place to learn about performance analysis, capacity planning, and related subjects.

This year CMG has separate subject areas for Load Testing and Performance Engineering.

Mentoring is available.

Proceedings up to 2007 are available to everybody (free registration required).

There is a good newsletter and local groups.