Performance Testing at MongoDB
MongoDB has the most advanced continuous performance testing I know about. It quite could be that some other companies have something interesting too – but they are not sharing that. [Another interesting project in that area is keptn, but that is about it. If I missed something interesting, please let me know.] MongoDB shared a lot of information on how we do performance testing and even open sourced some parts of it. One of main reasons I joined MongoDB recently was my excitement about the technology. As it was already described in detail by my colleagues, no need for me to write about it – I can just list references.
First, to set up the scene, MongoDB has its own Continuous Integration. Evergreen Continuous Integration: Why We Reinvented The Wheel, blog post from July 27, 2016, explains why and how it was done. Continuous performance testing is built on the top of Evergreen.
David Daly’s presentation at LTB 2020, How to Waste Time and Money Testing the Performance of a Software Product, is probably a good introduction [Video] — [Slides].
The MongoDB Podcast, Ep. 34 (2020), Performance Testing with David Daly, is another good introduction.
These two papers provide many more insights:
Automated system performance testing at MongoDB , DBTest 2020 [Video]
The Use of Change Point Detection to Identify Software Performance Regressions in a Continuous Integration System , ICPE 2020 [Video]
One of most important issues in continuous performance testing is variability. Blog post from April 30, 2019, Reducing variability in performance tests on EC2: Setup and Key Results, address that specific issue.
The latest update will be presented at ICPE 2021 by David Daly on April 21, 2021. The paper Creating a Virtuous Cycle in Performance Testing at MongoDB and pre-recoded video are already available.
David will also talk at the Workshop on Education and Practice of Performance Engineering (WEPPE) about Performance Engineering and Database Development at MongoDB. As well as we will have the Performance Testing in DevOps panel at the Load Testing and Benchmarking workshop where I would be one of the panelist. Both events would be on April 19, 2021.
