Improving our release process
You might have noticed that we didn’t release QuasarDB 3.9.0 on May 4th, 2020, as our previous release schedule would have commanded.
Since the beginning of the year, QuasarDB’s user base grew faster than we anticipated, creating many pain points in our release process, especially the QA phase.
Having more customers means typology changes. Early adopters will favor features for stability, pushing you to release as often as possible.
However, early adopters are atypical. Users want a solution for their specific problem and no longer touch the system once that works.
The problems we ran into
Our previous release schedule was:
- One release candidate every third Monday of the month
- One release every first Monday of the month
On paper, this looks like a tight release schedule; the problem is that it didn’t leave enough room to test thoroughly new important features or performance improvements.
Only one or two weeks would separate a release candidate from the official release, which was less than the full QA process, including feedback from our users.
The benefit was to have features available sooner to our customers, but administrators are reluctant to upgrade production database setups monthly, no matter how fluid the migration process is.
This tight release process impacted the quality of the 3.6 and 3.7 releases, which suffered several performance and usability issues.
This release process was inadequate and needed to be changed.
Our solutions
On the one hand, our user base appreciated having regular new builds to test new features and give us feedback about the direction the product was taking. Still, on the other hand, they wanted first and foremost their production to be stable!
We’ve deliberated internally, and these are our conclusions:
- We need to provide stable builds that have been through a thorough QA process (including our shadow environments reproducing the most stressful production environments);
- The stable builds don’t need to happen more than four times a year;
- One a stable build is released, only bug fixes will be applied to it—no new feature;
- New features need to be made available in clearly identifiable “beta” builds;
- Beta should be released at least monthly, enabling us to deploy it on our shadow environments, build demos, and deliver it to our early adopters.
- A shadow environment is a 1:1 replication of an existing customer production setup whose goal is to test the behavior of a new build.
Our new release process
Starting the next month, QuasarDB will be available in two versions.
- Stable builds, with an even minor version number (3.8.x, 3.10.x, 3.12.x…)
- Beta builds, with an odd minor version number (3.9.x, 3.11.x, 3.13.x…)
Beta release process
The beta build is where development happens. At some point in the development process, we branch a beta branch from master.
Master is always:
- Built and automatically tested (unit and integration tests) on FreeBSD, Linux (ARM32, ARM64, AMD64), OS X(AMD64), and Windows (IA32 and AMD64);
- Statically analyzed (Clang, Visual Studio);
- Dynamically analyzed (Clang, Valgrind, Visual Studio);
- At least one other developer has reviewed every commit.
Once this beta branch is created, we cherry pick features, fixes, and improvements from master and merge that into the beta release, and release new versions as we go.
We increase the patch number at each beta release. For example 3.9.0, 3.9.1, 3.9.2, etc.
Stable release process
Every 3 to 4 months, we release a new stable version. The exact duration will depend on how the beta behaves, the feedback we get from our users, and our confidence level in the build.
Five weeks before the release, we branch a stable version from the beta branch. If the current beta is 3.9, we thus branch a 3.10 version.
This branch will only receive bug fixes, and a thorough QA process will start.
This process includes, for example, high-speed multi-terabytes data loading, high-concurrency stress tests, shadow environments deployments, but also, UX assessment, and manual quality control.
Once the build passed all tests successfully, we release a stable release.
This stable release will only have bug fixes and resides in a separate branch. The bug fixes may come from either bug reports or problems identified in the next cycle of the beta release.
After the release of a stable build, we will release a new beta, branched from master, and the cycle restarts.
That beta will feature all improvements that happened in the master since the creation of the stable branch.
For example, if we release 3.10.0 in August, 3.11.0 will be released in September.
Looking forward
We just released QuasarDB 3.8.2, which has many important bug fixes.
We anticipate releasing 3.9 in early June and stabilize the new process this summer. Our goal is to have 3.10 released before this fall, and 3.12 before the end of the year.
Curious about QuasarDB? Why don’t you try our free of charge community edition?
Need to a one-click tick database? Try QTickDB!