Quasar "Seneca" Beta 3.13.4 Released

This is the beta 3.13 branch, not to be confused with the stable 3.10 branch.

We are very happy to announce the immediate availability of QuasarDB 3.13.4, a beta version update from 3.13.3.

This release is highly focused on performance and first time user experience (UX).

By performance, we mean working on the long tail where users would be experiencing underwhelming results in narrow use cases.

In the last three months, we went through every reported issue and analyzed and reworked every part of the engine as needed, little by little, to make sure those long tail issues are gone.

It was highly worthwhile: queries can have 10X to 20X performance improvement, especially queries using a combination of aggregation and LIMIT.

The order book Virtual Machine (VM) has been reworked to deliver even more performance while reducing the amount of data it needs to build a snapshot. That will significantly help users working on a very deep history.

Memory usage of the daemon has been dramatically reduced. Doing so required a deep dive into the memory allocator, but it was worth it! You will be able to do more with less.

On top of that, we worked on the first minutes user experience, everything related to setup and first API calls.

It goes from a better error message if you downloaded the wrong build for your CPU architecture to a more resilient installer that accounts for even the most exotic setups.

We've also changed some aggregation to make them less surprising for users coming from SQL databases.

Lastly, we fixed several OS X issues that prevented developers on a Mac from having a smooth experience with Quasar.

If you are still having issues, please mail us at [email protected]. Whether you're using the community version or not, we want to hear from you.

Oh, one more thing. Quasar now supports hot backups.

Get QuasarDB 3.13.4 here!

QuasarDB is also available on the AWS marketplace.

32-bit and 64-bit ARM binaries for Linux are now included with every release.

Change log

  • Protocol version 48
  • [api] Continuous queries can now be paced
  • [api] Correctly parse column names starting with "nan"
  • [api] Optimize client-side processing of queries for improved performance
  • [api] Support ORDERBOOK and SLIPPAGE in continuous queries
  • [daemon] Add more asynchronous pipelines statistics
  • [daemon] Better error messages when the CPU is missing required features
  • [daemon] Greatly reduce memory usage and improve eviction heuristics
  • [daemon] Improve the accuracy of memory statistics, introduce new ones
  • [daemon] Initial support for hot backups
  • [ingress] Support none values when doing "insert deduplicate"
  • [logging] Logs now use ISO8601 timestamps
  • [net] Improve .NET general compatibility
  • [osx] Improve general compatibility
  • [persistence] Allow for the rotation of RocksDB logs
  • [persistence] Improve default settings for better performance out of the box
  • [persistence] Optimize storage layout to improve lookup of cold data
  • [python] Made API fully zero copy wherever possible for up to 20X performance improvement
  • [query] Can now RESTRICT TO on multiple columns
  • [query] Enhance ORDERBOOK levels customization
  • [query] Enhance RESTRICT TO syntax to allow for more flexibility
  • [query] Fix corner cases related to comparison with NULL
  • [query] Greatly improve deduplication performance
  • [query] Greatly improve ORDERBOOK performance
  • [query] Greatly optimize queries with LIMIT statements
  • [query] Make behavior of first/last/min/max coherent with other aggregations
  • [query] Make the "slope" aggregation work with GROUP BY statements
  • [query] PREWHERE is now applied before RESTRICT TO
  • [query] RESTRICT TO is now applied before the WHERE statement

by Edouard Alligand

Quasar June 2022 Community update

As we are getting closer to our official 3.13.4 release, we felt the moment was right to share with our community what to expect from this new beta release.

Hot backups

Let’s start with a significant new feature: hot backups. In 3.13.4, you can now backup a live node incrementally, with control over the number of revisions. Backups happen in the background, and while a node is backed up, you can continue to ingest data and run queries.

Some important points to consider with the hot backup feature:

  • Backups happen in the background and will thus use some of your I/O capacity. It’s important to understand that this can impact your ingestion speed, even if backups have lower priority.
  • Quasar instances can be humongous (petabytes). One does not simply back up 10 petabytes of data.

Enhanced querying capability

In this update, we’ve enhanced the capabilities of the RESTRICT TO keyword and added more flexibility to ORDERBOOK:

  • RESTRICT TO can now accept more complex criteria making it very easy to handle multiple versions of the same data at a fantastic speed.
  • The ORDERBOOK and SLIPPAGE functions now offer more granularity and flexibility when selecting the levels (depth) range.

We’ve also fixed some quirks of the query language to make it behave more coherently and logically. For example, the treatment of the timestamp by the “first” aggregation was surprising for people coming from SQL databases. We fixed that.

Performance

Performance is the central focus of this release. Speed matters a lot at Quasar,  because our users typically have large datasets.

Based on your feedback, we’ve spent an (unreasonable) amount of time to work on several performance bottlenecks impacting queries across the board. These improvements revolve mostly around internal structure optimizations and improving the client-server communication logic to avoid unnecessary work.

These performance improvements can be drastic, queries which used to take 20ms now can take less than 1ms (20X improvement). These improvements also solve issues of specific queries taking an abnormal amount of time (sometimes a minute, even more) given their complexity.

The ORDERBOOK virtual machine, the piece of code responsible for executing raw orders to build the order book, has been reworked and we managed to get up to 100% speed improvement.

Performance isn’t just server-side but also client-side. For example, the Python API can be up to 20X faster thanks to a generalized usage of zero-copy.

Lastly, we've done several improvements that should result in an overall lower memory usage for identical loads.

Compatibility

Finally, we’ve made several quality-of-life improvements to improve compatibility of Quasar with containers and OS X.

Quasar should work on a broader range of Macs, out of the box, without needing to recompile, e.g. the Python API. If Quasar does not work correctly on your mac, please send a mail to [email protected], whether you’re a paying user or not, and we will look into the issue.

Another quality-of-life improvement we’ve made is to ensure the CPU verification code runs on a more extensive set of platforms. This way, if you run the Haswell optimized build on an older workstation, you will get a clear error message inviting you to download a different build.

Lastly, the .NET API could sometimes be unable to load, especially from a docker container. We fixed the issues that were causing that and made generally speaking the loading process more resilient.

Expected release date

We are working very hard to release Quasar 3.13.4 in June 2022.

Meanwhile, feel free to download Quasar, join our community channel on Slack, subscribe to our newsletter, or to contact us!

by Edouard Alligand