QuasarDB "Seneca" 3.14.1 Released

We are thrilled to announce the immediate availability of QuasarDB 3.14.1, an update from 3.14.0.

This update brings major new features while being 100% compatible with 3.14.0. Features such as streaming aggregation (named aggregated tables) and automatic data expiration are now available to all users.

Streaming aggregations enable you to aggregate the data as it comes and store the aggregate, unlocking ultra-low latency analytics while crushing disk usage.

In addition, we’ve added numerous performance enhancements, query functionalities, and bug fixes, that make the upgrade to 3.14.1 highly recommended!

Change log

(w.r.t. 3.14.0)

  • Protocol version 48
  • [arm64] Optimized several functions to leverage ARM Neon SIMD
  • [compression] Slight improvement in (de)compression speed
  • [daemon] Upsert now properly replaces all duplicates
  • [kernel] Add support for TTL in tables
  • [kernel] General availability of AGGREGATED tables (streaming analytics)
  • [kernel] Overall improvement of insertion performance across the board
  • [linux] Fixed stack trace symbol dump
  • [orderbook] Can now generate order books on a list of timestamps and offsets
  • [orderbook] Significant performance improvement of the Currenex order book engine
  • [orderbook] The account ID column is now properly displayed
  • [query] Add round, ceil, and floor functions
  • [query] Add support for complex ASOF joins across multiple tables
  • [query] Add support for count($timestamp)
  • [query] Add support for interpolation in GROUP BY queries
  • [query] Add support for JSON operators
  • [query] Add support for LEAD() and LAG() for sliding window queries
  • [query] ASOF now has a consistent behavior when there are NULL values
  • [query] Can now drop a column that has a name made of a single character
  • [query] Columns with different symbol tables can now be compared
  • [query] Fix invalid first() and last() result for certain complex queries
  • [query] Significant improvement for GROUP BY spanning thousands of tables
  • [query] Support composition for correlation
  • [query] Support for HAVING with PIVOT

by Edouard Alligand

QuasarDB "Seneca" 3.14.0 Released

We are thrilled to announce the immediate availability of QuasarDB 3.14.0, an update from 3.13.7.

Multiple performance and fixes have been added in this release, as well as support for multiple readers from a single S3 bucket, improved S3 backup speed, and, last but not least, initial support for windowing functions with the SQL OVER syntax.

This version sets a new standard in terms of speed, compression, and analytical power for timeseries databases and has been battle tested over multiple petabytes data sets in use case as diverse as asset health monitoring, orderbook analysis, cybersecurity, and electric grid simulations.

Get QuasarDB 3.14.0 here!

Change log

(w.r.t. 3.13.7)

  • Protocol version 48
  • [api] Add a function to trim a single entry
  • [client] Compiler, build, cpu, and allocator are now in the –version output
  • [daemon] Add support for multiple instances over the same S3 bucket (one writer, many readers)
  • [daemon] Add support for short-term credentials for S3 authentication
  • [daemon] Sensitive parameters are now hidden from the log
  • [daemon] Significantly increased S3 backup speed
  • [daemon] Trimming multiple entries is now done in a dedicated background thread
  • [kernel] Fix an issue where old buckets could not be updated after a new column was added to a table
  • [kernel] Trimming a timeseries now includes the root
  • [odbc] Add support for log size limit and rotation
  • [odbc] Log verbosity is now configurable, and the default value has been lowered
  • [orderbook] Currenex ORDERBOOK function is more tolerant of errors
  • [orderbook] Fix a bug related to time ranges and Currenex data
  • [orderbook] Greatly improved the performance of the ORDERBOOK functions for Currenex data
  • [persistence] Reduces the chances for data loss in case of an operating system failure during a trimming operation
  • [persistence] S3 Backups can now be done using multiple threads
  • [query] Initial support for Windowing functions (SQL OVER())

by Edouard Alligand

Important update to the Quasar Python API

We just pushed an important bug fix to the Python API for 3.13.5. The bug could result in data being rendered incorrectly in the client (data in the server is not affected). All users of 3.13.5 are strongly encouraged to update.

The new version is 3.13.5.post2

To install explicitly:

pip install quasardb==3.13.5.post2

Updating should automatically fetch the new version:

pip install -U quasardb

For further information, refer to the documentation or contact us.

by Edouard Alligand

Quasar "Seneca" Beta 3.13.5 Released

This is the beta 3.13 branch, not to be confused with the stable 3.10 branch.

We are very happy to announce the immediate availability of QuasarDB 3.13.5, a beta version update from 3.13.4.

This release fixes an important performance regression that was introduced in 3.13.4 when users would use the ORDER BY DESC statement.

This release also introduces a couple of performance and UX improvements, and you no longer need to specify a range for GROUP BY queries!

Get QuasarDB 3.13.5 here!

QuasarDB is also available on the AWS marketplace.

32-bit and 64-bit ARM binaries for Linux are now included with every release.

Change log

  • Protocol version 48
  • [api] Add support for ARRAY_ACCUM results for Apache Arrow conversion
  • [api] Add support for double natively in the key-value store
  • [api] Implement all missing "write" direct API functions
  • [daemon] Change the configuration parameter log_file_time_to_roll from seconds to milliseconds
  • [daemon] Check on startup that the volume complies with license restrictions
  • [daemon] More granular configuration of RocksDB's column family memory options
  • [net] Add support for ARRAY_ACCUM
  • [net] Improve performance of the batch writer
  • [query] $timestamp columns must now be explicitly selected
  • [query] Add support for PUT (key/value) in queries
  • [query] Extend the support for ASOF and aggregations
  • [query] Fix DESC performance regression introduced in 3.13.4
  • [query] Fix internal error for HAVING with empty columns
  • [query] Further improve HAVING query speed
  • [query] GROUP BY no longer requires a time range
  • [query] Improve DELETE FROM query speed
  • [shell] New "get_full_metadata" shell command to get all versions of an entry

by Edouard Alligand

Quasar "Seneca" Beta 3.13.4 Released

This is the beta 3.13 branch, not to be confused with the stable 3.10 branch.

We are very happy to announce the immediate availability of QuasarDB 3.13.4, a beta version update from 3.13.3.

This release is highly focused on performance and first time user experience (UX).

By performance, we mean working on the long tail where users would be experiencing underwhelming results in narrow use cases.

In the last three months, we went through every reported issue and analyzed and reworked every part of the engine as needed, little by little, to make sure those long tail issues are gone.

It was highly worthwhile: queries can have 10X to 20X performance improvement, especially queries using a combination of aggregation and LIMIT.

The order book Virtual Machine (VM) has been reworked to deliver even more performance while reducing the amount of data it needs to build a snapshot. That will significantly help users working on a very deep history.

Memory usage of the daemon has been dramatically reduced. Doing so required a deep dive into the memory allocator, but it was worth it! You will be able to do more with less.

On top of that, we worked on the first minutes user experience, everything related to setup and first API calls.

It goes from a better error message if you downloaded the wrong build for your CPU architecture to a more resilient installer that accounts for even the most exotic setups.

We've also changed some aggregation to make them less surprising for users coming from SQL databases.

Lastly, we fixed several OS X issues that prevented developers on a Mac from having a smooth experience with Quasar.

If you are still having issues, please mail us at [email protected]. Whether you're using the community version or not, we want to hear from you.

Oh, one more thing. Quasar now supports hot backups.

Get QuasarDB 3.13.4 here!

QuasarDB is also available on the AWS marketplace.

32-bit and 64-bit ARM binaries for Linux are now included with every release.

Change log

  • Protocol version 48
  • [api] Continuous queries can now be paced
  • [api] Correctly parse column names starting with "nan"
  • [api] Optimize client-side processing of queries for improved performance
  • [api] Support ORDERBOOK and SLIPPAGE in continuous queries
  • [daemon] Add more asynchronous pipelines statistics
  • [daemon] Better error messages when the CPU is missing required features
  • [daemon] Greatly reduce memory usage and improve eviction heuristics
  • [daemon] Improve the accuracy of memory statistics, introduce new ones
  • [daemon] Initial support for hot backups
  • [ingress] Support none values when doing "insert deduplicate"
  • [logging] Logs now use ISO8601 timestamps
  • [net] Improve .NET general compatibility
  • [osx] Improve general compatibility
  • [persistence] Allow for the rotation of RocksDB logs
  • [persistence] Improve default settings for better performance out of the box
  • [persistence] Optimize storage layout to improve lookup of cold data
  • [python] Made API fully zero copy wherever possible for up to 20X performance improvement
  • [query] Can now RESTRICT TO on multiple columns
  • [query] Enhance ORDERBOOK levels customization
  • [query] Enhance RESTRICT TO syntax to allow for more flexibility
  • [query] Fix corner cases related to comparison with NULL
  • [query] Greatly improve deduplication performance
  • [query] Greatly improve ORDERBOOK performance
  • [query] Greatly optimize queries with LIMIT statements
  • [query] Make behavior of first/last/min/max coherent with other aggregations
  • [query] Make the "slope" aggregation work with GROUP BY statements
  • [query] PREWHERE is now applied before RESTRICT TO
  • [query] RESTRICT TO is now applied before the WHERE statement

by Edouard Alligand

Quasar June 2022 Community update

As we are getting closer to our official 3.13.4 release, we felt the moment was right to share with our community what to expect from this new beta release.

Hot backups

Let’s start with a significant new feature: hot backups. In 3.13.4, you can now backup a live node incrementally, with control over the number of revisions. Backups happen in the background, and while a node is backed up, you can continue to ingest data and run queries.

Some important points to consider with the hot backup feature:

  • Backups happen in the background and will thus use some of your I/O capacity. It’s important to understand that this can impact your ingestion speed, even if backups have lower priority.
  • Quasar instances can be humongous (petabytes). One does not simply back up 10 petabytes of data.

Enhanced querying capability

In this update, we’ve enhanced the capabilities of the RESTRICT TO keyword and added more flexibility to ORDERBOOK:

  • RESTRICT TO can now accept more complex criteria making it very easy to handle multiple versions of the same data at a fantastic speed.
  • The ORDERBOOK and SLIPPAGE functions now offer more granularity and flexibility when selecting the levels (depth) range.

We’ve also fixed some quirks of the query language to make it behave more coherently and logically. For example, the treatment of the timestamp by the “first” aggregation was surprising for people coming from SQL databases. We fixed that.

Performance

Performance is the central focus of this release. Speed matters a lot at Quasar,  because our users typically have large datasets.

Based on your feedback, we’ve spent an (unreasonable) amount of time to work on several performance bottlenecks impacting queries across the board. These improvements revolve mostly around internal structure optimizations and improving the client-server communication logic to avoid unnecessary work.

These performance improvements can be drastic, queries which used to take 20ms now can take less than 1ms (20X improvement). These improvements also solve issues of specific queries taking an abnormal amount of time (sometimes a minute, even more) given their complexity.

The ORDERBOOK virtual machine, the piece of code responsible for executing raw orders to build the order book, has been reworked and we managed to get up to 100% speed improvement.

Performance isn’t just server-side but also client-side. For example, the Python API can be up to 20X faster thanks to a generalized usage of zero-copy.

Lastly, we've done several improvements that should result in an overall lower memory usage for identical loads.

Compatibility

Finally, we’ve made several quality-of-life improvements to improve compatibility of Quasar with containers and OS X.

Quasar should work on a broader range of Macs, out of the box, without needing to recompile, e.g. the Python API. If Quasar does not work correctly on your mac, please send a mail to [email protected], whether you’re a paying user or not, and we will look into the issue.

Another quality-of-life improvement we’ve made is to ensure the CPU verification code runs on a more extensive set of platforms. This way, if you run the Haswell optimized build on an older workstation, you will get a clear error message inviting you to download a different build.

Lastly, the .NET API could sometimes be unable to load, especially from a docker container. We fixed the issues that were causing that and made generally speaking the loading process more resilient.

Expected release date

We are working very hard to release Quasar 3.13.4 in June 2022.

Meanwhile, feel free to download Quasar, join our community channel on Slack, subscribe to our newsletter, or to contact us!

by Edouard Alligand