Quasar "Seneca" Beta 3.13.5 Released

This is the beta 3.13 branch, not to be confused with the stable 3.10 branch.

We are very happy to announce the immediate availability of QuasarDB 3.13.5, a beta version update from 3.13.4.

This release fixes an important performance regression that was introduced in 3.13.4 when users would use the ORDER BY DESC statement.

This release also introduces a couple of performance and UX improvements, and you no longer need to specify a range for GROUP BY queries!

Get QuasarDB 3.13.5 here!

QuasarDB is also available on the AWS marketplace.

32-bit and 64-bit ARM binaries for Linux are now included with every release.

Change log

  • Protocol version 48
  • [api] Add support for ARRAY_ACCUM results for Apache Arrow conversion
  • [api] Add support for double natively in the key-value store
  • [api] Implement all missing "write" direct API functions
  • [daemon] Change the configuration parameter log_file_time_to_roll from seconds to milliseconds
  • [daemon] Check on startup that the volume complies with license restrictions
  • [daemon] More granular configuration of RocksDB's column family memory options
  • [net] Add support for ARRAY_ACCUM
  • [net] Improve performance of the batch writer
  • [query] $timestamp columns must now be explicitly selected
  • [query] Add support for PUT (key/value) in queries
  • [query] Extend the support for ASOF and aggregations
  • [query] Fix DESC performance regression introduced in 3.13.4
  • [query] Fix internal error for HAVING with empty columns
  • [query] Further improve HAVING query speed
  • [query] GROUP BY no longer requires a time range
  • [query] Improve DELETE FROM query speed
  • [shell] New "get_full_metadata" shell command to get all versions of an entry

by Edouard Alligand

Quasar "Seneca" Beta 3.13.4 Released

This is the beta 3.13 branch, not to be confused with the stable 3.10 branch.

We are very happy to announce the immediate availability of QuasarDB 3.13.4, a beta version update from 3.13.3.

This release is highly focused on performance and first time user experience (UX).

By performance, we mean working on the long tail where users would be experiencing underwhelming results in narrow use cases.

In the last three months, we went through every reported issue and analyzed and reworked every part of the engine as needed, little by little, to make sure those long tail issues are gone.

It was highly worthwhile: queries can have 10X to 20X performance improvement, especially queries using a combination of aggregation and LIMIT.

The order book Virtual Machine (VM) has been reworked to deliver even more performance while reducing the amount of data it needs to build a snapshot. That will significantly help users working on a very deep history.

Memory usage of the daemon has been dramatically reduced. Doing so required a deep dive into the memory allocator, but it was worth it! You will be able to do more with less.

On top of that, we worked on the first minutes user experience, everything related to setup and first API calls.

It goes from a better error message if you downloaded the wrong build for your CPU architecture to a more resilient installer that accounts for even the most exotic setups.

We've also changed some aggregation to make them less surprising for users coming from SQL databases.

Lastly, we fixed several OS X issues that prevented developers on a Mac from having a smooth experience with Quasar.

If you are still having issues, please mail us at [email protected]. Whether you're using the community version or not, we want to hear from you.

Oh, one more thing. Quasar now supports hot backups.

Get QuasarDB 3.13.4 here!

QuasarDB is also available on the AWS marketplace.

32-bit and 64-bit ARM binaries for Linux are now included with every release.

Change log

  • Protocol version 48
  • [api] Continuous queries can now be paced
  • [api] Correctly parse column names starting with "nan"
  • [api] Optimize client-side processing of queries for improved performance
  • [api] Support ORDERBOOK and SLIPPAGE in continuous queries
  • [daemon] Add more asynchronous pipelines statistics
  • [daemon] Better error messages when the CPU is missing required features
  • [daemon] Greatly reduce memory usage and improve eviction heuristics
  • [daemon] Improve the accuracy of memory statistics, introduce new ones
  • [daemon] Initial support for hot backups
  • [ingress] Support none values when doing "insert deduplicate"
  • [logging] Logs now use ISO8601 timestamps
  • [net] Improve .NET general compatibility
  • [osx] Improve general compatibility
  • [persistence] Allow for the rotation of RocksDB logs
  • [persistence] Improve default settings for better performance out of the box
  • [persistence] Optimize storage layout to improve lookup of cold data
  • [python] Made API fully zero copy wherever possible for up to 20X performance improvement
  • [query] Can now RESTRICT TO on multiple columns
  • [query] Enhance ORDERBOOK levels customization
  • [query] Enhance RESTRICT TO syntax to allow for more flexibility
  • [query] Fix corner cases related to comparison with NULL
  • [query] Greatly improve deduplication performance
  • [query] Greatly improve ORDERBOOK performance
  • [query] Greatly optimize queries with LIMIT statements
  • [query] Make behavior of first/last/min/max coherent with other aggregations
  • [query] Make the "slope" aggregation work with GROUP BY statements
  • [query] PREWHERE is now applied before RESTRICT TO
  • [query] RESTRICT TO is now applied before the WHERE statement

by Edouard Alligand

Quasar June 2022 Community update

As we are getting closer to our official 3.13.4 release, we felt the moment was right to share with our community what to expect from this new beta release.

Hot backups

Let’s start with a significant new feature: hot backups. In 3.13.4, you can now backup a live node incrementally, with control over the number of revisions. Backups happen in the background, and while a node is backed up, you can continue to ingest data and run queries.

Some important points to consider with the hot backup feature:

  • Backups happen in the background and will thus use some of your I/O capacity. It’s important to understand that this can impact your ingestion speed, even if backups have lower priority.
  • Quasar instances can be humongous (petabytes). One does not simply back up 10 petabytes of data.

Enhanced querying capability

In this update, we’ve enhanced the capabilities of the RESTRICT TO keyword and added more flexibility to ORDERBOOK:

  • RESTRICT TO can now accept more complex criteria making it very easy to handle multiple versions of the same data at a fantastic speed.
  • The ORDERBOOK and SLIPPAGE functions now offer more granularity and flexibility when selecting the levels (depth) range.

We’ve also fixed some quirks of the query language to make it behave more coherently and logically. For example, the treatment of the timestamp by the “first” aggregation was surprising for people coming from SQL databases. We fixed that.

Performance

Performance is the central focus of this release. Speed matters a lot at Quasar,  because our users typically have large datasets.

Based on your feedback, we’ve spent an (unreasonable) amount of time to work on several performance bottlenecks impacting queries across the board. These improvements revolve mostly around internal structure optimizations and improving the client-server communication logic to avoid unnecessary work.

These performance improvements can be drastic, queries which used to take 20ms now can take less than 1ms (20X improvement). These improvements also solve issues of specific queries taking an abnormal amount of time (sometimes a minute, even more) given their complexity.

The ORDERBOOK virtual machine, the piece of code responsible for executing raw orders to build the order book, has been reworked and we managed to get up to 100% speed improvement.

Performance isn’t just server-side but also client-side. For example, the Python API can be up to 20X faster thanks to a generalized usage of zero-copy.

Lastly, we've done several improvements that should result in an overall lower memory usage for identical loads.

Compatibility

Finally, we’ve made several quality-of-life improvements to improve compatibility of Quasar with containers and OS X.

Quasar should work on a broader range of Macs, out of the box, without needing to recompile, e.g. the Python API. If Quasar does not work correctly on your mac, please send a mail to [email protected], whether you’re a paying user or not, and we will look into the issue.

Another quality-of-life improvement we’ve made is to ensure the CPU verification code runs on a more extensive set of platforms. This way, if you run the Haswell optimized build on an older workstation, you will get a clear error message inviting you to download a different build.

Lastly, the .NET API could sometimes be unable to load, especially from a docker container. We fixed the issues that were causing that and made generally speaking the loading process more resilient.

Expected release date

We are working very hard to release Quasar 3.13.4 in June 2022.

Meanwhile, feel free to download Quasar, join our community channel on Slack, subscribe to our newsletter, or to contact us!

by Edouard Alligand

Quasar "Seneca" 3.13.3 Beta released

This is the beta 3.13 branch, not to be confused with the stable 3.10 branch.

We are very happy to announce the immediate availability of QuasarDB 3.13.3, a beta version update from 3.13.2.

In this release we have focused on enhancing the analytical power of Quasar, to support complex use case such as bitemporal timeseries data, orderbook slippage computation, new indicators, enhanced order book syntax, and suffix based search.

Get QuasarDB 3.13.3 here!

QuasarDB is also available on the AWS marketplace.

Note: 3.13.3 is not wire compatible with 3.13.2.

32-bit and 64-bit ARM binaries for Linux are now included with every release.

Change log

  • Protocol version 48
  • [daemon] Add built-in automatic version check
  • [daemon] Improve performance for very short lived queries
  • [nodejs] Fix several stability issues in the nodejs API
  • [protocol] Improve backward compatibility with new versioning logic
  • [query] Add support for ARRAY_ACCUM
  • [query] Add support for AS_TABLES (virtual tables built on keys)
  • [query] Add support for Coinbase market data in the ORDERBOOK function
  • [query] Add support for Relative Strength Index (RSI)
  • [query] Add support for RESTRICT TO (multi version management)
  • [query] Add support for SLIPPAGE (order book slippage computation)
  • [query] Add support for suffix in FIND
  • [query] Add support for TOP and FULL computation in the ORDERBOOK
  • [query] Fix several minor issues related to moving averages and exponential moving averages
  • [query] Fix subquery bug when parent and child share selected columns
  • [query] Changed the syntax of the ORDERBOOK function

by Edouard Alligand

Quasar "Seneca" 3.13.2 Beta Released

This is the beta 3.13 branch, not to be confused with the stable 3.10 branch.

We are very happy to announce the immediate availability of QuasarDB 3.13.2, a beta version update from 3.13.1.

This release brings important performance improvements across the board, and bring major new features such as a brand new low-latency, high-performance, cryptocurrencies exchange feed handler, support for EMA and MACD, and support for Apache Arrow as input and output.

Last but not least, real-time order book reconstruction is now out of private beta and available in this release, directly from the query language.

Get QuasarDB 3.13.2 here!

QuasarDB is also available on the AWS marketplace.

32-bit and 64-bit ARM binaries for Linux are now included with every release.

Change log

  • Protocol version 46
  • [api] Query results can optionally return Apache Arrow structures
  • [api] Remove deprecated bulk inserter
  • [api] Simplify symbol API
  • [api] The batch inserter can optionally accept Apache Arrow structures for input
  • [client] Greatly improve multithreading performance
  • [crypto] Low latency crypto feed handler with native Coinbase support
  • [kernel] Greatly improve cache heuristics for lower I/O usage
  • [query] Add support for "infinite" shard size
  • [query] Add support for escaped characters in quoted strings
  • [query] Add support for Exponential Moving Average (EMA)
  • [query] Add support for Moving Average Convergence Difference (MACD)
  • [query] Add support for ORDERBOOK for real-time order book reconstruction
  • [query] Fix a bug where GROUP BY + WHERE could incorrectly split a range
  • [query] Fix an edge case with last aggregations on certain GROUP BY queries
  • [query] Greatly improve the performance of DELETE WHERE DUPLICATE
  • [query] Greatly increased WHERE performance across the board
  • [security] Add support for transitive permissions on tags
  • [shell] Add a command to reload the user configuration manually

by Edouard Alligand

Quasar "Seneca" 3.13.1 Beta Released

This is the beta 3.13 branch, not to be confused with the stable 3.10 branch.

We are very happy to announce the immediate availability of QuasarDB 3.13.1, a beta version update from 3.13.0.

This version includes two important bug fixes. If you are using the new symbol column type or the Java API you are strongly encouraged to upgrade to 3.13.1.

Get QuasarDB 3.13.1 here!

QuasarDB is also available on the AWS marketplace.

32-bit and 64-bit ARM binaries for Linux are now included with every release.

Change log

  • Protocol version 46
  • [api] Fix a client-side data corruption when pushing multiple symbol columns at the same time
  • [api] Fix an issue with symbol and batch API parameters
  • [daemon] Fix a performance issue with ALTER TABLE ADD
  • [daemon] Properly take hierarchical cgroup memory limits into account
  • [java] Fix several stability issues
  • [query] Add support for ALTER TABLE DROP column
  • [query] Add support for ASOF INTERPOLATE and LOOKBACK
  • [query] Add support for direction for ASOF LOOKBACK
  • [query] Full support for moving averages

by Edouard Alligand