Quasar "Seneca" 3.13.0 Beta Released

Where is 3.11?

You may have noticed that we skipped the 3.11 (beta) and 3.12 (stable) releases altogether. We took the decision to kill a branch of Quasar and work on a major improvement of the I/O engine. Since there were internal releases named 3.11, to avoid any confusion we decided to number the official new beta release 3.13.

New "Seneca" QuasarDB engine

This new beta branch introduces a new version of the QuasarDB engine.

The new engine, named after the Roman stoic philosopher Seneca the Younger, is the result of one year of intense R&D focused on greatly optimizing I/O for large (beyond 100 TiB) data sets. Rest assured that smaller data sets will also greatly benefit from these improvements!

We have analyzed hundreds of petabytes of data sets coming from our customers to flesh out a new data organization and improve our stochastic indexes to reduce read and write amplification (up to 20 times!).

These changes mean that 3.13 is not backward compatible with 3.10 or 3.9 and a data migration is required to upgrade.

We have not just improved persistence, with this release we are introducing three major new features:

  1. Continuous queries - Queries results can now be pushed from the server to the client. This is more than a convenience: polling the server for updates can be extremely inefficient.
  2. Symbol tables - Native support for "infinite" enumerations. Data is stored as integers, but the user is presented with strings. This greatly improves disk usage and query performance. All the symbol table upkeeping is done transparently and the load is distributed over the cluster.
  3. Sub queries - We are working on greatly extending the capabilities of the query language and have introduced sub queries in this release.

This is the first release of this new engine. Expect more from the Quasar team over the year 2022: we aim at nothing less than the impossible.

Get QuasarDB 3.13.0 here!

32-bit and 64-bit ARM binaries for Linux are now included with every release.

Change log

  • Protocol version 45
  • [daemon] Add support for cgroups v2
  • [daemon] Add support for user configuration remote update
  • [daemon] Upgraded to RocksDB 6.14.6
  • [kernel] Add more statistics for improved monitoring
  • [kernel] Add support for continuous queries (e.g. push notifications)
  • [kernel] Add support for priorities on asynchronous queues (disabled by default)
  • [kernel] Greatly improved ingestion rate for asynchronous writes (up to 10X)
  • [kernel] Native support for "symbols" column type (e.g. "infinite" enums)
  • [net] Add support for the new batch API
  • [odbc] Improve ODBC compatibility
  • [persistence] Improve compression, up to double the efficiency!
  • [persistence] Improve tiered storage support
  • [persistence] New indexes and data layout which can deliver up to 20X speed improvement on cold data
  • [query] Add support for DATEPART
  • [query] Add support for TRIM TABLE
  • [query] Improve DELETE speed
  • [query] Improve disk size computation accuracy
  • [query] Support for subqueries

by Edouard Alligand

Quasar showcased on AWS marketplace

We are very proud to announce that Quasar is front and center on the AWS timeseries solutions for financial services page.

What can Quasar do for you?

Quasar can store all your raw tick data (even level 3!), compress it, store that in blob storage for as long as needed, and rebuild the order book on demand, at any point in time, instantly.

Quasar supports natively many exchanges and asset classes, including crypto.

For more information, check out our case studies and don't forget about the free community edition!

by Edouard Alligand

QuasarDB is now Quasar

I am thrilled to announce that QuasarDB has become Quasar.

If you're already a Quasar customer, this will have no impact on the product and service. You may notice that our staff will write you from our new domain: quasar.ai.

This change is to reflect our mission statement: enable AI to create wealth.

Think of AI not as Artificial Intelligence but as Automated Intelligence. AI is that software chain that refines information to present us with actionable choices.

The challenge of AI today isn't the algorithms but the impedance difference between AI and data. As your data volume grows, it becomes exponentially more challenging to leverage it accurately.

At Quasar, we believe that more data isn't a problem: it's an opportunity.

Suppose you're currently doing predictive maintenance, network analysis, or quantitative analysis. What prevents you from feeding more data into your models? How much could you improve?

Quasar is a better answer than data lakes and data warehouses.

Quasar simplifies your data engineering platform by allowing you to capture an unlimited amount of raw timeseries data and extract features on the fly.

Quasar lets your analysts work interactively and forget about data engineering issues altogether.

If your curious about what Quasar can do for your business, our new website has an extensive list of case studies.

These last years have been a crazy ride, and I'm incredibly proud of our team.

I would like to take this opportunity to thank our investors and our customers for placing their faith in us when we were a tiny company with everything to prove. Thank you!

Happy Thanksgiving!

-Edouard, founder and CEO of Quasar

by Edouard Alligand

QuasarDB 3.10.0 Stable Released

We are very pleased to announce the immediate availability of QuasarDB 3.10.0. You can get it here.

The previous stable version, 3.8.0, was released on April 6th, 2020, almost 18 months ago!

3.10 has been battle-tested on multi-petabytes datasets with continuous data ingestions over several months.

3.10 is our most resilient, fastest, and most efficient release ever.

Compared to 3.8, some queries run several orders of magnitude faster, and memory usage is massively reduced. We also greatly increased the write resiliency, for example in the case of a brutal shutdown.

This release does not just improve performance. It also delivers on powerful new querying capabilities, enabling you to aggregate, pivot, and join data in the most flexible way possible using our tagging  logic.

3.10 is a direct branch of 3.9.9 with the improvements listed below. Switching to a stable release means we will now be much more conservative in the future 3.10.x releases, focusing on API stability and bug fixes.

If you want to have the new shiny and powerful features (such as continuous queries, symbol table, etc.), wait for the 3.13 release. We expect to release 3.13 later this month. 

Where is 3.11? Let's just say it's now in a better place.

Important: the default binaries are optimized for Haswell architectures. You will need to use the Core2 builds if your architecture is older than Haswell.

Fixes and improvements

(w.r.t 3.9.9)

  • Protocol version 37
  • [daemon] Add statistics for slow operations
  • [daemon] Change the configuration of sessions and multithreading to improve clarity
  • [daemon] Fix "alias not found" bug after changing a table schema
  • [daemon] Fix data duplication bug
  • [daemon] Fix DELETE ... WHERE DUPLICATE performance bug
  • [daemon] Greatly improve statistics collection performance
  • [daemon] Overall query performance improvement and memory usage
  • [daemon] Properly honor the "disable WAL" RocksDB setting
  • [daemon] Slight compression performance upgrade, both in speed and ratio
  • [persistence] Added more configuration parameters
  • [query] Add LIMIT to SHOW TABLES
  • [query] Add support for slope
  • [query] Add support for sum of absolute changes and sum of absolute values
  • [query] Extend GROUP BY capabilities
  • [query] Fix LOOKUP parsing ambiguity
  • [query] Greatly improve qualified FIND performance
  • [query] Improve FIND() capabilities
  • [query] New, zero copy, ultra-high-performance, batch insert API (the old API is still available but will be deprecated in the next stable release)
  • [shell] Correctly parse timeouts with the units
  • [shell] Do not print footer in CSV mode
  • [shell] Fix column layout print issues

by Edouard Alligand

QuasarDB 3.9.9 Beta released

We are very pleased to announce the immediate availability of QuasarDB 3.9.9.

This release brings significant performance and memory usage improvements.

We have worked hard on the query language as well, and are happy to share with you powerful new features of the ASOF, PIVOT, and LOOKUP functions.

32-bit and 64-bit ARM binaries for Linux are now included with every release.

Get it here!

Known bugs

We are currently working on a hot fix for this issue

  • [daemon] Frequent updates of the same bucket with "push fast" may result in data being duplicated and never deleted.

Fixes and improvements

  • [api] The C API is now versioned through macros
  • [clone] Initial release of the qdb_clone utility
  • [daemon] Add bucket operations statistics
  • [daemon] Can log operations that take an excessive amount of time
  • [daemon] Fix the evict_all speed problem on Windows
  • [daemon] Greatly reduce memory usage on all platforms
  • [daemon] Improve the caching algorithm heuristics for greater performance
  • [daemon] Improve multithreading model for even more scalability
  • [daemon] Log an error when shard sizes become too large
  • [daemon] Remove an invalid log error message that could create confusion
  • [lookup] LOOKUP now supports any kind of string
  • [query] Add support for $table in LOOKUP
  • [query] Add support for the string concatenation operator ||
  • [query] Fix aggregation computation when multiple empty values are present in a column
  • [query] Greatly improve performance of more complex queries, especially when multiple tables are select
  • [query] Properly delete rounded doubles when doing DELETE WHERE DUPLICATE
  • [query] Support arithmetic cast for timestamps
  • [query] Support for AS for LOOKUP
  • [query] Support for non-blob values in LOOKUP
  • [query] Support for the combination of GROUP BY and ASOF
  • [query] Support for the combination of LOOKUP and arithmetic operators
  • [query] Support for the combination of PIVOT and ORDER BY
  • [query] Support for the combination of PIVOT, ASOF, GROUP BY, and PREWHERE
  • [shell] Add prefix_count and suffix_count to shell
  • [shell] The location of the history file is now by default the home directory (and can be configured)

by Edouard Alligand

QuasarDB 3.8.11 Released

We are very pleased to announce the immediate availability of QuasarDB 3.8.11.

This release incorporates several fixes and improvements of the 3.9 branch into the stable 3.8 release branch.

Get it here!

Fixes and improvements

  • [api] Added a new compaction mode that is slower, but uses less disk
  • [api] Added an API call to enable or disable auto-compaction dynamically
  • [api] Trimming and compaction are now two distinct API calls
  • [daemon] Add an operation to nuke a corrupted bucket without shutting down the server remotely
  • [daemon] Correctly support dates before 1970 and beyond 5000
  • [daemon] Greatly improve asynchronous pipelines performance and memory usage
  • [daemon] Improve intra-node recovery on error for ACL permissions lookup
  • [daemon] Log when buckets are too large
  • [daemon] Make the persisted firehose shard size configurable
  • [query] Add support for duplicate rows deletion
  • [query] Fix an invalid timestamp returned for certain first/last queries
  • [query] Fix an issue related to DELETE … WHERE queries
  • [query] Fix GROUP BY results that could be returned as blobs when they are in fact strings
  • [query] Fix invalid results for adjacent_count on large queries
  • [query] Fix several GROUP BY issues related to timestamps
  • [query] Fix several incoherences in the way GROUP BY results were given
  • [query] Fixed an issue where rows could be partially removed after a DELETE WHERE

by Edouard Alligand