We are pleased to announce the immediate availability of QuasarDB 3.14.2, an update to version 3.14.1. This version is fully backward compatible with 3.14.1.
One year in the making, this release introduces significant improvements in features, performance, and reliability.
This release introduces our new OLAP interface that brings 100% SQL capability, effectively turning your Quasar instance into a fully functional data warehouse. The “Quasar SQL” and high-performance APIs are still available. This OLAP interface operates as a standalone node that connects to your Quasar cluster, enabling parallelization and containerization.
The kernel now features a more robust, scan-resistant cache heuristic, and you can indicate when writing to the database whether you think the data should be kept in cache or not. This is particularly beneficial when importing historical data, as it typically doesn’t need to be cached.
Performance-wise, this release brings improvements across the board, particularly in the parallelization of SELECT queries and network communication efficiency.
The core APIs and languages have also been thoroughly extended: bulk reader API, performance improvements, quantile functions, more flexibility (can create a table based on the result of a select, can insert the result of a select into a table…), the list is long, and you’re invited to look at the complete change log to learn more.
We’ve changed the way you import data into Quasar. Previously, you had to use an external tool called “QDB Import” (informally known as railgun); now, importing is done from the shell or a query using the “IMPORT” syntax. This release supports CSV/TSV files to a high degree of customization, and we are already adding native Parquet for the next release.
The monitoring capabilities are also significantly enhanced, featuring new JSON logging, robust and configurable user properties for improved tracing, and enhanced statistics, including compaction statistics to help fine-tune storage.
Lastly, this release is the result of intense and painstaking work on reliability. We’ve chased down rare but extremely annoying bugs around persistence (e.g., when the persistence layer fails, ensure Quasar fails gracefully), added options to mitigate S3 failures (yes, they happen), enhanced the REPAIR function to recover data in many, many scenarios, and fixed rare reliability bugs that could cause crashes when querying corrupted data.
Change log since 3.14.1
Protocol version 48
- [api] Add support for get metadata by ID
- [api] Batch operation to detach tags
- [api] Batch writers can now influence the server-side caching policy
- [api] Explicit support for timezone configuration in the API
- [api] New “lazy” batch table creation mode: create missing tables on insertion
- [api] New API to validate queries and get the expected schema of the result
- [api] New error code for asynchronous pipelines being full, giving better feedback to users
- [api] New high-performance bulk reader, for streaming large amounts of raw data from a Quasar server
- [api] New high-performance random bulk reader API, for loading raw data chunks that fit in RAM from a Quasar server
- [api] New qdb_set_tags_fast API
- [api] Properly truncate every bucket when doing INSERT TRUNCATE
- [compatibility] Minimum glibc version is now 2.26 (was 2.17)
- [general] Applications will no longer leave stale temporary files
- [kernel] Aggregated tables internal states can be persisted for extra reliability
- [kernel] Automatically cancel background trimming and compaction when shutting down the server
- [kernel] Better, scan-resistant, caching heuristic based on LRU-2
- [kernel] Enriched query logging
- [kernel] Fix a bug in the micro-index that could result in severely degraded performance
- [kernel] Fix a deduplication issue with asynchronous inserts
- [kernel] Fix a potential server-side crash with GROUP BY + WHERE after an ALTER
- [kernel] Fix an issue where Quasar could not restart after a system failure
- [kernel] Fix crash on deduplicated insertion on a column that has been added after table creation
- [kernel] Greatly improve the performance of first(x) and last(x)
- [kernel] Greatly improve user management, enabling remote listing and modifications
- [kernel] Improve cache metrics
- [kernel] Improve memory usage for string columns
- [kernel] Improve performance of PARTITION BY in AGGREGATED TABLE
- [kernel] Improve support for PARTITION BY in AGGREGATED TABLE
- [kernel] More asynchronous pipeline statistics for better monitoring and planning
- [kernel] More write statistics for better monitoring and planning
- [kernel] Server side option to disable micro-indexes for testing purposes
- [kernel] Support for infinite windows in aggregated tables
- [kernel] The aggregation engine is more resistant to corrupted or altered data
- [kernel] UPDATE and DELETE storage modifications are now much more resistant to hardware and system failures
- [logging] Support for JSON-format log output
- [logging] Support for user-provided properties in logging
- [odbc] Full support for SAS Viya caslib
- [odbc] Greatly improved SAS Viya support
- [orderbook] Fix rare “empty result” bug
- [orderbook] General performance improvements
- [orderbook] Support for symbol tables in ORDERBOOK functions
- [persistence] Added more persistence statistics
- [persistence] Configurable, paranoid S3 persistence mode: file upload verification, S3-side checksum, local checksum
- [persistence] Ensure files are properly truncated on disk to avoid excessive disk usage
- [persistence] Extended local validation before startup
- [persistence] Several configuration settings are now set per column family, and default values have been updated
- [persistence] Updated compaction default settings for more throughput
- [protocol] Changed on-the-wire encryption from AES 256 GCM to AEGIS 256
- [protocol] New on-the-wire data compression for improved network usage
- [query] Add fallback parameter to LEAD and LAG
- [query] Add support for BETWEEN in WHERE clauses
- [query] Add support for CREATE TABLE … AS SELECT
- [query] Add support for INSERT INTO … SELECT
- [query] Add support to specify time alignment of GROUP BY queries
- [query] Allow aliases for selected tables
- [query] count($timestamp) fix to work with restrict to
- [query] Disallow negative steps in ASOF RANGE
- [query] Ensure last(x) returns the last row of duplicated timestamps.
- [query] Every API will now validate that strings are valid UTF-8 sequences
- [query] Extended PIVOT support
- [query] Fix a bug in TWAP when used with GROUP BY that would return an internal timestamp list
- [query] Fix bug that made WHERE IN interpret an INT64 as a timestamp
- [query] Fix error with ASOF JOIN RANGE when aggregating on a string column
- [query] Fix performance issue with ASOF RANGE scanning an entire table
- [query] HAVING is now properly evaluated before OVER
- [query] Insertions with invalid timestamps will now be rejected
- [query] Massive performance improvement for DISTINCT COUNT
- [query] New multithreaded model for the SELECT engine, with drastic performance improvements across the board
- [query] Query engine is now timezone aware
- [query] SHOW TABLE correctly displays the TTL of a table
- [query] Support for adding tags in a table at creation
- [query] Support for approximate median
- [query] Support for DELETE without an explicit RANGE
- [query] Support for exact median
- [query] Support for explicit timezone value in queries with new AT TIME ZONE construct
- [query] Support for IF/ELSE
- [query] Support for LIKE
- [query] Support for PostgreSQL-style timestamp casts
- [query] Support for quantile (Histogram, Q-Digest, and T-Digest)
- [query] Support high-performance CSV/TSV files loading with IMPORT command
- [query] Support last(x).$timestamp in aggregated tables
- [query] Time grouping now starts from epoch
- [shell] Display license information on start
- [shell] Support for manual trimming
- [shell] Support for multi-line queries
