Energy trading: order book
Context
As the share of renewables grows in the energy mix, electrical supply and demand unpredictability puts massive stress on trading firms. In particular, building the order book becomes much more computationally intensive.
The order book is the list of buys and sells organized by price levels. It provides valuable trading information, which helps traders and improves market transparency.
In a nutshell, what Quasar delivered
- Deployment of complete market and weather data capture, from start to finish, in less than a week
- Instant order book rebuilding at any point in the day while using less cloud resources
- Storage footprint divided by 10
- Future proof solution thanks to built-in scalability
Challenge
Because of renewables, electricity production can suddenly drop or increase, creating sudden demand, which, in turn, puts stress on order book construction.
There can be an increase of at least one order of magnitude in the volume of orders, making the order book two orders of magnitude more computationally expensive.
Energy trading firms typically built an architecture relying on a relational database. That was sufficient to handle previous volumes. This approach does not work anymore regardless of the increase in computational power dedicated to the relational database. Building the order book is more and more expensive as the number of orders grows.
The solution
Quasar is connected directly to the market data provider and captures the data without a third-party tool. Capture is done with virtually no latency.
Quasar uses its unique suite of compression algorithms to compress the data faster than it arrives and use its micro-indexes to reduce future lookup and feature extraction time. This results in significantly reduced disk usage without impacting the queryability of the data.
Each instrument is stored as a timeseries a two dedicated tables (one for orders, one for trades). Weather data is organized by source.
Tables can be tagged at will, enabling a flexing querying mechanism based on those tags. Tags can be queried recursively. For example, obtaining all the instruments of an index or all the weather data of a specific region.
Queries use a SQL interface, dedicated API, or one of the numerous connectors available. For example, traders can load data directly into Tableau using the Quasar ODBC driver. Quants can load timeseries directly into Python Pandas using the Quasar Pandas API.
Quasar has a built-in order book computation module that enables sub-second order book rebuilding at any time of the day, even for deep order books. These order books are then accessible from a web browser using a dedicated dashboard built using the Quasar framework.
Benefits
- Firms can keep up with the market and have linear scalability capabilities thanks to the clustering capabilities of Quasar, increasing their agility and competitiveness
- Compatibility with all standard analytical tool suites for maximum flexibility
- Faster time to a solution: Quasar abstracts away all data engineering complexity
- Outstanding TCO thanks to the combination of data compression and rapid order book rebuilding, saving on disk storage, CPU power, and development time
Energy Trading: Price Forward Curve
Context
As the share of renewables grows in the energy mix, the unpredictability of electrical supply and demand creates the need to compute Price Forward Curves (PFC) more frequently.
The PFC is used to hedge trades and is often a source of alpha for trading firms.
In a nutshell, what Quasar delivered
- Deployment of complete market and weather data capture, from start to finish, in less than a week
- 100X speed up on Price Forward Curve computation
- Storage footprint divided by 10
- Future proof solution thanks to built-in scalability
Challenge
A Price Forward Curve is heavily influenced by supply and demand, especially for commodities like electricity. That's because transporting and storing energy is extremely expensive.
Seasonality is a typical factor of price variation. With the introduction of renewables in the energy grid, the price variation has a much greater unpredictability.
This change requires trading firms to incorporate weather data and recompute the PFC regularly, possibly every minute. That puts massive stress on the system that was not designed for that data pressure.
Energy trading firms typically built a system architecture relying on a relational database that was sufficient to handle previous volumes. The increased workload on the database results in unrealistic budgets or infrastructure size.
The solution
Quasar is connected directly to the market data provider and captures the data without a third-party tool. Capture is done with virtually no latency.
Quasar uses its unique suite of compression algorithms to compress the data faster than it arrives and use its micro-indexes to reduce future lookup and feature extraction time. This results in significantly reduced disk usage without impacting the queryability of the data.
Each instrument is stored as a timeseries a two dedicated tables (one for orders, one for trades). Weather data is organized by source.
Tables can be tagged at will, enabling a flexing querying mechanism based on those tags. Tags can be queried recursively. For example, obtaining all the instruments of an index or all the weather data of a specific region.
Queries use a SQL interface, dedicated API, or one of the numerous connectors available. For example, traders can load data directly into Tableau using the Quasar ODBC driver. Quants can load timeseries directly into Python Pandas using the Quasar Pandas API.
Quasar has built-in capabilities to accelerate the computation of Price Forward Curve. It can transform the data in real-time to significantly simplify the effort needed to get a result.
Benefits
- Firms can recompute the Price Forward Curve as often as needed, increasing alpha
- Firms can keep up with the market and have linear scalability capabilities thanks to the clustering capabilities of Quasar, increasing their agility and competitiveness
- Compatibility with all standard analytical tool suites for maximum flexibility
- Faster time to a solution: Quasar abstracts away all data engineering complexity
- Outstanding TCO thanks to the combination of data compression and built-in aggregation functions, saving on disk storage, CPU power, and development time
Power black out detection
Context
Power blackouts have a major impact on the supply and production chain, damage assets and wasted product, and even cause accidents.
Being able to anticipate blackouts can significantly reduce their impact.
In a nutshell, what Quasar delivered
- Setup of complete electrical waveform data capture in less than a week
- Powerful and flexible analysis as well as short feedback loop enable finding patterns of precursors faster
- Future proof solution thanks to built-in scalability
Challenge
It is possible to anticipate power blackouts by looking for weak signals that prelude significant events. Finding these week signals requires capturing data at the highest resolution possible and performing spectral analysis on the data.
The most significant part of the data consists of the complete tri-phasic electrical waveform data. Resolution can go up to 20 kHz: 20,000 points per second per waveform. It means that a multi-month history is typically at the petabyte scale.
A data lake is a typical solution as storing data in files is simple. Data warehousing solutions are just not cost-effective for these data volumes.
However, a data lake cannot be queryied. This means that once the data is captured, computations (for example, FFT) need to be done ad hoc, adding to latency and cost.
The solution
Quasar can capture the complete waveform data at the highest resolution directly from the MQTT channel.
Each waveform is stored in a separate table, organized in a timeseries. Tagging allows for multiple tables to be queried at the same time.
Feature extraction is done by Quasar and does not require a dedicated program. Quasar has built aggregation functions that can work directly on the data, leveraging the SIMD capabilities of modern processors. For example, Quasar can transform alternate current in direct current in real-time.
Data scientists can get the data exactly how they need it when they need it without going through any additional steps. This dramatically increases research speed.
For monitoring purposes, electrical data can be visualized in real time by leveraging Grafana and the Quasar Grafana connector.
Quasar is used both for live data and historical data, making model deployments smooth.
Benefits
- Significantly accelerate research speed to find weak signals indicating power blackout, preventing millions of dollars of damage and waste
- Building and deploying models is frictionless, leading to a quick feedback loop to improve the accuracy and thus the gains
- Limitless historical capabilities, giving potentially unlimited accuracy
- Outstanding TCO: extracting features does not require writing custom programs and uses less CPU. Up to 20X less storage space is needed compared to a typical data warehousing solution and up to 1,000X faster feature extraction compared to data lakes
Liquidity Provider: pool management
Context
A liquidity provider has many responsibilities; one of them is to help their customers validate that they are compliant with best execution. Another one is to analyze order books to advise their customers, for example, about their order book spread or which trading opportunities they may miss.
Best execution ensures you conform to the auction rules of the trading you are participating in (for example, sell to the highest bidder). The order book is the list of buys and sells organized by price levels.
In a nutshell, what Quasar delivered
- Replaced a constellation of custom tools, spreadsheets, and scripts to deliver a consistent and immediate view of the liquidity pool. What took a day of work is now instant
- Greatly increased the quality of advices given to the users thanks to increasing analytical capabilities
- Future proof solution thanks to built-in scalability
Challenge
The challenge to adequately cater to the needs of your customer for a liquidity provider are multiple.
As order data comes in, rebuilding the order book takes longer and longer since there are more orders to go through. This creates pressure on the data system, especially during peak activity. When doing end-of-day analysis, building the complete order book means going through all the day's orders.
However, building the order book is half the battle since the liquidity provider needs to estimate the order book spread to see if a participant could improve their trading with a tighter book.
Connecting the order book with the relationship matrix (who may trade with whom) helps to advise users regarding what opportunity they would have if they expanded the breadth of actors they are trading width.
In addition to the order book, participants need to ensure they are complying with best execution. For example, ensuring they are selling to the highest bidder. This is done by matching a trade with orders available at that time.
These specialized processes are complicated to do with a regular database management system and often require specialized tools or programs that lack flexibility and generally result in an expensive, complex toolchain.
The solution
Quasar is connected directly to the market data provider and captures the data without a third-party tool. Capture is done with virtually no latency.
The data is stored, compressed, and managed by Quasar. Information is organized in timeseries and is tagged in a way that makes queries easy and convenient. This approach is future-proof and allows you to change your mind "whenever you want" without touching the data itself.
Relationship matrixes are also ingested in the database as they get updated.
Queries use a SQL interface, dedicated API, or one of the numerous connectors available. For example, traders can load data directly into Tableau using the Quasar ODBC driver. Quants can load timeseries directly into Python Pandas using the Quasar Pandas API.
Quasar has a built-in order book computation module that enables sub-second order book rebuilding at any time of the day, even for deep order books. These order books are then accessible from a web browser using a dedicated dashboard built using the Quasar framework.
Quasar can also compute the top and the spread of any order book (for the whole pool or a given actor). From there, evaluating the spread of a specific actor relative market activities is a straightforward task and allows to provide high-value advice to a customer.
Using the dashboard, the liquidity provider can also see if a given actor order book would be matched by another actor they are currently not trading with and thus
Lastly, best execution can be done using advanced joins available in Quasar (ASOF JOINS) that instantly match a trade with an order at a given time.
Benefits
- Liquidity providers can leverage Quasar to give high-value feedback to their users without relying on custom development or third-party tools; increasing value offered to customers
- Firms can keep up with the market and have linear scalability capabilities thanks to the clustering capabilities of Quasar, increasing their agility and competitiveness
- Compatibility with all standard analytical tool suites for maximum flexibility
- Faster time to a solution: Quasar abstracts away all data engineering complexity
- Outstanding TCO thanks to the combination of data compression and built-in aggregation functions, saving on disk storage, CPU power, and development time.
Manufacturing Process Historian
Context
Manufacturing plants use historians, such as GE Historian, to display process data, such as energy consumption, on dashboards or inject them into analytical tools to create models.
These models help to analyze the various operating conditions, for example, find patterns of energy consumption correlated with known incidents of product defects.
In a nutshell, what Quasar delivered
- Completely removed all limits imposed by historians in terms of history depth, data volume, and query complexity
- Virtually unlimited history
- No change in the reporting tool used, Quasar replaces the backend transparently
- Storage footprint divided by 20
- Future proof solution thanks to built-in scalability
Challenge
Historians struggle quickly when the depth of history rises. To build an accurate model, one may need to pull a year of more than data; these represent data volumes historians cannot manage, even for smaller plants.
That means that users need to choose between history depth and sampling accuracy to stay within the constraints of the historian.
That makes it very difficult for data scientists to do a detailed analysis of energy consumption, resulting in inaccurate or incorrect models.
The solution
Quasar is connected to the historian with its high-speed ingestion API to capture all the data as it comes, without any loss of quality or precision.
Quasar will use its unique algorithm to compress the data faster than it arrives and use its micro-indexes to reduce future lookup and feature extraction time. This results in significantly reduced disk usage without impacting the queryability of the data.
Each sensor is stored in a dedicated table storing timeseries data. Tables can be tagged at will, enabling a flexing querying mechanism based on those tags. For example, obtaining all the sensors of an asset or in a specific area of the plant.
Once Quasar collects all the information, it will serve all queries at the highest resolution possible.
Energy data, for example, can be shown at the highest resolution, something it was not possible to do with a historian without compromising dynamic response.
Benefits
- Build accurate models to improve process quality
- Complete data precision to improve the accuracy of the models
- Circumvent historian limitations without replacing them, minimizing IT infrastructure disruption while enabling next-generation AI
- Outstanding TCO: extracting features does not require writing custom programs and uses less CPU. Up to 20X less storage space is needed compared to a typical data warehousing solution and up to 1,000X faster feature extraction compared to data lakes
Manufacturing facility monitoring
Context
In any manufacturing facility, breakdowns are expensive and disruptive. They can result in wasted material and extensive damage to the assets.
It is possible to anticipate these breakdowns, perform root cause analysis, and create a virtuous circle to reduce their occurrence with complete monitoring.
In a nutshell, what Quasar delivered
- From hours to seconds to produce a report
- Virtually unlimited history
- No change in the reporting tool used, Quasar replaces the backend transparently
- Storage footprint divided by 20
- Future proof solution thanks to built-in scalability
Challenge
To provide insights about plant operations, one needs to collect, analyze, and visualize numerous metrics.
These metrics allow having a clear understanding of breakdowns, the meantime to respond and repair the breakdowns, and ultimately anticipate them.
However, a plant with many assets generates tens of millions of records per second. This volume exceeds the capabilities of a typical relational database, a default choice for many of these setups.
As volume increases, updates are longer and longer, sometimes taking hours.
When connecting the data to, for example, Power BI, queries are sluggish, and the data lag prevents the monitoring from being effective.
The solution
Quasar can collect all events as they come and store them efficiently. The collection is direct and does not require any third-party software.
Quasar has no practical limit to how many records per second can be inserted and can thus handle hundreds of millions of updates per second at low latency.
Quasar will use its unique algorithm to compress the data faster than it arrives and use its micro-indexes to reduce future lookup and feature extraction time. This results in significantly reduced disk usage without impacting the queryability of the data.
Each sensor is stored in a dedicated table storing timeseries data. Tables can be tagged at will, enabling a flexing querying mechanism based on those tags. For example, obtaining all the sensors of an asset or in a specific area of the plant.
Microsoft Power BI, or any similar tool, can connect to Quasar directly using ODBC, and queries are instant, delivering sub-minute end-to-end data freshness and enabling operational intelligence.
Benefits
- Manufacturing Facility can achieve operation excellence by having exhaustive access to all the data, resulting in significantly lower operational costs
- Exhaustive, low-latency, monitoring of all assets at the highest precision possible, increasing accuracy and thus potential gains
- Building and deploying models is frictionless, leading to a quick feedback loop to improve the accuracy and thus the gains
- Connect tools such as Microsoft Power BI directly and do data science without writing code, reducing the strain on dedicated teams
- Outstanding TCO: extracting features does not require writing custom programs and uses less CPU. Up to 20X less storage space is needed compared to a typical data warehousing solution and up to 1,000X faster feature extraction compared to data lakes