Market data mainly consists of orders and trades.
The order book is necessary to evaluate how orders may be filled and is a powerful tool to identify actors’ intentions.
On top of that, reconciling orders with trades is a legal requirement: actors need to conform to best execution.
The order book is the list of buys and sells organized by price levels. It provides valuable trading information, which helps traders and improves market transparency.
Nasdaq enables subscribers to track the status of each order from the first time it is entered until it is either executed or canceled via the ITCH protocol. To properly participate, one thus needs software capable of decoding the protocol.
Once the data is decoded, one needs to store that data in a form that can be later processed to rebuild the order book, which means processing every order since the opening of the markets.
Lastly, it is necessary to match orders with executed trade to ensure that best execution has been respected. This means parsing every order and finding the best match at the time of the trade.
Nasdaq data volumes are in the region of 2 TB per day.
These volumes make it impossible to rely on standard data warehousing technologies.
Typically, firms will store data in files processed by custom-made software for historical data analysis and keep one or two days of data in a dedicated operational database.
Quasar has an ITCH data importer that can decode and capture the data at very high speed without any intermediary conversion. This ensures data capture is as fast and efficient as possible.
Data is compressed using Quasar optimized compression algorithm, significantly reducing the disk footprint of the history.
Market data is organized in the following way: two (2) tables representing timeseries data per security, one (1) for the trades, one (1) for the orders. Using tagging, it’s possible to group different tables for any query. This organization that is made possible thanks to Quasar tagging capability increases compression efficiency and significantly boosts querying speed.