Introduction to the WARP storage engine.


WarpSQL currently ships a beta release of the WARP storage engine. The WARP storage engine is based on Fastbit, which is a column store with word aligned hybrid compressed bitmap indexes. WARP storage engine is an acronym for word-aligned relational partitioned storage engine. Fastbit is a mature database technology which has been used in areas such as particle physics and grid computing.

Columnar storage
The WARP storage engine is a columnar engine. This means that it stores each column separately on disk, rather than storing the entire row together on disk. This has significant IO advantages, especially for wide tables, because columns that are not included in the query are not read from disk. This can result in significant performance improvements compared to a row store.

DML operations are supported
Many columnar engines, such as ClickHouse, are append only. The WARP engine supports INSERT, UPDATE, and DELETE operations, but they may be more expensive than comparable operations on a row store.

All MySQL data types except GEOMETRY are supported
Unlike many column stores, all MySQL data types are supported, including long text and long blob columns.

Automatic Indexing
The WARP storage engine automatically indexes tables to match the queries executed against WARP storage engine backed tables.

The WARP storage engine uses engine condition pushdown to efficiently evaluate complex expressions in the storage engine that MySQL can not normally optimize. WARP supports more than 15 types of bitmap indexes, and the desired index characteristics can be selecting using a column comment.

Word-aligned hybrid compressed bitmap indexes
WARP features compressed bitmap indexes. Bitmap indexes have certain advantages and disadvantages compared to traditional sorted indexes such as B*Tree indexes. Bitmap indexes can be used for complex query conditions which B*Tree and hash indexes can not answer. Unlike traditional storage engine, WARP is able to utilize more than one index to answer a query.

MySQL 8 supports hash joins.
Make sure you adjust join_buffer_size appropriately to improve performance of the hash join.

ACID Compliant with MVCC capabilities
WARP beta 2 supports MVCC transactions, with non-locking REPEATABLE-READ and READ-COMMITTED transaction isolation levels. Statements are atomic and the database is crash safe.

Download the WARP storage engine from Github
https://github.com/greenlion/warp

Advertisement

Published by greenlion

I am the author of WarpSQL. I received a MySQL community award in 2015 for my work on Swanhart-Tools, which is the basis for many portions of WarpSQL.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: