Skip to main content
Version: Nightly

Data Persistence and Indexing

Like other LSM-tree storage engines, GreptimeDB persists data from memtables to durable storage such as a local filesystem or object storage. It uses Apache Parquet as the persistent file format.

SST File Format​

Parquet is an open source columnar format that provides fast data querying and has already been adopted by many projects, such as Delta Lake.

Parquet organizes data as row groups, column chunks, and pages. A row group contains one column chunk for each column, and each column chunk contains one or more pages. Pages are the units of encoding and compression; column chunks are the I/O units for reading selected columns.

First, clustering data by column makes file scanning more efficient, especially when only a few columns are queried, which is very common in analytical systems.

Second, values within a column tend to be similar, which improves compression with techniques such as dictionary encoding and run-length encoding (RLE).

The following diagram from the Apache Parquet specification also shows the physical file layout: column chunks are stored by row group, while file metadata and its length are written in the footer.

Apache Parquet file layout

Source: Apache Parquet FileLayout.gif. Copyright 2014 The Apache Software Foundation, licensed under the Apache License 2.0.

Data Persistence​

GreptimeDB provides a configuration item region_engine.mito.global_write_buffer_size, which is flush threshold of the total memory usage for all MemTables.

When the size of data buffered in MemTables reaches that threshold, GreptimeDB will pick MemTables and flush them to SST files.

Indexing Data in SST Files​

Parquet stores row-group column statistics such as minimum, maximum, and null count in each column chunk's metadata. Page metadata and optional column indexes can provide finer-grained statistics.

A name predicate uses Parquet column statistics to skip one row group while retaining another as a read candidate.

For example, a query filtering for name = Emily can skip row group 0 because the maximum name value is Charlie. This avoids reading that row group.

Index Files​

When an SST has one or more configured index outputs, GreptimeDB writes them to a Puffin file associated with that SST. An SST with no applicable index does not need a Puffin file.

Puffin provides a container for index blobs and their metadata, allowing different index structures to share one file.

Puffin

GreptimeDB stores several index structures in the Puffin file as Blobs, including the inverted index, the skipping index (backed by a bloom filter), and the full-text index. The inverted index was the first one supported and is described in detail below.

Inverted Index​

In version 0.7, GreptimeDB introduced the inverted index to accelerate queries.

The inverted index is a common index structure used for full-text searches, mapping each word in the document to a list of documents containing that word. GreptimeDB applies this search-engine technique to indexes over time-series data.

Search engines and time series databases operate in separate domains, yet the principle behind the applied inverted index technology is similar. This similarity requires some conceptual adjustments:

  1. Term: In GreptimeDB, it refers to the column value of the time series.
  2. Document: In GreptimeDB, it refers to the data segment containing multiple time series.

The inverted index enables GreptimeDB to skip data segments that do not meet query conditions, thus improving scanning efficiency.

Inverted index searching

The query above uses the inverted index to identify data segments where job equals apiserver, handler matches .*users, and status matches 4... It scans only those segments before applying the remaining filters.

Inverted Index Format​

An inverted-index blob contains one index per column followed by footer metadata; each column index contains a null bitmap, posting bitmaps, and an FST.

GreptimeDB builds inverted indexes by column. Each column index contains a null bitmap, multiple posting bitmaps, and an FST. The blob footer records the offsets, sizes, and metadata needed to locate and decode the column indexes.

The FST (Finite State Transducer) enables GreptimeDB to store mappings from column values to Bitmap positions in a compact format and provides excellent search performance and supports complex search capabilities (such as regular expression matching). The Bitmaps maintain a list of data segment IDs, with each bit representing a data segment.

Index Data Segments​

GreptimeDB divides an SST file into multiple indexed data segments, with each segment housing an equal number of rows. This segmentation is designed to optimize query performance by scanning only the data segments that match the query conditions.

For example, if a data segment contains 1024 rows and the list of data segments identified through the inverted index for the query conditions is [0, 2], then only the 0th and 2nd data segments in the SST file—from rows 0 to 1023 and 2048 to 3071, respectively—need to be scanned.

The number of rows in a data segment is controlled by the engine option index.inverted_index.segment_row_count, which defaults to 1024. A smaller value means more precise indexing and often results in better query performance but increases the cost of index storage. By adjusting this option, a balance can be struck between storage costs and query performance.

Unified Data Access Layer: OpenDAL​

GreptimeDB uses OpenDAL to provide a common access layer for local filesystems and object stores. Changing the configured storage backend does not migrate existing data.