CockroachDB Launches Advanced Key-Value Store Pebble

Pebble has faster reversal iteration using the skiplist of the memtable's backward links and better concurrency with a quicker commit pipelin
Listen to this story

Pebble is a key-value store developed by CockroachDB that is backed on LevelDB/RocksDB with special focus on performance and internal usage. The file formats from RocksDB are carried over to Pebble, along with a few extensions, including range deletion tombstones, table-level bloom filters, and MANIFEST format updates. Pebble particularly targets the use case and feature set required by CockroachDB and does not incorporate every functionality in RocksDB. 

RocksDB features that are included in Pebble:

  • Block-based tables
  • Checkpoints
  • Indexed batches
  • Iterator options (lower/upper bound, table filter)
  • Level-based compaction
  • Manual compaction
  • Merge operator
  • Prefix bloom filters
  • Prefix iteration
  • Range deletion tombstones
  • Reverse iteration
  • SSTable ingestion
  • Single delete
  • Snapshots
  • Table-level bloom filters

RocksDB features that are not included in Pebble:

  • Backups
  • Column families
  • Delete files in range
  • FIFO compaction style
  • Forward iterator/tailing iterator
  • Hash table format
  • Memtable bloom filter
  • Persistent cache
  • Pin iterator key/value
  • Plain table format
  • SSTable ingest-behind
  • Sub-compactions
  • Transactions
  • Universal compaction style

But Pebble may silently corrupt data or behave incorrectly if used with a RocksDB database that uses a feature that Pebble doesn’t support. In CockroachDB v20.1 (published in May 2020), Pebble was introduced as an alternate storage engine to RocksDB and was utilised in production then. In CockroachDB v20.2, Pebble became the default storage engine (released November 2020). Users of CockroachDB are using Pebble in production on a large scale.

Advantages of Pebble

  • Faster reversal iteration using the skiplist of the memtable’s backward links.
  • Better concurrency is achieved using a quicker commit pipeline.
  • Indexed batch iterations that are seamlessly combined. The batch’s mutations theoretically occupy a different memtable level.
  • Smaller, easier-to-understand code base.

RocksDB Compatibility

Pebble aims for RocksDB 6.2.1 forward compatibility (the latest version of RocksDB used by CockroachDB). Forward compatibility allows Pebble to use a database created by RocksDB. When employing its FormatMostCompatible format, Pebble offers bidirectional compatibility with RocksDB (a DB generated by Pebble may be used by RocksDB). New format major versions are gated behind new functionality that is incompatible with earlier versions. In general, Pebble only offers compatibility with the portion of features and settings that CockroachDB deploys. RocksDB functionality and settings cover too much ground to test and document all the incompatibilities thoroughly. Pebble is based on the incomplete Go version of LevelDB.

Download our Mobile App

Shritama Saha
Shritama is a technology journalist who is keen to learn about AI and analytics play. A graduate in mass communication, she is passionate to explore the influence of data science on fashion, drug development, films, and art.

Subscribe to our newsletter

Join our editors every weekday evening as they steer you through the most significant news of the day.
Your newsletter subscriptions are subject to AIM Privacy Policy and Terms and Conditions.

Our Recent Stories

Our Upcoming Events

3 Ways to Join our Community

Telegram group

Discover special offers, top stories, upcoming events, and more.

Discord Server

Stay Connected with a larger ecosystem of data science and ML Professionals

Subscribe to our Daily newsletter

Get our daily awesome stories & videos in your inbox

6 IDEs Built for Rust

Rust IDEs aid efficient code development by offering features like code completion, syntax highlighting, linting, debugging tools, and code refactoring