Article complete

PayPal Releases Its Open-Source Spark Indexing Library “Dione”

Dione would enable faster interaction with Hadoop Data and can be used to solve challenges using the index as a “shadow” table of the original data.

A PayPal-branded screen is set up outdoors in a landscaped area with trees and seating.

Key takeaways

  • PayPal's Dione is an open-source Spark Indexing library designed for faster Hadoop Data interaction.
  • Dione addresses traditional challenges by using an index as a 'shadow' table, avoiding data duplication.
  • The library supports multiple indices and operates solely on Spark, Hive, and HDFS without external services.
  • Dione's index is structured in a special Avro B-Tree format, allowing for quick ad-hoc data retrieval.

Takeaways drafted with AI from the article text.

Dione can be used to solve such challenges using the index as a “shadow” table of the original data. It contains only the key columns and pointers to the data and is saved in a special format inspired by Avro and bucketing. Based on this index, the library provides APIs to join, query, and fetch back the original data in the required SLAs.

A few advantages of using Dione are:

  • Relies only on Spark, Hive and HDFS. No external services.
  • Semi-managed — does not modify, duplicate or move the original data.
  • Supports multiple indices on the same data.
  • The index is exposed as a standard Hive table.
  • The special Avro B-Tree format supports ad-hoc single-row fetch in an SLA of seconds.
Image: Paypal Engineering
Image: Paypal Engineering

Dione solves two main challenges that are currently faced using traditional methods: 

  • Given a pointer (a line in the index), how to fetch the data quickly?
  • How to store the index table to meet the required SLAs?

The library makes use of its two main architectural components — Indexer and AvroBtreeFormat. The final index solution is composed of the inter-relations between these components, although in principle, each of them can stand on its own.

The Indexer’s main goal is to solve the multi-row load task. The single-row fetch task as using Spark to scan through all the index tables does not meet the required SLA. For that, the index’s storage format is used. Inspired by Avro’s SortedKeyValueFile, bucketing and traditional databases indexing systems, the development team decided to create a “new” file format — Avro B-Tree.

Both the Indexer and Avro B-Tree File Format libraries are independent packages and rely only on HDFS. Users can save any table in the Avro B-Tree format so it will be accessible for both batch analytics with Spark and single-row fetches. For a simplified Spark user experience, a high-level Spark API has been added for creating and using an index. The API is available in Scala and Python.

PayPal has open-sourced this library to share this functionality with the community and get feedback.

Published

Spotted an error? Tell our editors. Read our editorial standards and corrections.