Recently, Spotify open-sourced an AI framework at the 2020 International Society for Music Information Retrieval Conference, known as Klio. Klio is an ecosystem which allows developers to process audio files or any other binary files at any scale. 

Built by Spotify, Klio runs large-scale audio intelligence systems at the digital music platform and is used by teams of engineers and audio researchers to help develop and deploy next-generation audio algorithms.

Behind Klio

Klio is built upon Apache Beam, and the jobs are opinionated data pipelines in Python. Tuned for audio and binary file processing, the goal of this framework is to create smarter data pipelines for audio.

Spotify built Klio as a standardised framework for running audio analysis allowing teams to share a common useful foundation instead of duplicating each others’ work. Klio reduces the operational and backend engineering load on these teams, allowing them to focus on doing the processing, machine learning and other algorithmic work.

Klio is meant for processing media and can be used on cloud infrastructure, or locally on one’s computer. Klio focuses Beam on analysing, manipulating, and transforming large binary media (e.g. images, audio, video) where the content in its native form can’t really fit or be analysed in a database in any meaningful way.

The Klio ecosystem is made up of multiple, separate Python packages. Besides the internal packages, there are some packages that are user-facing, which are klio-cli, klio and klio-audio.

Klio Pipelines

A Klio pipeline uses the reference identifiers to audio files from event inputs. It downloads those files onto worker machines, runs processing algorithms, such as librosa, ffmpeg, trained ML models, anything over these files and then saves the resulting output to the data store of choice.

The processing algorithms can be ML-based or otherwise can be wrapped in Python. According to its developers, Klio jobs can scale up to process an entire corpus of media, or down to a single item for fast iteration. 

How Spotify Uses Klio

Spotify uses Klio to run its audio intelligence platforms, which powers the audio features, aspects of the audio ad platform as well as the audio research workflow. Further, the developers at the digital music platform use Klio for tasks like running Tensorflow models over audio files, creating new audio from existing audio in areas like audio fingerprinting and ad generation, and saving analysed data to databases.

Why Use Klio

This framework helps organisations to build media processing systems that share tooling and infrastructure between production systems and research teams. It provides an architecture that encourages reusable jobs and shared outputs, lowering maintenance and recomputation costs. Klio also enables continuous event-driven processing of rapidly growing catalogues of content.

The framework also helps organisations by enabling several benefits, which are-

• Researchers to benefit from working with the same infrastructure as engineers responsible for production systems as well as processing power for complete catalogues of content
• Engineers have a simpler framework to directly productionise large media processing jobs produced by researchers when they’re already using the same frameworks for development
• Organisations to benefit from these pipelines immediately as they add new content, allowing new content to be processed immediately on ingestion through a streaming pipeline and for backfills to be handled as a batch job with the same code.

Wrapping Up

Before you start working with Klio, it is important to understand the basics of Google Pub/Sub and Apache Beam’s Python software development kit, how they work, how to write Beam pipelines, steps to launch a Beam job on data flow and other such.

According to the developers, this framework is currently in rapid development, which means that the APIs and the features will evolve. It is recommended to upgrade the Klio installation as new releases become available.