Recently, researchers at Stanford University launched a new data library called Meerkat for working with complex machine learning datasets. The source code of the project is available on GitHub. 

Data is the oxygen for machine learning. From training and validation data to future predictions, embeddings and metadata, it drives all parts of the machine learning development process. However, organising and managing data is challenging.

To that end, Stanford researchers have proposed a new Python library to help researchers and ML practitioners wrangle data. Data wrangling is a process of cleaning and unifying messy and complex datasets for easy access and analysis. 

How does Meerkat work? 

In a Notion Press blog, ‘Meerkat: Datapanels for machine learning,’ Stanford researchers Sabri Eyuboglu, Arjun Desai and Karan Goel talked about a few areas where Meerkat could solve the data complexity in the machine learning lifecycle.

• Dataset manipulation techniques like slicing, shaping and transforming datasets have become an increasingly important part of the development process. As the quality of machine learning models and evaluations are primarily products of the data, more time goes into tuning datasets than tuning models. 
• Model evaluation is emerging as a new bottleneck when building high-performing ML systems. For instance, models have been commoditised to the extent that resources like HuggingFace’s Model Hub can give you a model for text, speech or vision in seconds. But, they are hard to get right, and their failure modes can be opaque. 
• Multi-modal datasets that combine multiple, complex data types are becoming more prevalent. For example, OpenAI’s CLIP combines natural language with images. 

Meerkat provides the DataPanel abstraction. The DataPanel facilitates interactive dataset manipulation, where it can house diverse data modalities and lets you evaluate models carefully with Robustness Gym. “We built DataPanels like DataFrames because they are naturally interactive and work seamlessly across development contexts: Jupiter Notebooks, Python scripts, and Streamlit,” the researchers said.

The goal is to make Meerkat DataPanel an interactive data substrate for modern machine learning across the machine learning lifecycle. 

Besides Robustness Gym, Meerkat can also be integrated into other popular benchmark datasets and works well with existing libraries and tools like WILDS, Huggingface Datasets, DOSMA, Streamlit.

What makes Meerkat different?

The data structures typically fall into two categories: those supporting complex data types and multiple modalities (PyTorch Dataset, Tensorflow Dataset), and those that support manipulation and interaction (Pandas DataFrame). “With the Meerkat DataPanel, we support all of these desiderata in one data structure,” said the researchers.

Advantages: 

• Meerkat can store complex data types (images, graphs, videos and time series) 
• Supports datasets that are larger than RAM (Kinetics, MIMIC-CXR, ImageNet) with efficient I/O under-the-hood  
• Supports multimodal datasets 
• Supports data creation and manipulation 
• Supports data selection 
• Support inspection in interactive environments

Comparing Meerkat with other machine learning data structures (Source: Notion Press) 

The experiment

The researchers ran an experiment to detect pneumothorax (a collapsed lung) in chest X-rays. For developing a model for this task, the researchers encountered various types of data — X-ray images to structured metadata to embeddings extracted from a trained model. 

Here, Meerkat’s DataPanel (a columnar data structure) could house all these data types under one roof. “Keeping them together enables quicker model iteration, fine-grained error analysis, and easier data exploration and inspection,” said the Stanford researchers. The codes used for running this experiment are available here. 

Meerkat addressed the desiderata by facilitating the inspection and manipulation of datasets that combine multiple complex data types. Meerkat provides high-level data abstractions because its data structures are written in Python and have few low-level optimisations, unlike Pandas, NumpY or Apache Arrow. 

“This does not mean that the Meerkat DataPanel is slow: each column type is as fast as the data structure it is built upon,” said the researchers.