Meet Meerkat, A New Data Library To Wrangle Complex ML Datasets

Recently, researchers at Stanford University launched a new data library called Meerkat for working with complex machine learning datasets. The source code of the project is available on GitHub

Data is the oxygen for machine learning. From training and validation data to future predictions, embeddings and metadata, it drives all parts of the machine learning development process. However, organising and managing data is challenging.

To that end, Stanford researchers have proposed a new Python library to help researchers and ML practitioners wrangle data. Data wrangling is a process of cleaning and unifying messy and complex datasets for easy access and analysis. 


Sign up for your weekly dose of what's up in emerging technology.

How does Meerkat work? 

In a Notion Press blog, ‘Meerkat: Datapanels for machine learning,’ Stanford researchers Sabri Eyuboglu, Arjun Desai and Karan Goel talked about a few areas where Meerkat could solve the data complexity in the machine learning lifecycle.

  • Dataset manipulation techniques like slicing, shaping and transforming datasets have become an increasingly important part of the development process. As the quality of machine learning models and evaluations are primarily products of the data, more time goes into tuning datasets than tuning models. 
  • Model evaluation is emerging as a new bottleneck when building high-performing ML systems. For instance, models have been commoditised to the extent that resources like HuggingFace’s Model Hub can give you a model for text, speech or vision in seconds. But, they are hard to get right, and their failure modes can be opaque. 
  • Multi-modal datasets that combine multiple, complex data types are becoming more prevalent. For example, OpenAI’s CLIP combines natural language with images. 

Meerkat provides the DataPanel abstraction. The DataPanel facilitates interactive dataset manipulation, where it can house diverse data modalities and lets you evaluate models carefully with Robustness Gym. “We built DataPanels like DataFrames because they are naturally interactive and work seamlessly across development contexts: Jupiter Notebooks, Python scripts, and Streamlit,” the researchers said.

The goal is to make Meerkat DataPanel an interactive data substrate for modern machine learning across the machine learning lifecycle. 

Besides Robustness Gym, Meerkat can also be integrated into other popular benchmark datasets and works well with existing libraries and tools like WILDS, Huggingface Datasets, DOSMA, Streamlit.

What makes Meerkat different?

The data structures typically fall into two categories: those supporting complex data types and multiple modalities (PyTorch Dataset, Tensorflow Dataset), and those that support manipulation and interaction (Pandas DataFrame). “With the Meerkat DataPanel, we support all of these desiderata in one data structure,” said the researchers.


  • Meerkat can store complex data types (images, graphs, videos and time series) 
  • Supports datasets that are larger than RAM (Kinetics, MIMIC-CXR, ImageNet) with efficient I/O under-the-hood  
  • Supports multimodal datasets 
  • Supports data creation and manipulation 
  • Supports data selection 
  • Support inspection in interactive environments

Comparing Meerkat with other machine learning data structures (Source: Notion Press) 

The experiment

The researchers ran an experiment to detect pneumothorax (a collapsed lung) in chest X-rays. For developing a model for this task, the researchers encountered various types of data — X-ray images to structured metadata to embeddings extracted from a trained model. 

Here, Meerkat’s DataPanel (a columnar data structure) could house all these data types under one roof. “Keeping them together enables quicker model iteration, fine-grained error analysis, and easier data exploration and inspection,” said the Stanford researchers. The codes used for running this experiment are available here

Meerkat addressed the desiderata by facilitating the inspection and manipulation of datasets that combine multiple complex data types. Meerkat provides high-level data abstractions because its data structures are written in Python and have few low-level optimisations, unlike Pandas, NumpY or Apache Arrow. 

“This does not mean that the Meerkat DataPanel is slow: each column type is as fast as the data structure it is built upon,” said the researchers.

More Great AIM Stories

Amit Raja Naik
Amit Raja Naik is a seasoned technology journalist who covers everything from data science to machine learning and artificial intelligence for Analytics India Magazine, where he examines the trends, challenges, ideas, and transformations across the industry.

Our Upcoming Events

Conference, in-person (Bangalore)
Machine Learning Developers Summit (MLDS) 2023
19-20th Jan, 2023

Conference, in-person (Bangalore)
Rising 2023 | Women in Tech Conference
16-17th Mar, 2023

Conference, in-person (Bangalore)
Data Engineering Summit (DES) 2023
27-28th Apr, 2023

Conference, in-person (Bangalore)
MachineCon 2023
23rd Jun, 2023

Conference, in-person (Bangalore)
Cypher 2023
20-22nd Sep, 2023

3 Ways to Join our Community

Whatsapp group

Discover special offers, top stories, upcoming events, and more.

Discord Server

Stay Connected with a larger ecosystem of data science and ML Professionals

Subscribe to our newsletter

Get the latest updates from AIM