Article complete

Hugging Face Introduces Cosmopedia, the Largest Open Synthetic Dataset

The dataset consists of over 30 million samples and f 25 billion tokens, generated by Mixtral.

Hugging Face Introduces Cosmopedia, the Largest Open Synthetic Dataset

Key takeaways

  • Hugging Face launches Cosmopedia, the largest open synthetic dataset with over 30 million samples.
  • Dataset features 25 billion tokens across various content types, enhancing global knowledge compilation.
  • Cosmopedia serves as a foundational resource for synthetic data research, inspired by Phi1.5's work.
  • Structured into eight splits, it includes diverse sources like Stanford, WikiHow, and KhanAcademy.
  • Access dataset easily with provided code snippets for specific splits, including a smaller subset.

Takeaways drafted with AI from the article text.

The dataset aims to compile global knowledge by mapping information from web datasets like RefinedWeb and RedPajama. It features essential information, including prompts, synthetic content, seed data sources, token lengths, text formats (e.g., textbook, blog post), and target audiences. The comprehensive breakdown of splits, distributions, and creation methodology is presented, offering researchers insights into the dataset's structure and potential applications.

Inspired by Phi1.5's work, this initial version of Cosmopedia provides a foundation for research in the synthetic data domain. It serves as a comprehensive resource for diverse topics, emphasizing its potential for further enhancement in subsequent iterations.

The dataset is structured into eight splits, each derived from distinct seed samples. These splits include web_samples_v1 and web_samples_v2, constituting approximately 75% of the dataset, sourced from an internal web dataset akin to RefinedWeb. 

The Stanford split utilizes scraped course outlines from stanford.edu, while the stories split incorporates generated narratives from UltraChat and OpenHermes2.5. Additionally, WikiHow, OpenStax, KhanAcademy, and automathtext splits involve prompts related to their respective sources.

To facilitate dataset access, users can employ the provided code snippet to load specific splits. A smaller subset, Cosmopedia-100k, is also available for those seeking a reduced dataset. Furthermore, a larger model, Cosmo-1B, has been trained on Cosmopedia, demonstrating scalability and versatility.

The dataset creation process involves a topic clustering method for web samples, refining prompts iteratively, and addressing contamination issues. The objective is to maximize diversity by tailoring prompt styles and audiences, significantly reducing duplicate content.

Published

Corrections: AIM corrects errors of fact and notes the change on the story. Spotted one? Tell the editors. Read our editorial standards.