Article complete

Karya Partners with Microsoft Research for Extensive Evaluation of Indian LLMs

The project involved over 90,000 human evaluations of 30 models in 10 Indian languages, completed in under three weeks.

A colorful graphic featuring a tractor and figures amidst vibrant shapes, symbolizing innovation in technology and agricultur

Key takeaways

  • Karya and Microsoft Research conducted over 90,000 evaluations of LLMs in 10 Indian languages.
  • The evaluation addressed linguistic diversity and cultural nuances in model performance.
  • Results include detailed leaderboards comparing model performance on specific tasks.
  • Karya's project is one of the largest multilingual human evaluations of LLMs to date.
  • The evaluation assessed models across various conversation benchmarks for perception and reasoning.

Takeaways drafted with AI from the article text.

The evaluation addressed challenges such as linguistic diversity, benchmark contamination, and the incorporation of local cultural nuances. Karya engaged a broad community of workers representing the average Indian population to ensure a comprehensive assessment.

The evaluation results, including detailed leaderboards on model performance, are available in the published paper here. Karya’s services encompass various benchmarks, including linguistic acceptability, cultural sensitivity, and reasoning.

A presentation of Karya’s LLM Evaluation Services can be viewed here. The project is noted for its complexity, with Karya’s team successfully delivering accurate results.

Multi-Turn Evaluation: Karya workers assessed models using various multi-turn conversation benchmarks, including tasks such as recollection, expansion, refinement, and follow-ups. The evaluation measured model performance in perception, reasoning, and creativity.

Model Comparison: The evaluation also involved comparing multiple models against a common benchmark to determine their relative performance. This analysis aimed to identify the best-performing model for specific tasks.

The execution of this project was led by Jeevitha and included Bipin, Victor, Sakshi, Deepsikha, Swarup, Rizwan, Praveen, Hari, Iqbal, Rupal, Neha, Anand, Monali, Aishwarya, and Anushree. Karya aims to expand such high-complexity digital tasks, potentially increasing earning opportunities for workers and advancing pathways out of poverty.

Founded by Manu Chopra and Vivek Seshadri in 2021, Karya provides a variety of AI services, including generating high quality training data, multimodal data labelling and performing culturally sensitive language model evaluations. By offering industry-leading wages and investing in worker welfare, Karya is pioneering an equitable model in the AI data industry.

Published

Spotted an error? Tell our editors. Read our editorial standards and corrections.

About the author

Tech Journalist

Siddharth is a media graduate who loves to explore tech through journalism and putting forward ideas worth pondering about in the era of artificial intelligence.

All stories
Belamy · Daily

The day’s AI news, before your first meeting.