Apple Unveils MMAU: A New Benchmark for Evaluating Language Model Agents Across Diverse Domains

Researchers from Apple have recently unveiled the Massive Multitask Agent Understanding (MMAU) benchmark, a new evaluation framework designed to assess the capabilities of large language models (LLMs) as intelligent agents across diverse domains and skills. 

Read the full paper here

MMAU evaluates models on five key capabilities: understanding, reasoning, planning, problem-solving, and self-correction. It spans five domains: tool use, directed acyclic graph question answering, data science and machine learning coding, contest-level programming, and mathematics.

The benchmark comprises 20 carefully designed tasks with over 3,000 distinct prompts, offering a more granular assessment of LLM capabilities compared to existing benchmarks. MMAU aims to provide insights into where model failures stem from by isolating and testing specific skills.

Key findings from evaluating 18 models on MMAU revealed that commercial API-based models like GPT-4 consistently outperformed open-source models across various domains. The models demonstrated varying proficiency levels in different capabilities– problem-solving was more universally achievable, while self-correction posed significant challenges for many models. 

High-quality planning also boosted performance for all models in mathematical tasks. Interestingly, larger models did not always perform better, underscoring the importance of training strategies and model architectures

The researchers emphasise that MMAU is designed to complement, not replace, existing interactive evaluations. They acknowledge limitations in the current scope and call for future work to expand into more domains and refine capability decomposition methods.

By providing a comprehensive and granular evaluation framework, MMAU aims to drive progress in developing more capable and well-rounded AI agents. The datasets and evaluation scripts have been made publicly available to facilitate further research in this area.

Also, recently, Apple introduced LazyLLM, a novel technique aimed at improving the efficiency of large language model (LLM) inference. This approach seeks to accelerate response generation in transformer-based language models while maintaining accuracy.