Meta Unveils Details of its Latest AI Chip, MTIA

In a race amongst its competitors also building custom chips for AI models, Meta’s AI chip, the Meta Training and Inference Accelerator (MTIA) is poised to revolutionise the training of ranking and recommendation models,

Key highlights of the new MTIA chip include a significant boost in on-chip memory capacity, with 256MB compared to its predecessor's 128MB, and a higher clock speed of 1.3GHz compared to 800GHz. Early tests conducted by Meta have shown a remarkable threefold performance improvement across multiple models evaluated.

Dubbed internally as "Artemis," the MTIA v2 project underscores Meta's commitment to advancing AI capabilities, extending beyond inference to encompass training tasks. This move aligns with a broader trend in the industry, with major players like Google, Microsoft, and Amazon investing in custom AI chips to meet the escalating demand for compute power.

This week, Google announced the general availability of its fifth-generation custom chip, TPU v5p, for training AI models to Google Cloud users. Additionally, Google introduced its inaugural chip designed specifically for model execution, named Axion. Amazon has developed multiple families of custom AI chips, while Microsoft entered the field last year with the Azure Maia AI Accelerator and the Azure Cobalt 100 CPU.

Originally slated for release in 2025, Meta surprised the industry by announcing that both MTIA versions, including the upcoming iteration, are already in production. While MTIA currently focuses on training ranking and recommendation algorithms, Meta aims to broaden its scope to include training generative AI models like its Llama language models in the future.

The forthcoming MTIA chip is part of Meta's broader full-stack development program for custom silicon, catering specifically to its distinctive workloads and systems. Notable improvements in the next-gen MTIA include more than doubling the compute and memory bandwidth compared to its predecessor, while maintaining close alignment with Meta's workload requirements, particularly for ranking and recommendation models.

At the heart of the new MTIA design lies a focused architecture aimed at striking the right balance between compute, memory bandwidth, and capacity, crucial for serving ranking and recommendation models efficiently. The chip boasts an 8x8 grid of processing elements (PEs), delivering significantly enhanced dense and sparse compute performance compared to MTIA v1.

Furthermore, the new MTIA iteration features an upgraded network on chip (NoC) architecture, doubling the bandwidth and facilitating low-latency coordination between different PEs, essential for scaling MTIA to a wider range of challenging workloads.

In terms of hardware, Meta has developed a large rack-based system capable of accommodating up to 72 accelerators, each housing two chips. The system, meticulously designed to support the next-generation silicon, enables higher compute, memory bandwidth, and capacity, thereby accommodating a broad spectrum of model complexities and sizes.

Software integration has been a key focus for Meta, with the MTIA stack seamlessly integrating with PyTorch 2.0, leveraging features like TorchDynamo and TorchInductor. The Triton-MTIA compiler backend further optimises the software stack, enhancing developer productivity and expanding support for PyTorch operators.

Performance results indicate a significant leap in efficiency, with early tests showcasing a 3x improvement over the first-generation chip across key models. Meta's ongoing investment in custom silicon underscores its commitment to building the most powerful and efficient infrastructure for its AI workloads, with MTIA set to play a pivotal role in this long-term roadmap.