How to train compute optimal large language models?

New research from DeepMind attempts to investigate the optimal model size and the number of tokens for training a transformer language model under a given compute budget.
compute optimal large language

New research from DeepMind attempts to investigate the optimal model size and the number of tokens for training a transformer language model under a given compute budget. The team trained over 400 language models ranging from 70 million to 16 billion parameters on 5-500 billion tokens. The team found that for compute optimal training, the model size and the number of training tokens should be scaled equally. This means that the doubling of model size should be accompanied by the doubling of the training tokens.

Rise of large language models

This is truly the age of large language models. When GPT-3 model was introduced, it caught the fancy of the research world – 175 billion parameters were unheard of. It’s been two years since its introduction, and in this time, several models have been launched, each larger than the previous one. The large autoregressive transformers display impressive performance on many tasks by using various evaluation protocols like zero-, few-shot, and fine-tuning.

This impressive performance comes at the cost of massive compute and energy requirements. This has been a subject of much debate. The negative implications of such large models have been raised time and again; one popular example is AI researcher Timnit Gebru who was ousted from Google allegedly because of a paper she co-authored, which spoke about the downside of building, maintaining, and training such massive models.

Subscribe to our Newsletter

Join our editors every weekday evening as they steer you through the most significant news of the day, introduce you to fresh perspectives, and provide unexpected moments of joy
Your newsletter subscriptions are subject to AIM Privacy Policy and Terms and Conditions.

The research

The training compute budget is often calculated in advance. Since it is feasible to train these large models not more than once, it becomes very critical to accurately estimate the best model hyperparameters for a given compute budget. In the past, it has been proved that there exists a power-law relationship between the number of parameters and the performance of an autoregressive language model. 

An earlier study showed that large models should not be trained to their lowest possible loss to be compute optimal. While the DeepMind researchers have concluded the same in their recent study, they also estimate that the large models must be trained for many training tokens than recommended earlier. The previous study showed that for a ten times increase in the computational budget, the size of the model should increase 5.5 times, and the number of training tokens must increase by 1.8 times. However, the DeepMind study shows that the model size and the number of training tokens should scale in equal proportions.

Based on the estimated compute-optimate frontier, the DeepMind researchers predicted that for training Gopher (280 billion parameter language model), an optimal model should be four times smaller and should be trained on four times more tokens. This was verified by training a compute-optimal 70 billion model called Chinchilla on 1.4 trillion tokens. The researchers could show that Chincilla outperformed its larger counterpart, Gopher, and also reduced the inference cost considerably (due to reduced model size), which facilitates downstream uses on smaller hardware. The benefits of a more optimally trained smaller model extend beyond the immediate benefits of its improved performance.

Need for quality datasets

The DeepMind research calls for an increased focus on dataset scaling, which in turn is only beneficial when the data is of high quality. “Larger datasets will require extra care to ensure train-test set overlap is properly accounted for, both in the language modelling loss but also with downstream tasks,” the authors wrote.

Apart from this, the research community must also account for the ethical and privacy concerns associated with such large models. As observed in the past, the large datasets collected from the web contain toxic language, bias, and other private information. A better understanding of large language model performance and its interaction is an important future research area.

Read the full paper here.

Shraddha Goled
I am a technology journalist with AIM. I write stories focused on the AI landscape in India and around the world with a special interest in analysing its long term impact on individuals and societies. Reach out to me at

Download our Mobile App


AI Hackathons, Coding & Learning

Host Hackathons & Recruit Great Data Talent!

AIM Research

Pioneering advanced AI market research

Request Customised Insights & Surveys for the AI Industry


Strengthen Critical AI Skills with Trusted Corporate AI Training

Our customized corporate training program on Generative AI provides a unique opportunity to empower, retain, and advance your talent.

AIM Leaders Council

World’s Biggest Community Exclusively For Senior Executives In Data Science And Analytics.

3 Ways to Join our Community

Telegram group

Discover special offers, top stories, upcoming events, and more.

Discord Server

Stay Connected with a larger ecosystem of data science and ML Professionals

Subscribe to our Daily newsletter

Get our daily awesome stories & videos in your inbox

Subscribe to Our Newsletter

The Belamy, our weekly Newsletter is a rage. Just enter your email below.