OpenAI Unveils 16MB, 10-Min Model Training Challenge on NVIDIA H100 GPUs

OpenAI has launched Parameter Golf, a model training challenge focused on building highly efficient language models under strict size and compute constraints.

The challenge involves participants training the best-performing language model that fits within a 16MB artefact and can be trained in under 10 minutes on an NVIDIA 8x H100 GPU cluster, with evaluation based on compression performance on the FineWeb validation set.

OpenAI is framing the initiative as a shift from scaling models up to optimising them under tight constraints, effectively treating it as a parameter-limited optimisation problem.

“If you're familiar with neural scaling laws, you can consider this challenge a form of L(N) optimisation, where the objective is to optimise the lowest loss given a fixed number of parameters,” the company said.

The setup pushes participants toward unconventional approaches in architecture and compression, including parameter tying, low-rank training, and novel tokenisation strategies.

“We're excited to see how optimising for a parameter-constrained setting pushes people toward unique architectures… and other creative submissions,” OpenAI said in a blog.

To support participation, OpenAI is offering $1 million in compute credits to developers and researchers working on submissions. “We also know compute is expensive, so OpenAI is sponsoring $1,000,000 in compute credits to help people get started training their models,” the company said.

The company is also using the challenge as a talent signal, particularly for early-career researchers.

“The Model Craft Challenge is designed in that spirit: testing the ability to tackle unfamiliar problems with creativity and rigour,” OpenAI added.

The challenge started on March 18 and will run till April 30, with submissions evaluated on reproducibility, compression efficiency, and adherence to constraints. Entries must be fully self-contained and meet strict evaluation requirements.

OpenAI said the broader goal is to explore the limits of parameter-efficient AI systems, a direction that is becoming increasingly relevant as developers look to deploy capable models under cost, latency, and hardware constraints.