AI Scaling Laws Crumble Under Token Pressure

Have LLMs Hit a Wall? Microsoft chief Satya Nadella tackled this hot-button issue at Microsoft Ignite 2024, offering a refreshingly candid take on the discussion.  “There’s a lot of debate on whether we have hit the wall with scaling laws. Is it going to continue? The thing to remember, at the end of the day, is that these are not physical laws. They are just empirical observations that held true, much like how Moore’s Law did for a long time,” he said.  Nadella welcomed the scepticism and debates, calling them beneficial to push innovation in areas such as model architectures, data regimes, and systems architecture. He also discussed OpenAI’s new scaling law, which focuses on test-time computing, and how it will be integrated into features like Copilot Think Deep