Behind every AI model is a fragile, high-performance system where breakdowns are inevitable. The real challenge is not preventing failures but managing them in real time.
Speaking at MLDS, India’s biggest agentic conference organised by AIM, Mohamed Imran K R, CTO of E2E Networks, offered a rare, ground-level look at what it really takes to deploy and run large-scale AI systems.
Imran began by reframing how engineers should think about AI systems. Rather than prioritising models or inference pipelines, he stressed the importance of infrastructure resilience.
“The technology is so complicated that there is always a statistical probability of one of the key components going down.”
He compared AI infrastructure to space missions, which are highly advanced but never perfectly reliable. Unlike traditional systems, AI clusters operate at extreme limits of compute, bandwidth, and temperature where failures are not exceptions, but expected.
Power Meets Complexity
Modern AI systems rely heavily on GPUs, particularly from NVIDIA.
“A new generation GPU card has 17,000 CUDA cores. That’s a very massive number of cores,” he said, adding that it is a complicated piece of technology.
According to him, the new NVIDIA Blackwell series can achieve interconnect speeds of 6.4 terabits per second, with operating temperatures ranging between 70°C and 80°C. At this scale, systems push hardware to its limits. Even a small failure rate becomes significant.
Small Failures, Big Impact
Imran warned that minor failure rates can cascade into major performance losses.
Citing the example of xAI, which currently operates more than hundreds of thousands of nodes, he explained that if 1% of GPUs fail, one can end up with a 10% degradation across the cluster.
Imran said this happens because failures are not neatly distributed. Instead, they occur randomly across nodes, disrupting parallel workloads and slowing down the entire system.
He cited Netflix’s approach to resilience as an early example of designing systems for failure. The company built an open-source tool called Chaos Monkey that deliberately shuts down random servers in a cluster without warning.
The Economics of AI
AI completely changes the cost structure, with infrastructure becoming a far larger expense compared to software systems.
“Previously, SaaS companies would spend 2–3% of revenue on infrastructure. With AI, that number can be 60-70%,” Imran said, counting the costs of GPUs and ML engineers.
This has elevated the role of finance teams. “The CFO has a much bigger role to play today,” Imran added.
GPU utilisation, particularly metrics like Model FLOP Utilisation (MFU), has become a critical business KPI. Poor utilisation directly translates to wasted millions.
Designing for Failure
Rather than trying to eliminate failures, modern AI systems are built to recover from them seamlessly.
Checkpointing forms the foundation of this approach. As Imran put it, “Checkpointing is… the most basic resiliency. If something fails, you can always restart.”
Despite all this complexity, the goal remains simple: to keep GPUs busy.
“If your MFU goes down, a training run that should take 10 days becomes 12 or 15.”
This is why organisations obsess over metrics like MFU and network bandwidth. Even small inefficiencies can dramatically increase costs and timelines.
He also touched on a key industry debate about why companies invest in their own infrastructure despite the rise of cloud GPUs.
Imran pointed to two main reasons for this. The first is data control and compliance. “If you are working in fintech… your data cannot go outside the organisation’s boundary,” he said. The second is long-term economics, where owning infrastructure can become more cost-effective at scale.
“As you scale, your per-GPU cost is much simpler when you run your own infrastructure.”
While cloud works well for lightweight use cases, serious AI development, especially foundational models, often requires dedicated infrastructure.
According to Imran, infrastructure is not about perfection but about resilience. In the AI era, success depends not on preventing failures but on building systems that adapt, recover, and keep running no matter what breaks.
Your reaction
Discussion
What's your take on this story?
No comments yet — be the first to share your view.