Generative AI is moving into production at scale. Yet while much of the hype has focused on what these systems can do, far less attention has been paid to what they cost to run.
What works in controlled demos is now faltering under real-world conditions.
Rising costs, inconsistent outputs, and reliability challenges are exposing a deeper gap, not in the models themselves, but in how they are operated.
“In my experience, what breaks first is not the model, it’s the system around it,” says Aravind Putrevu, Senior Business Leader, Code Rabbit. “Pilots are often built in isolation, with clean datasets and controlled prompts. In production, you face messy, dynamic data, latency constraints, and strict reliability expectations.”
This transition from pilot to production is proving far more complex than anticipated. According to Bindu Sunil, Chief AI Officer at Mindsprint, the issue often begins at the very start. “What I have seen most consistently is that the pilot was never designed to scale in the first place,” she says.
“Teams,” she explains, “make shortcuts on data pipelines, hardcode things, and skip monitoring entirely”, only to find that “there is no proper feature store, no CI/CD for models, or no drift detection in place” when they attempt to deploy.
Beyond infrastructure gaps, organisational challenges are compounding the problem. “In the enterprise environments I have worked with, that no-man's-land between data science and engineering is where AI use cases quietly die,” Sunil adds, pointing to a lack of ownership and coordination.
As systems scale, costs are emerging as another major fault line. Unlike traditional software, GenAI usage is highly variable and difficult to predict. “GenAI costs are highly non-linear at scale,” said Putrevu. “Token consumption grows silently, and without strong controls, it can spiral quickly.”
In many cases, enterprises are seeing dramatic increases in spending. “Typically, I see costs increase anywhere between 3x to 10x when moving from pilot to production,” Putrevu says. Token usage, he notes, can grow “exponentially, sometimes 5x or more”, due to real-world variability.
Manoj Dhanda, Founder and CEO, Utho Cloud, echoed this unpredictability. “GenAI costs at scale are highly volatile because they are driven by inconsistent and often unpredictable user behaviour,” he says. A small number of users often account for the majority of consumption, creating “sudden and significant cost fluctuations.” Without monitoring, “a stable monthly bill” can turn into “a multiple-fold increase within a short period.”
At scale, even the cost structure itself changes. “In production, this balance evens out… moving closer to a 50-50 split,” Dhanda points out, as infrastructure requirements grow alongside model usage.
These pressures are forcing enterprises to rethink how they deploy AI systems. Increasingly, organisations are moving away from relying on a single model.
“This shift is happening faster than many people expected,” Sunil says. “A year ago, most enterprises were asking which single model they should use. Today, the smarter ones are asking how to orchestrate multiple models across a workflow.”
This shift is driven by both cost and performance considerations. Using a single large model for all tasks is “both expensive and often the wrong tool for the job,” she said. Instead, enterprises are adopting layered approaches tailored to task complexity, latency, and data sensitivity.
Dhanda pointed to similar trends in the field, where enterprises select models based on “cost… data security… latency… and workload”, often combining large models with smaller or domain-specific alternatives.
However, this move towards multi-model systems introduces a new level of operational complexity—one that traditional DevOps or MLOps frameworks are not fully equipped to handle.
That gap is giving rise to LLMOps.
Enter LLMOps
LLMOps, or large language model operations, refers to the layer of systems, tools, and practices required to run GenAI reliably in production within enterprises. It goes far beyond simply calling a model API, covering prompt management, model routing, monitoring, cost control, guardrails, and continuous feedback loops.
As multiple experts noted, it is what enables enterprises to manage multi-model setups, track performance and costs, and ensure consistent outputs at scale. Practitioners of LLMOps typically include a mix of AI engineers, platform teams, data scientists, and increasingly dedicated LLMOps or AI operations teams within enterprises and IT services firms. It is becoming essential because, without it, companies lack visibility into usage, cannot control spiralling costs, and struggle to maintain reliability and governance as GenAI systems scale beyond pilots into real-world environments.
“LLMOps is exactly what makes multi-model setups manageable in practice. Without it, you are essentially flying blind,” Sunil says. It provides “observability, evaluation pipelines, version control, and cost tracking that turn a collection of model calls into something you can actually govern and improve over time.”
In production environments, LLMOps extends far beyond simple API integration. “It involves prompt management, versioning, evaluation pipelines, monitoring output quality, and managing drift,” Putrevu mentions. It also includes guardrails, caching strategies, and feedback loops to continuously refine outputs.
Dhanda describes it as a behavioural and operational layer. “How to filter the prompt, how to minimise the prompt… what guardrails can be installed… what routing or fallback can be set up,” he says, along with “tracing every call… how many tokens have been used… what cost has come, and what outcome has come.”
Without such systems, he adds, “it will not be possible” to control costs or ensure data privacy at scale.
Even so, the space is still evolving. Varun Khare, CEO and Co-Founder, NimbleEdge, notes, “LLMOps are still largely coming out as prepacked solutions rather than something requiring specialist teams,” though this could change as deployments become more customised.
At the same time, LLMOps itself is expanding in scope. Khare observes that it is “moving more towards code generation and runtime for dynamic tools and skills,” which brings new security challenges.
India’s approach to this transition is being shaped by its own constraints. Data privacy and sovereignty are emerging as key priorities, particularly in regulated sectors. “Sovereignty is not negotiable in any way,” Dhanda says, noting growing demand for self-hosted models and GPU infrastructure.
Cost sensitivity is also pushing enterprises towards optimisation. Indian firms, experts say, are more likely to experiment with smaller models and tighter controls rather than relying entirely on large external providers.
This is also creating an opportunity for the country’s IT services ecosystem. Dhanda pointed out that companies like TCS, Infosys, and Wipro are “making LLMOps platforms and selling them everywhere”, signalling a shift towards operational AI services.
For companies building infrastructure, the focus is increasingly on enabling scale.
“We see LLMOps as making GenAI actually work at scale,” Padma Reddy Sama, Co-Founder, BharathCloud says, adding that “in reality, it’s not about one model. You’ll have setups with a single LLM or multiple LLMs working together, and alongside them, SLMs for more specialised tasks.”
Despite differing views on maturity, there is broad agreement on where things are headed.
“I do see LLMOps becoming a critical discipline,” Putrevu says, describing it as a natural evolution alongside DevOps and MLOps.
Sunil expects rapid adoption. “Within two to three years, I would expect every enterprise that is serious about AI to have some version of a formal LLMOps function”, though it will be “more complex than DevOps” due to the non-deterministic nature of AI systems.
Dhanda was more direct: “It will not work without it… Within two to three years, it will become huge.”
As enterprises move beyond experimentation, the shift is becoming clear. Building AI models is no longer the hardest part. Running them—reliably, efficiently, and at scale—is.
Your reaction