Enterprise AI is Turning Model Choice Into a Routing Problem
By Ridhika Basnet ·
Enterprise AI adoption is creating a new infrastructure problem. Beyond enterprises being spoilt for choice, the more relevant question is which model should handle a particular request, and where that model should run.
Stephen Watt, Distinguished Engineer and VP of the Office of the CTO at Red Hat, has been watching that shift as companies move from small AI pilots to larger-scale deployments. He sees a rapid growth in AI use, but also a problem that becomes harder to ignore at scale: inference costs. On what enterprises discovered as they moved beyond pilots, Watt is blunt: "They ran out of money."
Uber is a case in point. Earlier this year, the cab aggregator announced that it had used its entire annual AI budget in just four months. Watt calls it the first "canary in the coal mine" for the cost problem.
He tells AIM that part of the challenge was how companies access models. Enterprises often use intermediary services to access models from providers such as Anthropic and OpenAI, allowing them to move quickly but making granular, up-to-the-minute spending harder to see.
That creates an economic problem when AI moves from a handful of pilots to widespread use. An application may make only a few model calls during testing, but an agent operating across tools and systems can make repeated inference requests as it works through a task.
The Agentic Paradox
Watt calls this the "agentic paradox." Enterprises find it easier and quicker to adopt hosted models. There is little infrastructure to build, and companies can access increasingly capable models through an API.
"Using someone else's infrastructure is always faster," Watt says. "You have much less control."
The problem becomes clearer with scale. "The paradox is, you'd be crazy not to start with a hosted model, but after a while, it eats up so much of your margin because of the expenses," he adds.
That does not mean enterprises have to abandon frontier models. It changes how they use them.
Watt's approach starts by evaluating an AI system based on individual skills it needs to perform. "You can decompose your agent into a series of skills that it's using, and you can do agent evals on each skill," he explains.
That allows an enterprise to identify where a frontier model is necessary and where another model may be sufficient. Red Hat's open-source vLLM inference server can run open-weight models on enterprise infrastructure.
Watt cites Alibaba’s Qwen and Z.ai’s GLM as examples of models that can be deployed this way, potentially reducing the cost of inference compared with hosted frontier models.
But moving workloads to self-managed infrastructure requires enterprises to have the technical skills to operate the systems with confidence that an application or agent will perform adequately. That makes model selection and infrastructure selection increasingly connected decisions.
Routing Becomes the Control Layer
This is where inference routing enters the architecture. Watt describes Red Hat's model-as-a-service layer as an inference gateway or router that can determine where an inference request should go.
A request could be sent to a hosted frontier model, an open-weight model running in an enterprise data center or, for suitable workloads, a model running at the edge.
The decision is not based solely on price. Different models perform differently on different tasks, and different deployment environments impose constraints around memory, latency, and throughput.
Watt's team tested NVIDIA Jetson devices as an example of the edge environment. "The memory first becomes your first arbiter," he notes.
If a model does not fit into the available memory, it cannot run there regardless of its other advantages. Latency then becomes another constraint. Watt says his team was seeing roughly 20 tokens per 100 milliseconds as an average rate on the edge devices it analysed.
The same calculation changes in a data centre. Larger models can require substantially more sophisticated hardware and system configuration. Watt cites Kimi K3, Moonshot AI's newest open-weight model, as an example, estimating it needs infrastructure built for a model "roughly" in the trillion-plus-parameter range.
Red Hat's llm-d project is designed to distribute inference components to help run larger models with more predictable throughput.
Watt shares two internal prototypes that illustrate how this routing approach could work economically. One, informally called "Price Tag," performs semantic analysis on a request and determines where to route it.
His deliberately simple example was a "hello" sent to a top-tier frontier model. Watt estimates that such a request could cost around 20 cents. Sending it instead to a self-hosted open-weight model would carry ordinary data centre costs such as power, cooling, storage, and server amortisation, which he estimates "rounds down to about like less than a cent for the request."
The second prototype, called "Side-Eye," uses a frontier model to supervise a cheaper open-weight model. In a coding workflow, the cheaper model can create a plan, while the frontier model reviews that plan and provides targeted feedback. The cheaper model then incorporates the feedback and handles execution.
That changes the economics because of how frontier providers price tokens. Watt says input tokens sent to a frontier model typically cost about 25% of what output tokens cost, so a workflow that sends a large, detailed plan as input but gets back only a short review as output shifts most of the cost onto the cheaper side of that pricing split.
Hybrid AI
Once enterprises begin routing requests, deployment location becomes part of the same calculation. Some workloads may justify the capabilities of a frontier model hosted externally. Others may be suitable for an open-weight model running on enterprise infrastructure.
Watt says most hybrid, self-managed inference today involves smaller models that can run on one server. Larger models require increasingly sophisticated systems, and model-as-a-service infrastructure can provide another layer for routing requests between environments.
"Hybrid's just another word for having control," Watt remarks.
He argues that enterprises should assume their requirements will change. A model that works well today may be replaced by another model with different performance, cost, or hardware requirements.
If the infrastructure underneath an application can change without forcing the application itself to be rebuilt, companies can make those changes without repeatedly absorbing the cost of switching.
Cost and Security Move Together
The same infrastructure layer is also becoming important for security. On what concerns CIOs and CEOs the most, Watt narrows it to two issues. "It's cost, and then security."
The security problem becomes more complicated when agents can act across enterprise systems. Watt describes cases involving privilege escalation, including situations where an agent could gain access beyond what a user intended.
Red Hat's OpenShell project is designed to sandbox agents and constrain their access to networks and files.
That places cost and security on the same infrastructure layer. Enterprises need to control not only which model receives a request, but also what that model or agent is allowed to do once it receives it.
The emerging architecture is therefore less about finding a single model for enterprise AI and more about controlling the path an AI request takes.