Constraint Beats Cleverness in AI Systems, says Millennium AI Lead at MLDS 2026

At the Machine Learning Developers Summit 2026, Vaibhav Jain, Team Lead of AI Engineering at Millennium, delivered a grounded, experience-heavy talk on a problem that has quietly become a common bottleneck for engineers: why AI agents that shine in demos collapse in production, and what it actually takes to make them reliable at scale.

Jain began with a question that immediately resonated with the audience: how many have built agents that worked perfectly in proofs of concept but failed in production? A sea of raised hands confirmed the pattern. 

The issue, he argued, isn’t the capability of AI models. “The problem is not the technology itself. The problem is how we are managing that technology when the stakes get real,” Jain said.

At Millennium, where billions of dollars are under management, accuracy is critical to making informed investment decisions, he said. This context shaped the central thesis of his talk: AI systems are inherently non-deterministic, but production systems must behave deterministically. The challenge, therefore, is not eliminating uncertainty, but managing it.

From Craftsmanship to Assembly Lines

To explain this shift, Jain reached back into industrial history. Early automobile manufacturers like Daimler Truck operated like artisans, crafting each car individually with little standardisation. The process was slow, inconsistent, and heavily dependent on human variability. Ford changed that with the assembly line, breaking production into discrete, repeatable steps that enabled scale and reliability.

Jain drew a direct parallel to AI systems today. Building an agent in a lab is like crafting a prototype car. Deploying it in production requires an assembly-line mindset, standardised workflows, controlled execution, and predictable outputs.

This shift, he argued, unlocks three critical pillars: reduced decision latency, scalability, and operational resilience. These are not optional optimisations; they are prerequisites for any AI system operating in high-stakes environments.

The Real Problem: Trust

Notably, Jain dismissed what many consider the “hard problems” in AI today. Choosing between models, frameworks, or tools, whether Claude, OpenAI, or specific orchestration libraries, is relatively trivial.

“The difficult decision is how you gain the trust of the user,” he said.

User trust, he explained, is fragile and quickly eroded. Even a few incorrect outputs can make users abandon a system entirely. In production, reliability perception matters as much as actual performance.

Most agentic systems don’t fail in intelligence, but in consistency and transparency.

If You Can’t Trace It, You Can’t Trust It

A recurring theme in Jain’s talk was observability. Every component of an agentic system must be logged and traceable, from tool calls and intermediate states to final outputs.

“If you can’t trace it, you can’t trust it,” he emphasised.

This goes beyond debugging. In high-stakes environments, auditability is essential for accountability, compliance, and iterative improvement. Teams must be able to reconstruct exactly why an agent made a particular decision.

This requirement also introduces a shift in engineering practices. Traditional CI/CD pipelines must evolve into CI/CD, along with continuous evaluation systems that measure outputs against predefined benchmarks and constraints.

One of Jain’s more counterintuitive insights was that constrained systems often outperform unconstrained ones.

In retrieval systems, for instance, allowing a model to search an entire embedding space can lead to poor results. Introducing filters and boundaries improves both accuracy and reliability. The same principle applies to agents: clearly defined roles, limits, and policies lead to better outcomes than unrestricted autonomy.

This challenges a common instinct in AI development: maximising flexibility. In production, discipline beats creativity.

Jain also addressed a growing issue in large language model systems: context overload.

While modern models can handle long contexts, he warned against exceeding 1,70,000 to 2,00,000 tokens. Beyond that, performance degrades due to attention dilution and recency bias. The model starts prioritising newer information over earlier inputs.

The solution isn’t bigger models, but better context management, loading only what is necessary, when it is necessary.

Human-in-the-Loop: Necessary, But Not Overused

Despite the push towards automation, Jain strongly advocated for keeping humans in the loop, especially in high-risk domains. However, he cautioned against over-reliance.

“The agent should do the bulk of the lifting, and the human should only take the most important decision,” he said.

This balance is critical. Too much human intervention defeats the purpose of automation. Too little invites risk.

To measure this balance, Jain introduced the concept of ‘edit distance’, the number of changes a human must make to an agent’s output before it becomes usable. Ideally, this number should approach zero.

Tracking this metric over time provides a clear signal of system improvement.

Start With a Baseline

One of the most practical takeaways from the talk was the importance of establishing baselines.

Before deploying agents, teams must quantify current performance, task duration, output accuracy, and the amount of human effort involved. Only then can improvements be meaningfully measured.

Jain shared an example: if a research report takes 20 days to produce manually and an agent reduces that to 20 minutes, the value is undeniable. But without the baseline, the impact cannot be proven.

Jain concluded with a pragmatic roadmap: start slow, run agents in parallel with humans, and gradually phase them into production as confidence builds.

This approach mirrors the broader theme of his talk: AI adoption is not just a technical journey, but a behavioural one. Users must see, verify, and trust the system before they rely on it.

In the end, the future of AI agents won’t be determined by how intelligent they are, but by how well they are managed.

And as Jain made clear, managing them is less about breakthrough models and more about discipline, structure, and trust.