AMD and Kimi Deliver 3.2× Lower Latency for Coding

AMD and Moonshot AI have jointly developed a new inference stack for Kimi K2.6 that they say significantly improves the performance of AI coding agents running on AMD Instinct MI355X GPUs. 

Instead of optimising existing chatbot infrastructure, the companies redesigned the serving software around AI agents that write code, call tools and maintain long-running conversations. 

In benchmark tests, the new stack reduced worst-case response latency by 3.2× and increased total token throughput by 7.7%.

The companies argue that AI agents place different demands on infrastructure than chatbots because they maintain much longer conversations, repeatedly access earlier context and spend much of their time waiting for tools such as web search or code execution. 

As AMD puts it, “Agentic coding is cache-dominated and bursty,” making efficient memory management more important than simply adding more compute.

To address this, AMD built a new software layer called Unified Memory & Bandwidth Pool (UMBP) that allows AI models to retrieve previously processed context from GPU memory, system memory or storage instead of recalculating it. 

According to the company, this turns cache management into something the scheduler can actively optimise rather than treating it as passive storage. AMD said the architecture lets the system decide not just “who has the prefix” but “who can serve it fastest.”

The companies evaluated the software using a custom benchmark derived from ProgramBench, which runs real Kimi coding sessions rather than synthetic inference requests. 

In one configuration with 128 concurrent coding sessions, enabling the new cache architecture reduced p99 time-to-first-token from 17.3 seconds to 5.4 seconds while increasing throughput by 7.7%. 

A second experiment, which simulated requests moving between inference workers, produced a 4.9% throughput improvement and a 1.6× reduction in tail latency by allowing cached context to remain accessible after migration.

Beyond the cache architecture, AMD introduced additional optimisations including incremental transfer of model state between inference stages, a tokeniser that processes only newly added conversation turns instead of re-tokenising entire sessions, and scheduling techniques that overlap tool execution with model inference to reduce idle GPU time.

AMD said it also verified that the infrastructure changes did not reduce model quality using Moonshot AI’s Kimi Vendor Verifier. 

The company reported that enabling constrained tool decoding achieved 100% JSON schema accuracy while maintaining tool-calling performance comparable with Moonshot’s reference implementation. AMD said the “performance wins come with no accuracy tax.”

The companies said future work will focus on optimising the stack for the upcoming Kimi K3 model and extending the software to additional inference engines including AMD’s ATOM platform as well as the open-source SGLang and vLLM frameworks.

The announcement follows Moonshot AI’s release of Kimi K3 last week, a 2.8-trillion-parameter open-weight model designed for reasoning, coding and agentic workloads. 

The model quickly rose to the top of several coding benchmarks and supports a 1 million-token context window, intensifying competition with frontier models from Anthropic and OpenAI.

Recently, AMD has broadened adoption of its AI hardware across the industry. The company has unveiled partnerships with OpenAI, Meta, Oracle, and Microsoft around its next-generation AI infrastructure, while its latest agreement with Anthropic includes deploying up to 2 GW of AMD Instinct MI450 GPUs and a multi-year engineering collaboration.