‘Claude Code is Built Around Prompt Caching’

Anthropic’s engineering team revealed that it designed Claude Code around a core architectural constraint: prompt caching must work at very high hit rates, or long-running agentic workflows become economically and operationally infeasible.

“At Claude Code, we build our entire harness around prompt caching,” Thariq Shihipar, a member of technical staff of Anthropic, said in a post on X. “A high prompt cache hit rate decreases costs and helps us create more generous rate limits for our subscription plans.”

Prompt caching allows large language models to reuse computation from previous request prefixes rather than reprocess the entire prompt on every turn.

For systems like Claude Code—where agent sessions can stretch into tens or hundreds of thousands of tokens—this reuse determines whether latency and cost remain manageable.

Shihipar explained that the system relies on strict prefix matching. The API caches everything from the start of a request up to each cache-control breakpoint, meaning any change inside that prefix invalidates everything after it. As a result, ordering and stability of prompt components become first-order design decisions.

“Prompt caching is a prefix match. Any change anywhere in the prefix invalidates everything after it,” Shihipar wrote. “Design your entire system around this constraint.”

Claude Code therefore places static content first—system prompt and tool definitions—followed by project-level files, session context, and finally conversation messages. This maximises how many requests share identical prefixes across sessions and users. 

Shihipar noted that the approach is fragile in practice, citing past failures caused by shuffled tool order, parameter changes, or embedding timestamps inside static prompts.

Rather than modifying the system prompt when information changes, Claude Code injects updates as system messages in later turns, preserving the cached prefix while still conveying new facts. 

“Use system messages instead of system prompt changes,” Shihipar wrote, arguing that editing the system prompt is one of the fastest ways to trigger expensive cache misses.

Model switching is also avoided mid-session. Because caches are model-specific, moving from a larger model to a cheaper one requires rebuilding the entire context. “If you’re 100k tokens into a conversation with Opus and want to ask a question that is fairly easy to answer, it would actually be more expensive to switch to Haiku than to have Opus answer,” Shihipar wrote.

Tool management follows the same logic. Claude Code never adds or removes tools during a conversation. Instead, tools are used to represent state transitions, such as entering or exiting plan mode. “Never add or remove tools mid-session,” Shihipar wrote. “Use tools to model state transitions rather than changing the tool set.”

For large tool catalogues, Claude Code uses deferred loading, where lightweight stubs are always present, while full schemas are fetched only when selected. Shihipar described this as a way to keep the cached prefix stable while still giving the model access to dozens of potential tools.

Compaction, which is summarising long conversations when the context window fills, also had to be made cache-safe. 

Claude Code performs compaction using the same system prompt, user context, and tool definitions as the parent session, appending the summarisation request at the end so the cached prefix is reused.

“Monitor your cache hit rate like you monitor uptime,” Shihipar wrote. “A few percentage points of cache miss rate can dramatically affect cost and latency.”