Lovable’s Internal LLM Routing Handles 1 Bn Tokens/Min While Preserving Prompt Caching

Lovable, the AI-powered coding platform, has built an internal routing system capable of handling more than 1 billion tokens per minute across multiple large language model providers while maintaining reliability and efficiency.

The company disclosed details of the system in a recent engineering blog post, where it outlined how its platform manages traffic across different AI model providers to maintain reliability during outages, rate limits, or slow responses.

“We use LLMs extensively at Lovable, across a range of different use cases,” wrote Lovable engineer Mårten Wiman in the blog post. “Under peak traffic, we’re processing more than one billion tokens per minute.”

The system was designed to address situations where the model providers' platform depends on experienced service disruptions. The company stated that its infrastructure relies on several external providers, including Anthropic, Google Vertex AI, and Amazon Bedrock.

Its AI agents rely heavily on prompt caching, which allows subsequent requests to reuse previously processed prompts and context. The company added that this approach helps reduce both latency and operational costs during agent workflows.

“Prompt caching relies on consecutive requests being sent to the same provider,” Wiman wrote in the post.

According to Lovable, conventional fallback systems that simply redirect traffic to another provider when failures occur can break this continuity. Such rerouting forces the system to reprocess the full context of a request, increasing latency and token usage.

Lovable’s routing infrastructure addresses this by assigning projects to specific provider sequences for short time windows. According to the company, this design allows consecutive requests from the same project to be routed consistently while still enabling failover when a provider becomes unavailable.

“Using multiple fallback chains with project-level affinity gives us a way to preserve the benefits of prompt caching,” Wiman wrote.

Lovable added that the system continuously monitors provider performance and automatically adjusts traffic allocation. This allows the platform to shift traffic away from providers returning errors while directing more requests to providers with available capacity.

The automated system allows the platform to maintain stable performance even when individual LLM providers experience disruptions, the company stated.