Article complete

Anthropic’s CI Jobs Grew 25x in 6 Months as Claude Wrote 80% of Code

The number of tests in its codebase grew tenfold with only a nominal increase in engineering staff.

Illustration of abstract human profiles with lines and nodes, symbolizing connections and AI development.

Key takeaways

  • Anthropic's continuous integration jobs surged 25 times in six months, reflecting a significant increase in code production.
  • Claude, the AI system, authored 80% of the code, highlighting its pivotal role in the development process.
  • The number of tests in the codebase expanded tenfold, but engineering staff remained relatively unchanged.
  • The increased workload strained the test impact analysis service, necessitating multiple patches and a redesign.
  • Improvements in the CI process are essential, as delays can lead to outdated test data and inefficiencies.

Takeaways drafted with AI from the article text.

The surge strained Anthropic’s test impact analysis service, which determines which tests need to run for each code change. 

Engineers patched the service three times before redesigning it, with the fixes lasting 70 days, 29 days and less than a day, respectively.

“Writing code is no longer the constraint, and once PR review gets accelerated, CI starts feeling the pressure,” Anthropic engineer Sachin Malhotra wrote in the blog.

The original implementation operated as a single process, with one writer maintaining the running history for each test. As CI jobs increased, the listener began falling behind the incoming queue.

A 20-minute lag could leave tens of thousands of test updates unapplied to the selector, the company said. That could cause the system to use stale data when selecting tests, resulting in unnecessary test runs or failures to run newly added or fixed tests.

The first response was to double the service’s CPU cores. 

Anthropic later used Claude Tag, an internal version of its AI tool, to monitor the service and alert Malhotra when the listener fell more than 50,000 jobs behind. 

In February, engineers split the listener’s state into package-level shards, allowing separate workers to process results in parallel. The change lasted 29 days before the service came under pressure again.

By March, the process was reaching its memory limit by mid-afternoon on most weekdays. Daily restarts bought less than a day of relief and pushed the listener further behind, sometimes by more than an hour.

Anthropic ultimately replaced the singleton architecture with a distributed design backed by an in-memory data store. 

Listener workers can now process results, append them to a journal and move on without retaining the full state in memory. A separate consumer periodically rolls the journal into per-test history, which the selector can query.

The redesign took one engineer three weeks, compared with an estimated quarter a year earlier. Anthropic said the service has remained stable since the cutover and tuning, although the distributed architecture costs more to operate.

Malhotra advised engineering teams to plan for rapid increases in CI load as AI coding agents generate more pull requests and tests. 

“Assume your architecture will be at a 25x load within two quarters,” he wrote, adding that teams can now design initial systems for 10–20 times their perceived scale where budgets allow.

He also recommended keeping state out of individual processes and avoiding critical services running as a single instance unless they can be measured and subjected to controlled changes.

“CI is evolving too quickly to proceed any other way,” Malhotra wrote.

Published

Spotted an error? Tell our editors. Read our editorial standards and corrections.