Cloudflare Launches Controls to Block AI Training While Keeping Search Access

Cloudflare has announced a new Accountable designation for AI crawling and launched Disallow AI Training, a setting that lets website owners refuse AI training while continuing to allow search access from Accountable mixed-use crawlers.

Cloudflare said the move addresses mixed-use crawlers, which collect content for both search indexing and AI training. The company said these crawlers now account for 36.6% of verified crawler traffic on its network.

Cloudflare also said fewer than 1% of website owners block search crawlers, while 17% restrict AI training. Until now, the company said website owners had to choose between allowing AI training and blocking crawlers that also provide search access.

The new Disallow AI Training setting is designed to address this tradeoff. Cloudflare said it allows website owners to indicate that their content should not be used for AI training while allowing Accountable mixed-use crawlers to continue accessing the content for search.

“This is how we make the Internet better: preserving the openness that makes search valuable while giving the people and businesses behind the web meaningful control over how their work is used,” Matthew Prince, Co-founder and CEO of Cloudflare, said in a statement. 

Cloudflare is replacing its Block AI Bots switch with three independent controls for search, training, and agent activity.

The company will also provide recommended settings based on a website's business model. For sites that carry advertising, Cloudflare recommends allowing search crawling, disallowing AI training, and blocking AI agents on ad-carrying pages. For other sites, it recommends allowing search, AI training, and AI agents.

Cloudflare is also replacing its Managed Robots.txt feature with Bot Preference Sync. The feature allows website owners to set their crawling preferences once and automatically apply them across supported crawlers. Customers can change their settings and preferences at any time.

Cloudflare has established four criteria for crawler operators to receive its Accountable designation. 

The criteria include providing website owners with a way to opt out of AI training through robots.txt or a comparable standard, a way to opt out of AI-generated search summaries, URL-level visibility into how their content is used for search and training, and public confirmation that opting out of training will not affect a site's ranking in traditional search.

Cloudflare said Apple, Google, and Microsoft are now labelled Accountable because they have either demonstrated that they meet these criteria or provided specific timelines to meet them.

“We look forward to continued collaborative engagement with companies like Apple, Google, and Microsoft as we work together to build a healthy ecosystem,” Prince added.

Google offers Google-Extended, which allows websites to opt out of AI training without opting out of Search. Cloudflare said other leading AI companies operate separate crawlers for search and training. This allows Cloudflare to block their training crawlers without affecting search, and the company also categorises the relevant crawlers as Accountable.

Cloudflare said it publishes the full list of qualifying operators on Cloudflare Radar.

The company is also participating in open standards work at the Internet Engineering Task Force, including ai-prefs, a specification under development that would allow websites to express their AI access preferences in a standardized and portable way.

Cloudflare said every operator it designates as Accountable must provide website owners with a way to opt out of AI summaries. Today, this is a yes-or-no choice that is set separately with each operator. By early next year, Cloudflare aims to allow website owners to control how much of their content is included in AI-generated summaries from one place.