In a U-Turn, Cloudflare Now Lets You Crawl Entire Websites With a Single API Call

Cloudflare has introduced a new API endpoint that allows developers to crawl entire websites with a single request, according to a product update published in the company’s changelog.

The feature is part of Cloudflare’s Browser Rendering platform and introduces a new ‘/crawl’ endpoint that automatically discovers and extracts content from websites, starting from a single URL. 

In the announcement, Cloudflare said developers can “crawl an entire website with a single API call using Browser Rendering's new /crawl endpoint.”

According to the company, developers submit a starting URL and the system automatically finds and processes pages across the site. 

“Submit a starting URL, and pages are automatically discovered, rendered in a headless browser, and returned in multiple formats, including HTML, Markdown, and structured JSON,” Cloudflare said in the update.

Cloudflare stated that the feature is designed for tasks such as AI data collection, research, and monitoring web content. 

The company said the capability is “great for training models, building RAG pipelines, and researching or monitoring content across a site.”

Crawl jobs run asynchronously. “You submit a URL, receive a job ID, and check back for results as pages are processed,” Cloudflare added in the post.

According to the documentation, the crawler can automatically discover pages through sitemaps and internal links. 

Developers can also configure parameters such as crawl depth, page limits, and URL inclusion or exclusion patterns to control how the crawler navigates a site.

Cloudflare said the API can return results in several formats, including HTML, Markdown, and structured JSON. The JSON output uses the company’s AI tools to extract structured data from pages.

Additional features include incremental crawling options that allow users to skip pages that have not changed recently, reducing compute usage for repeated crawls. 

The crawler can also operate in a static mode that fetches HTML without rendering JavaScript, which Cloudflare said enables faster crawling for static websites.

The crawler also respects website policies. Cloudflare said the tool “honours robots.txt directives, including crawl-delay.”

The new crawling capability is currently available in open beta and can be accessed by developers using both the free and paid plans of Cloudflare Workers.

After the announcement, several users on X pointed out the irony that Cloudflare, long revered as a defender of websites against unwanted AI crawlers and scrapers through tools like one-click bot blocking and strict enforcement of robots.txt, has now launched its own web crawling service. 

Last year, the company announced that it would start blocking AI crawlers by default. Every new domain signing up with the service now is asked whether they want to allow AI crawlers.  

The default option is a “no”, and companies like Gannett Media, Condé Nast, Quora, Ziff Davis, and Reddit backed the initiative. In December last year, Cloudflare told WIRED that it had blocked 416 billion AI bot scraping requests in five months.