- Published on April 1, 2025
- In AI Features
AI Crawlers: The Nasty Bugs Causing Trouble on the Internet
AI crawlers are eating up web resources, and site administrators are looking for defence mechanisms to protect against big bills.
Image by Diksha Mishra
- By Ankush Das
AI tools with web search capabilities, such as Anthropic’s Claude, browse the internet to deliver users the needed information. Perplexity, OpenAI, and Google offer similar features through ‘Deep Research’.
In a blog post, Cloudflare explained that these web crawlers, often referred to as AI crawlers, deploy the same techniques as search engine crawlers to gather available information.
While the aim of AI crawlers is to assist users, they may be causing more damage on the internet than one realises. They are believed to increase server resource usage for website administrators, leading to unwanted bills and causing disruptions.
AI Crawlers on The Rise of Being a Hassle
Gergely Orosz, creator of The Pragmatic Engineer newsletter, shared on LinkedIn, “AI crawl
📣 Want to advertise in AIM? Book here
Ankush Das
I am a tech aficionado and a computer science graduate with a keen interest in AI, Coding, Open Source, Global SaaS, and Cloud. Have a tip? Reach out to ankush.das@aimmediahouse.com
Related Posts
Top 10 Companies That Crowned Hyderabad as India’s Greenfield GCC Leader in 2025
Telangana has attracted over 75 greenfield GCCs in 2025, compared with 40-plus in Karnataka.
The AI Coding Gold Rush Ends Where Harness Begins
“Only 30% of software engineering happens on the laptop. The real 70% starts after you commit the code,” says Jyoti
How Gradient-Boosting is Quietly Powering India’s Research Push
From groundwater and slopes to carbon sinks, tools like CatBoost are enabling Indian scientists to extract insights and drive sustainability.
India’s Data Centre Boom Is Running Into a Talent Wall
With capacity expected to more than double this decade, the industry is investing in training as graduates struggle to meet
This Firm Wants to be the ‘Next Big Disruptor’ in Networking
Arrcus positions itself as a horizontal software layer that can run across different types of networking hardware.
Will 2026 be the year of AI IPOs?
With CoreWeave’s listing and Fractal Analytics going for an IPO, an array of AI companies are now looking to raise
Fighting Deepfakes May Not Be a Technology Problem
Defenders must be active at all times, while attackers need only one opportunity.
India’s Data Centre Expansion Is Decentralising
Without compute buildup beyond metros, the next wave of digital adoption will be constrained
Download the easiest way to
stay informed
stay informed
