Fetcher bots, on the other hand, they said, “access website content in response to user actions. For example, when a user requests up-to-date information on a specific topic, a fetcher bot retrieves the relevant page in real time. They are also used to help surface website links that match a user’s search query, directing them to the most relevant content. Crawler bots contribute nearly 80% of the total AI bot request volume, with fetcher bots making up the remaining 20%.”
‘Massive surge in AI bot traffic’
Real-time fetching by AI bots, the report states, is a greater challenge than peak crawler rate. Analysis revealed that in one case, a single crawler reached a peak of around 1,000 requests per minute.
Real-time fetching, on the other hand, is significantly more aggressive: in one instance, a fetcher bot made 39,000 requests per minute to a single website at peak load. “This traffic volume can overwhelm origin servers, strain server resources, consumer bandwidth, and cause expensive DDoS-like effects even without malicious intent,” the report noted.
