Who Are These Bots Fool'in?

Microsoft's Bingbot

Bingbot fetched it dozens of times. The real GPTBot, ClaudeBot, PerplexityBot and Gemini crawlers fetched it zero times, despite making thousands of requests to the same site. AI crawlers overwhelmingly discover content by following links, not by reading sitemaps. Submitting a sitemap is a search engine ritual; it does very little for AI visibility, as far as we know.

A robots.txt token controlling whether crawled content trains and grounds Gemini

It never makes a request. Google states that Google-Extended has no separate user agent of its own, so crawling still happens through the normal Google crawlers and the token only governs what happens to that content afterwards: training future Gemini models, and grounding answers in Gemini Apps and Vertex AI. Google is also explicit that it does not affect a site's inclusion in Google Search and is not a ranking signal, so blocking it costs nothing in Search and will not reduce your crawl traffic by a single hit.

Reverse DNS plus the vendor's published IP ranges

User-agent strings are free text. Anyone can send any of them, and plenty of scanners do.

ChatGPT-User fetching a page because a human asked about it

User-initiated fetches outnumbered training crawls by roughly six to one. Most AI bot traffic is not training, it is a real person's question being answered in real time.

A spoofed user-agent

We caught exactly this: one scanner on a single Google Cloud network sent well over a thousand requests in three days, rotating between OpenAI, Perplexity and Gemini user-agents, hitting one URL per path. That is enumeration, not crawling. Real OpenAI crawlers egress from Azure.

Each company runs several distinct bots with different jobs

OpenAI's own documentation lists four: GPTBot for training data, ChatGPT-User for a live fetch when a human asks, OAI-SearchBot for ChatGPT's search results, and OAI-AdsBot for checking pages submitted as ads. Anthropic and the others follow the same pattern. OpenAI states each setting is independent, so a publisher can allow OAI-SearchBot to appear in search while disallowing GPTBot. That means "should I allow AI bots?" is not one decision, it is four.

ChatGPT-User and OAI-SearchBot

Blocking one token blocks one bot. OpenAI states each setting is independent, so the search crawler, the ads crawler and the live user fetch all carry on. ChatGPT-User is the awkward one: because those requests are initiated by a person rather than by a crawl schedule, OpenAI notes robots.txt rules may not apply to it at all. Most "we blocked AI" configurations stop training and leave the bots that actually drive user-visible answers completely untouched.

Some crawlers brute-force guess sitemap filenames instead of reading the Sitemap directive

Meta's crawler does this. A correct Sitemap line in robots.txt is no guarantee a crawler will use it.

Many AI clients send no referrer, so the visit is recorded as direct

Referrer data from AI assistants is patchy by design. Desktop and mobile apps frequently pass no referer header at all, in-app browsers behave inconsistently, and some clients send only a bare domain. The traffic lands in direct alongside bookmarks and typed URLs. Referral-side numbers are a floor, not a measurement.

Search is still around 20 to 25 times larger

Google plus Bing made well over a hundred thousand requests in three days; all verified AI crawlers combined made a few thousand. AI crawling is growing and real, but it has not displaced search crawling, and headlines suggesting otherwise are usually counting spoofed user-agents.