Why We Re-Tested Our Own Insights
In March 2026 we published twenty takeaways about AI bot behavior, drawn from 48 days of server logs on a single property. This is what a longer, later window did to them.
Bot research ages badly, and most of it never gets checked. We wanted to know how badly, so we ran the original list back against six months of logs across three properties and recorded a verdict for every insight. Seventeen could be re-tested. Eleven of them had moved inside five months.
The original article is still published exactly as it was: AI Bot Traffic Is Accelerating Fast, the 48-day study. Nothing there has been quietly edited. This piece is the audit of it.
The 20 Takeaways, Re-Tested Six Months Later
Superseded and shifted together account for 11 of the 17 insights, 65% of the list, in roughly five months. Each verdict describes what the crawler population did next, not a defect in the original measurement. Verdicts come from a later, longer window: 154,864 classified requests across three properties between February 1 and July 13, 2026, cross-referenced against a larger ecommerce property.
- 01Shifted
Your sitemap.xml just became more important. GPTBot and ClaudeBot both pushed sitemap consumption to a higher threshold in March 2026 and never looked back. If your sitemap is stale, incomplete, or missing language variants, AI crawlers will miss content.
Six months later as of 7/20/26Sitemap consumption climbed again and held there. ClaudeBot is now the dominant consumer by a wide margin, roughly 6x Googlebot, and both bots were already fetching sitemaps in February, so March marked an escalation rather than a start.
- 02Still holds
robots.txt is not universally respected. GPTBot and Meta-WebIndexer never check it. If your AI content strategy depends on robots.txt directives, know that two of the most active crawlers ignore them entirely.
Six months later as of 7/20/26GPTBot made 2 robots.txt requests against 6,872 content requests. Meta made zero across 3,169. Confirmed on both.
- 03Superseded
ChatGPT-User traffic is a direct signal of brand citation in AI conversations. Each request represents a real person pasting your URL into ChatGPT. This is measurable word-of-mouth, and it is growing fast.
Six months later as of 7/20/26Up 11% across five months with a dip in the middle: 3,827 in February against 4,236 in June. The signal is real, and the growth flattened after the window we measured.
- 04Still holds
AI bots crawl in bursts, not steady streams. GPTBot hit 114 req/min in a 3-minute window. If your server can't handle burst traffic, AI crawlers may get throttled or hit errors during their indexing runs.
Six months later as of 7/20/26Confirmed and intensified. Peak is now 270 requests a minute, with four bots above 130.
- 05Still holds
OpenAI and Anthropic each operate 3 separate bots. One for training/indexing, one for search, one for live user sessions. Blocking one does not block the others. Your robots.txt needs separate directives for each.
Six months later as of 7/20/26A floor rather than a ceiling. OpenAI now runs at least four, including an ads crawler that publishes its own IP range list. Three each is a floor, not a count.
- 06Superseded
OAI-SearchBot and Googlebot are the only bots that fetch images at volume. If your article images carry meaningful content (charts, diagrams, data visualizations), these are the bots that will use them in search results.
Six months later as of 7/20/26Six bots now fetch a higher share of images than Googlebot. The two named here rank 7th and 10th. Image fetching is common across the crawler population.
- 07Shifted
ChatGPT-User only extracts text. Zero images, zero CSS, zero JS. Your HTML content is what gets pulled into AI conversations. Structured, clear text matters more than visual design for AI visibility.
Six months later as of 7/20/26Close to it, but not zero any more: 1.9% images against 95.3% pages. The CSS and JavaScript half of it cannot be checked, because the log pipeline strips those files before storage.
- 08Shifted
AI crawlers peak at different hours. GPTBot hits at 04:00 UTC. Claude-SearchBot peaks overnight. PerplexityBot bursts at 23:00, 05:00, and 09:00. If you deploy site changes during off-peak US hours, AI bots may be the first to see them.
Six months later as of 7/20/26The pattern holds and every specific hour has moved. GPTBot now peaks at 17:00 UTC, and it is the most temporally concentrated crawler we measured, with 21.4% of its entire volume inside that one hour.
- 09Shifted
Meta is the most aggressive AI crawler by volume. Meta-WebIndexer sent more requests than any other bot in this dataset, with zero robots.txt checks. If you are not tracking Meta's crawlers, you are missing the biggest player.
Six months later as of 7/20/26This one ran ahead of the data when written and is catching up to it now. Across the full six months Meta ranks 12th. In July it is the single heaviest AI crawler on the property, after being completely absent until May.
- 10Superseded
llms.txt adoption is still theoretical. Zero AI bots requested /llms.txt across 48 days. It may become a standard eventually, but no crawler currently looks for it.
Six months later as of 7/20/26Since then it has started being fetched continuously, but overwhelmingly by scanners, auditors and research probes rather than the frontier labs. CCBot, ClaudeBot and Googlebot fetch it. GPTBot does not.
- 11Shifted
Applebot renders your pages fully. It fetches CSS, JS, and images (47% of its traffic). If your content requires JavaScript rendering to be complete, Applebot will see it, but most AI bots will not.
Six months later as of 7/20/26Directionally right, materially lower now: images are 27.6% of Applebot traffic, against 47% then. The CSS and JavaScript part is untestable here for the same pipeline reason as insight 9 above.
- 12Still holds
Technical, how-to content gets referenced most in AI conversations. The top ChatGPT-User pages were all implementation guides and technical explainers. Deep, specific content earns AI citations.
Six months later as of 7/20/26The most robust finding in the original list. All twelve top destinations are implementation guides, tool comparisons or technical analyses. Nothing generic, nothing promotional. A single migration guide drew 34% of all ChatGPT-User traffic to the network.
- 13Superseded
Bytespider and CCBot only check robots.txt and never crawl. They are consuming your robots.txt directives without following through. This may change, but currently they generate compliance overhead with zero content indexing.
Six months later as of 7/20/26Reversed since we measured. Bytespider is now the third-heaviest crawler on the property, and 94% of its 16,810 requests are content rather than compliance.
- 14Still holds
AI crawl volume can shift overnight. GPTBot went from 0 to 187 requests in a single week. Your crawl budget projections need to account for sudden step-changes, not gradual growth.
Six months later as of 7/20/26Confirmed, with a cleaner example than the one originally cited. Meta went from zero to 2,615 requests a month in about eight weeks.
- 15Superseded
IP analysis reveals bot identity. ChatGPT-User's near 1:1 IP-to-request ratio proves individual user sessions. GPTBot's 2 IPs prove centralized infrastructure. IP patterns help distinguish real user-triggered fetches from automated crawling.
Six months later as of 7/20/26The method survives and every number in it has moved. ChatGPT-User now runs 12.4 requests per IP across 1,588 addresses. GPTBot runs 129 IPs, not 2. The near-1:1 fingerprint described here now belongs to Claude-User at 1.4.
- 16Still holds
Coordinated crawl events happen across bot families. GPTBot and OAI-SearchBot fired simultaneously on March 19 from the same Microsoft infrastructure. When one OpenAI bot ramps up, expect the others to follow.
Six months later as of 7/20/26Independently confirmed on the exact date cited, one of only two days in six months where both bots cleared those thresholds together.
- 17Superseded
The bots you have never heard of are already visiting. PromptingBot, LinkupBot, Brightbot, Observer, and others are actively crawling content. The AI bot landscape is larger than the well-known names suggest.
Six months later as of 7/20/26All four named bots made zero requests across the new six months. The underlying point survives with an entirely different cast: the long tail is now dominated by llms.txt scanners and AI-readiness auditors. Any named list has a shelf life of months.
This data is representative of wislr.com only and should not be read as an overall industry trend. Our intent is to spark conversation, share data, and encourage brands to inspect this for themselves.
What Six Months of Logs Changed
The original study ran for 48 days on a single property. This re-test covers 154,864 classified requests across three properties between February 1 and July 13, 2026, cross-referenced against a larger ecommerce property carrying 7.26 million requests, with bot identity resolved by user-agent against a 75-pattern reference table.
The original 48-day study covered one site and said so. This re-test spans three properties we operate, labeled Site A, Site B and Site C, plus a larger comparison property. We do not map the labels to domains, so the six-month figures are reported without attribution.
Eleven of the seventeen insights we could re-test moved inside five months, six superseded by new crawler behavior and five shifted materially. That rate is the finding. The pattern underneath it matters more than any single correction: the behaviors held and the numbers moved. Bursts, coordinated vendor events, the bias toward technical content, all still there. Every specific bot name, ratio and peak hour we published has changed.
Meta went from absent to first in eight weeks
The single biggest change in the dataset. meta-externalagent recorded nothing at all until May, one solitary request that month, then 553 in June and 2,615 in the first thirteen days of July. It is now the heaviest AI crawler on the property, and July is a partial month, so that figure understates it.
Over the full six months Meta ranks twelfth. In July it ranks first. Both statements are true, which is exactly why a single-window ranking is a poor basis for a crawl-budget decision. It also made zero robots.txt requests across 3,169 hits.
The bot rankings churn, the behaviors do not
Fourteen bots with enough volume to track month by month. Watch how little the shape of the population resembles itself from February to July.
The IP fingerprint inverted
The original reading used requests-per-IP to tell user-triggered fetches apart from centralized crawling. The method still works. Every number in it has moved.
Image fetching is now common
When we measured, OAI-SearchBot and Googlebot were the only bots pulling images at volume. Image fetching has since spread across the crawler population: six bots now fetch a higher share than Googlebot, and the two originally named sit seventh and tenth.
llms.txt is measured, not consumed
When we measured, no bot had ever requested /llms.txt. That has changed completely, though not in the way the standard's advocates would want. It is fetched continuously, almost entirely by tools that exist to check whether you publish one.
Recognized large-vendor crawlers seen fetching it: CCBot, ClaudeBot, Googlebot and GoogleOther. Notably absent: GPTBot.
Bytespider does not just check robots.txt
This one did not drift, it reversed. Bytespider is now the third-heaviest crawler on the property and 94% of its requests are content.
What Held Up
Six of the original insights survived the longer window intact, and they are the ones describing behavior rather than naming a leader.
Technical content earns the citations. The most robust finding in the original list. All twelve top ChatGPT-User destinations are implementation guides, tool comparisons or technical analyses. Nothing generic, nothing promotional. A single migration guide drew 6,716 requests, 34% of all ChatGPT-User traffic across the network.
GPTBot still ignores robots.txt. Two robots.txt requests against 6,872 content requests. Meta made zero across 3,169.
Bursts got worse, not better. Peak rates are now well above the 114 requests a minute originally reported.
Vendors still run bot fleets, and we undercounted them. OpenAI operates at least four distinct crawlers, including an ads crawler that publishes its own IP range list. Anthropic runs ClaudeBot, Claude-User and Claude-SearchBot. Three each was a floor, not a count.
Coordinated events are real. Confirmed on the exact date the original post cited: March 19 shows GPTBot at 206 requests and OAI-SearchBot at 114, one of only two days in six months where both cleared those thresholds together.
Bots still peak at different hours, and every hour moved. GPTBot no longer peaks at 04:00 UTC. It now peaks at 17:00, and it is the most temporally concentrated crawler we measured: 21.4% of its entire six-month volume lands in that one hour of the day.
What We Could Not Test
Two apparent findings turned out to be artifacts of how the logs are collected. Both are worth stating, because each could have produced a confident headline the data does not support.
robots.txt is invisible on the comparison property. It returned zero robots.txt requests from any bot, which looks like a dramatic compliance finding and is not. That site serves robots.txt at the CDN edge, so the request never reaches the logged layer. Every robots.txt conclusion here comes from the primary property only.
The CSS and JavaScript findings cannot be checked either way with this data. The log sanitiser strips .css, .js, fonts and source maps before storage. Insights 9 and 13 can only be tested on their image components.
Two smaller cautions. The dataset spans three related properties rather than one, and the most-cited content sits on a single site within that network, which concentrates the citation findings. And July is a partial month, running to the 13th, so July figures understate: Meta reaches first place despite that.