AI Crawlers Also Reach You by Following Backlinks
A crawler-forensics hypothesis. Your server logs already hold a rough index of the third-party pages most likely feeding what AI says about your brand. You are almost certainly collecting the data and throwing the signal away.
Most people picture AI crawlers the way they picture Googlebot: it reads your sitemap, walks your internal links, and indexes your pages. That happens. But it is not the only way GPTBot and the others reach you.
They also arrive by following a backlink: a forum thread comparing you to a competitor, a review on a site you have never heard of, a roundup article that links to your product page. When an AI crawler follows one of those links, it happens close to real time and lands in your access logs with a timestamp.
The common way to find out which third-party pages are shaping AI answers is to prompt ChatGPT, Perplexity and the rest and scrape the sources out of the JSON they return. That works, and you should do it. But it looks at the problem from the outside, one prompt at a time. Your own logs let you look from the inside, continuously, on data nobody can rate-limit or take away.
Most Crawlers Will Not Tell You Where They Came From
A backlink-follow is only useful if the crawler tells you which page it followed, and that lives in one place: the Referer header the crawler sends with its request.
Almost none of the LLM crawlers send it. They fetch your page and record nothing about where they came from. You see the visit. You do not see the source.
So most of the crawl you care about is anonymous at the source. You can prove an AI bot visited. You cannot prove what sent it. One crawler closes that gap.
PetalBot Is the Exception
PetalBot is the crawler behind Huawei's Petal Search. It is not an AI lab's bot. But it does one thing almost nothing else does: on its requests, it reports the referring URL. The exact third-party page it followed to reach yours, query string and all.
In our logs, that behavior is not occasional. It is the rule.
One PetalBot log line closes the gap. A normal AI-crawler hit tells you a bot arrived. A PetalBot hit tells you a bot arrived and names the page that sent it.
203.0.113.x - - [20/Jul/2026:14:22:07 +0000] "GET /products/widget-pro HTTP/1.1" 200 18244 "https://forum.example.com/thread/best-widgets-2026?sort=top" "PetalBot"
We also rank PetalBot among the more polite crawlers we see, measured by peak requests per minute in our logs. It is not hammering your origin. There is little operational cost to leaving it alone, and a real signal cost to blocking it.
PetalBot and the LLM Crawlers Read the Same Web
The next step is where the method stops being observation and becomes inference.
PetalBot and the LLM crawlers are largely reading the same web. They follow the same links, from the same review sites and forums and comparison articles, because that is where the mentions of your brand live. So when PetalBot hands you a referring URL, it is pointing at a page the AI crawlers are plausibly reading too.
PetalBot reports its referrer and the LLM crawlers do not. Use what it reports as a stand-in for what they withhold.
The technique is triangulation. You cannot see the target directly, so you measure something that moves with it. Here, the measurable thing is PetalBot's referral trail, sitting in a log file you already own.
Build the Graph: From Raw Logs to a Citation Shortlist
None of this needs a new tool. It needs the logs you already collect and a few filtering steps.
Log at the edge, and keep the Referer header
Capture requests at the server or CDN, not in a JavaScript tag, because crawlers do not run the tag. The one field this whole method depends on is Referer. If your logging strips it, or truncates query strings, fix that first. You cannot recover a header you never stored.
Filter to PetalBot hits that carry a third-party referrer
Match the PetalBot user-agent, then keep only the hits whose referrer points at a domain that is not yours. Those off-site referrers are your citation candidates. Drop self-referrals and empty referrers; they are noise for this purpose.
Deduplicate to pages, then roll up to domains
The same forum thread will appear many times. Collapse repeated URLs into unique pages, then group pages under their domain. Now you have two views: the individual pages linking to you, and the sites those pages live on.
Weight by recency and recurrence
A referrer seen fifty times this week outranks one seen once last quarter. Weight each candidate by how often it recurs and how recently it was last walked. Recency is the important half: a page being crawled now is a page in circulation now.
Cross-reference against the LLM crawlers on your own pages
Line up the timing. When a PetalBot referral points at your product page, check whether GPTBot, ClaudeBot or the others hit that same page in the same window. Overlap in timing strengthens the case that the third-party page is part of what AI is currently reading about you.
Rank it, and treat it as a shortlist
What comes out is a ranked list of third-party pages and domains most likely feeding AI answers about your brand, built entirely from first-party data. Treat it as a lead list to investigate, not a proven citation set.
Steps 1 through 6 are exactly what WISLR.ai runs for you. It captures AI crawler and PetalBot activity at the edge, keeps the referrers, and turns them into a ranked, always-current feed of the third-party pages most likely shaping what AI says about your brand, cross-referenced against the LLM crawlers hitting your own site. No log wrangling required.
What the Graph Is Good For
A ranked feed of pages talking about you, refreshed continuously, is useful whether or not the AI-grounding hypothesis holds.
Correct misinformation at the source. If a page in your referral feed is saying something wrong about your brand, and it is being crawled often enough to show up, you now know exactly where to respond. You are no longer guessing which of a thousand sites matters. The crawlers told you which ones are in circulation.
Discover mentions you did not know existed. Most brand monitoring watches a list you built. This watches a list the web built, and hands you reviews, threads and comparison pages you were never tracking, straight from your own access logs.
What We Are Assuming, and What We Cannot See
Two assumptions sit under this method, and neither is confirmed. It is worth using anyway, but only if you know where it stops.
We can see PetalBot's path. We cannot see OpenAI's. The overlap is a reasonable bet, because both crawl the open web where brand mentions live, but it is a bet. Some of PetalBot's referrers will be pages no AI crawler ever reads, and some pages AI reads will never show up in PetalBot's trail.
It might. It might feed training instead. It might feed something we have not named. The log data cannot tell us which, and anyone who tells you it can is guessing with more confidence than the evidence allows. What the logs prove is narrow and real: a third-party page linked to you, and a crawler followed it.
It is not a measurement of AI citations. It is a first-party proxy for them, built cheaply from data you already collect. We have built exactly this for clients: a first-party dataset of third-party citation candidates, pulled from their own logs. What is speculative is the label we put on the graph, not the graph itself.
We can build your citation-candidate feed from your own logs.
WISLR reads AI crawler behavior at the edge, per URL and per day, on your first-party data. That includes pulling PetalBot's referral trail into a ranked feed of the third-party pages most likely shaping what AI says about your brand, and cross-referencing it against the LLM crawlers hitting your site.