A team pulls a report showing forty thousand visits from OpenAI last month and starts planning a budget around it. Almost all of it was a crawler. There were perhaps two hundred humans.
This confusion is everywhere, it is easy to make, and it corrupts everything downstream — traffic figures, conversion rates, content decisions, and any argument you make internally about whether AI is worth investing in. The fix is a taxonomy, and the taxonomy has four categories rather than the two most people assume.
The four categories
AI traffic taxonomy
Only the last row is a person. The first three are software, with very different purposes.
| Category | What it is | Human present? | Can convert? |
|---|---|---|---|
| Training crawler | Bulk collection of content for model training | No | No |
| Search indexer | Building an index to retrieve from at answer time | No | No — but affects whether you are cited |
| User-triggered fetch | A single page fetched because someone asked about it | Indirectly | No, but signals real interest |
| Human referral | A person clicking a link in an assistant's answer | Yes | Yes |
Training crawlers
Bulk collection to build training corpora. GPTBot is the well-known example; most major labs run an equivalent.
These are high-volume, breadth-first, and completely disconnected from any individual person. A spike in training crawler traffic tells you a crawl is running. It tells you nothing about demand, interest, or revenue.
The robots.txt decision here is a genuine business judgement with arguments on both sides — content control versus presence in future models — and it is separate from every other question in this post.
Search indexers
A different job from training. These crawlers build a retrieval index that the assistant queries at answer time to ground responses in current information. OAI-SearchBot is OpenAI's; other vendors run their own.
The distinction matters commercially. Blocking a training crawler removes your content from a future model's weights. Blocking a search indexer removes you from the assistant's ability to cite you today. Many organisations that thought carefully about the first decision applied the same rule to the second without noticing they are different questions with different consequences.
Check what your robots.txt actually blocks
Wildcard rules and copied-in blocklists routinely catch search indexers alongside training crawlers. If you want content control but still want to be citable in AI answers, those tokens must be handled separately. This is the most common self-inflicted AI visibility problem there is, and it is usually discovered months later.
User-triggered fetches
The interesting category, and the one most taxonomies miss.
When someone pastes a URL into an assistant and asks "what does this page say", the assistant fetches that page right now, because that specific person asked. Vendors generally use a distinct user agent for this — names in the ChatGPT-User, Claude-User, Perplexity-User family — precisely because it is a different kind of request from a bulk crawl.
There is no browser and no session. There is a human, one step removed, expressing interest in one specific page.
This is not a visit and should never appear in session metrics. But it is also not noise. A page receiving a lot of user-triggered fetches is a page people are actively asking assistants about, which is a demand signal you cannot get any other way. It deserves its own count, on its own dashboard, labelled as what it is.
Human referrals
A person read an answer, saw your link, and clicked it. This is the only category that can browse your site, evaluate your product, and pay you.
It is also the hardest of the four to detect, because unlike the three above it does not announce itself in a user agent. This is where the entire referrer problem lives.
There is an unfortunate symmetry: the three categories that cannot buy anything are the easy ones to identify, and the one that can is the hard one.
The bots identify themselves. The humans don't. Every measurement problem in this field follows from that one asymmetry.
What conflation actually breaks
Four specific damages, in rough order of how expensive they are.
Conversion rate collapses. If crawler requests are counted as sessions, they land in the denominator of every rate metric. Crawlers convert at exactly zero. A page that genuinely converts at 4% will report 0.4% if crawlers outnumber humans nine to one, which for many pages they do. Teams then "fix" pages that were never broken.
Traffic reporting inflates. The forty-thousand-visits report. Someone makes a decision on it, and the decision is based on a number that includes no people.
Content decisions invert. Crawlers crawl breadth-first — everything, including pages nobody reads. Humans arrive concentrated on pages that answer questions. Mix them and your "most visited pages" report ranks by crawl coverage rather than human interest, which will point you at exactly the wrong content to invest in.
Cost attribution goes wrong. Bandwidth and origin load from crawlers get charged against a marketing channel that did not cause them.
Illustrative shape of AI-related requests to a content site
A worked illustration of typical proportions, not measured data. The exact mix varies enormously by site — the point is the order-of-magnitude gap between crawler volume and human referrals.
Illustrative. Measure your own distribution before drawing conclusions.
The proportions above are illustrative, but the shape is the point: the category you care about commercially is usually the smallest by request count and the largest by value.
Separating them in practice
Three mechanisms, in increasing order of reliability.
User agent matching. The obvious first step. Bot user agents are documented and reasonably stable, and vendors generally publish them.
It is also unauthenticated. Anyone can send any user agent string, so this identifies cooperative bots — which, for the major labs, is most of the volume. It is a starting point, not a security control.
Reverse DNS and published IP ranges. The traditional verification layer: check that the request actually originates from infrastructure the operator controls. It works, it requires the operator to publish and maintain ranges, and it puts a lookup on your request path.
Cryptographic signatures. The direction the ecosystem is moving. An agent signs its request with a key published under a domain you can verify, and you check the signature. This is the only mechanism in the list that cannot be forged, and it is worth understanding properly.
Classifying an incoming AI-related request
Each stage answers a different question. Only the last one is about a person.
Is it signed?
A verified signature identifies the operator cryptographically. Strongest available evidence.
Does it claim to be a bot?
A documented bot user agent, ideally corroborated by reverse DNS or a published IP range.
Which kind of bot?
Training crawler, search indexer, or user-triggered fetch. Different agents, different meaning.
Otherwise: a human
No bot signal means a browser, and the source has to be inferred from everything else the request carries.
How to report them
Four categories, four places, no blending:
Crawlers get an infrastructure dashboard. Request volume, bandwidth, crawl coverage, error rates. This is an operations concern. It does not belong in a marketing report at all.
Search indexers get a visibility dashboard. Which pages are being indexed, how often, and whether anything important is being missed. This is the closest thing to a coverage report the AI ecosystem offers, and it is genuinely actionable.
User-triggered fetches get a demand dashboard. Which pages people are asking assistants about. Rank it. There is no other way to see this, and it is a real signal about which of your pages answer questions people are actually asking.
Human referrals get the revenue dashboard. Sessions, behaviour, conversions, attributed revenue. This is the only category that belongs in an acquisition report, and it is the only one where attribution is even a meaningful concept.
The number to check first
Take your reported AI traffic figure and ask which of the four categories it contains. If nobody knows, it contains all of them, and it is not a traffic number — it is a request count with a marketing label. That single question resolves most disagreements about whether AI is "working" for a given site.
Why the confusion persists
Partly naming: everything in this space has "AI" in the name, and the vendors are the same. GPTBot and a ChatGPT referral share a brand and share nothing else.
Partly tooling: most analytics tools have a binary bot filter, one checkbox, no taxonomy. A tool that offers "exclude bots" as a single toggle is implicitly claiming these four things are two things.
And partly incentive. A big number is easier to present than a small one. "Forty thousand AI visits" makes a better slide than "two hundred AI referrals, converting at 6%" — even though the second is the sentence that should change what anyone does next.
Frequently asked
An AI crawler is software fetching your pages — either to collect training data or to build a search index the assistant retrieves from at answer time. An AI referral is a person who read an assistant's answer, saw your link, and clicked it. Only the referral is a potential customer. Crawlers identify themselves in the user agent; human referrals usually carry no identifying signal at all.
GPTBot is a bulk training crawler operating independently of any individual person. ChatGPT-User is a user-triggered fetch: a single page retrieved because a specific person asked ChatGPT about that specific URL. Neither is a browser session, but the second is a real demand signal about one page, and it deserves its own counter rather than being lumped in with either crawls or sessions.
That is a business decision about content control, and it should be made separately for training crawlers and search indexers. Blocking a training crawler keeps your content out of future model weights. Blocking a search indexer removes your ability to be cited in AI answers today. Wildcard rules and copied blocklists frequently catch both when only the first was intended.
Crawler requests enter the denominator of every rate metric while converting at exactly zero, so conversion rates collapse. They also crawl breadth-first, which distorts your most-visited-pages report toward crawl coverage rather than human interest — pointing content investment at exactly the wrong pages.
Adequate for cooperative bots, which is most of the volume from major AI labs, and useless as a security control since anyone can send any string. Reverse DNS or published IP ranges add real verification. Cryptographic request signatures under RFC 9421 are the only mechanism that cannot be forged, and adoption is growing.
Sources & further reading
- 01Overview of OpenAI crawlers and user agents — OpenAI
- 02Anthropic crawler and user agent documentation — Anthropic Support
- 03PerplexityBot and Perplexity-User — Perplexity
- 04Google crawlers and fetchers overview — Google Search Central
- 05RFC 9421: HTTP Message Signatures — IETF