A visitor arrives on your pricing page. No referrer. No UTM. Nothing in the request says where they came from, and your analytics files them under Direct alongside the people who typed your domain from memory.
Four weeks earlier, a search-index crawler fetched that page. Two weeks after that, an assistant started citing it in answers to a question your product happens to solve. Today somebody read one of those answers and clicked.
Three events, one causal chain, spread across five weeks, and the only one your analytics recorded is the last, stripped of every clue about the first two. This is the central measurement problem of AI search, and it is not a tracking problem. It is a latency problem.
The chain, stated properly
How one AI-referred visit is actually produced
Stage 3 is the only one a browser analytics tool observes, and by then every identifier connecting it to stages 1 and 2 has been discarded.
Crawl
A search-index crawler fetches the page. Observable server-side, at the moment it happens, with an exact timestamp.
Index
The content enters the assistant's retrieval corpus. Not observable from outside. Duration unknown and variable.
Citation
The page starts being surfaced in answers to particular questions. Observable only by asking the questions yourself.
Click
A human follows the link. Observable, but usually with no referrer and no campaign parameters.
Every attribution tool ever built assumes stages 3 and 4 are effectively the same moment: someone sees your link somewhere, clicks it, and the click carries a header saying where "somewhere" was. Search works this way. Social works this way. Email works this way.
AI answers break the assumption twice. The gap between the referring content being produced and being acted on can be weeks. And the handoff carries no referrer for a large share of assistants, for reasons that are entirely by design.
Why the lag is not a detail
If crawl and click were minutes apart, none of this would matter much: you would treat crawl volume as a proxy for imminent traffic and move on. The lag is what makes the naive readings wrong.
It breaks the before-and-after test. You publish a page. Nothing happens for a month. The obvious conclusion (the page did not work) is the wrong one if your crawl-to-citation lag is five weeks. Teams kill content on the basis of a measurement window shorter than the mechanism they are measuring.
It inverts the diagnosis. AI traffic drops this month. Was it a ranking change, a competitor, an assistant's model update? Frequently the real answer is that the crawl fell off six weeks ago (a site migration, a robots.txt edit, a run of 503s during a deploy), and you are watching the delayed consequence of an incident that already ended.
It makes crawl data a leading indicator, and referral data a lagging one. Which is exactly backwards from how most teams weight them: they watch the referral chart daily and have never once looked at the crawl.
The reframe
Referral traffic tells you what your site looked like a month ago. Crawl coverage tells you what it will look like a month from now. Only one of those is still actionable.
What the lag looks like
There is no published constant here, and anyone quoting one is guessing. The lag depends on the assistant, how often it recrawls your site, how competitive the question is, and how much the answer for that question has already settled around other sources.
What is worth internalising is the shape: the stages compound, and the variance is enormous.
Where the elapsed time goes
An illustrative decomposition, not a benchmark. The purpose is to show which stages are observable and which are not. Measure your own with the method below.
| Stage | Typical order of magnitude | Can you observe it? |
|---|---|---|
| Publish → first crawl | Hours to weeks, driven by your crawl cadence | Yes, server-side, exactly |
| Crawl → available for retrieval | Days to weeks, entirely operator-controlled | No |
| Available → actually cited | Highly variable; competitive questions settle slowly | Only by polling prompts yourself |
| Cited → the click you record | Immediate, but usually referrer-less | Yes, minus the provenance |
Two of the four stages are dark from outside. That is not a reason to give up on the measurement; it is a reason to anchor it on the two you can see, and to treat the interval between them as the quantity of interest.
Measuring your own lag
The method is a per-URL join between two datasets your stack probably keeps in different places, or, more often, keeps only one of.
Left side: crawl events. Path, timestamp, declared agent, resolved category, verification status. This comes from server-side middleware or log parsing; browser analytics cannot produce it. Only search-index crawlers belong in this join: OAI-SearchBot, PerplexityBot and their peers. A training crawler hit tells you nothing about being citable today, and mixing them in will smear your distribution beyond usefulness.
Right side: AI-referred sessions. Landing path, timestamp, detected source. For assistants that send a referrer this is easy. For the ones that do not, you are relying on whatever detection you have, and honest reporting means carrying a confidence score rather than pretending the inference is a fact.
The join: for each landing page that received AI traffic, find the most recent search-index crawl of that same path preceding the first AI session. The interval is one observation. A few hundred observations give you a distribution.
p50
Median crawl-to-first-AI-visit interval. Your planning number
set review windows to at least this
p90
The tail. How long before you can call a page a failure
kill content after this, not before
Coverage
Share of priority URLs crawled by a search indexer in 30 days
uncrawled means uncitable
Three caveats worth stating out loud, because a chart of this will look more certain than it is.
The join is correlational. A crawl preceding a visit does not prove it caused it. What makes it useful is aggregate consistency: if the interval clusters tightly across hundreds of pages, you have found a real property of your site's relationship with an assistant. If it is uniform noise, you have found that you cannot yet measure this and should say so.
Pages get crawled repeatedly, so "the crawl that mattered" is a choice. Most recent preceding crawl is the defensible default; first-ever crawl systematically overstates the lag on long-lived pages.
And the whole thing depends on detecting the AI-referred session correctly in the first place. Which is a hard problem with an honest error rate, not a solved one.
What the distribution changes
Your content review window. If p90 is seven weeks, then a four-week performance review of new content is measuring nothing but noise, and every decision made from it is arbitrary. Set the window from the data.
Your incident response. Crawl coverage becomes a monitorable metric with a real alert condition. Search-index crawls of your priority URLs falling to zero is an incident now, even though the traffic consequence will not appear for a month. Nobody alerts on this, because nobody is collecting it.
Your migration planning. A replatform that breaks crawler access shows up in revenue two months later, after the team has moved on and the obvious causes have been ruled out. With crawl data the regression is visible the same day.
Your reporting language. "AI traffic is down 30%" invites a scramble. "Search-index crawls of our top 50 pages fell 60% five weeks ago, and this is the arrival of that" is a diagnosis with a fix attached.
The chain is not unobservable. It is observed at both ends, by two systems that have never been asked to talk to each other.
The practical sequence
- Start collecting crawl events now. The distribution needs weeks of history, so the first day you can answer this question is weeks after the day you start. There is no shortcut and no backfill.
- Separate search-index crawls from training crawls at collection time. Merging them destroys the signal and the fix is not retroactive.
- Track crawl coverage of your top pages weekly. This alone catches migration and
robots.txtregressions long before revenue does. - Join to AI sessions per landing URL once you have both sides, and compute the p50 and p90.
- Rewrite your content review cadence around the p90 rather than around the calendar.
None of this makes an anonymous visit suddenly announce where it came from. What it does is give you a documented lineage for a page (crawled on this date, by this verified agent, first produced AI traffic this many days later), which is a great deal more than Direct.
Frequently asked
Several assistants apply a referrer policy that strips the header on outbound links, some open links in contexts that never send one, and a large share of AI-influenced visits are not clicks at all: someone read an answer, then typed your name into a browser later. The result is that a majority of AI-referred visits arrive with no referrer and land in the Direct bucket of standard analytics.
There is no published constant, and any single number quoted for it is a guess. The interval is the sum of four stages (publish to crawl, crawl to indexed, indexed to actually cited, and cited to clicked), of which only the first and last are observable from outside. Measure it for your own site by joining search-index crawl timestamps to first AI-referred session per URL and taking the median.
The share of your commercially important URLs that a search-index crawler has fetched within a recent window, typically thirty days. It matters because a page that has never been fetched by a retrieval crawler cannot be cited in an answer, no matter how good it is. Coverage is the earliest measurable point in the chain that ends in AI-referred revenue.
Not as a direct causal claim, and you should not present it as one. What you can build is a defensible lineage: this URL was crawled by a verified search-index agent on this date, first produced detected AI-referred sessions this many days later, and those sessions converted at this rate. That is correlational evidence, stated as such, and it is considerably more than the Direct bucket offers.
Yes, and the separation has to happen at collection time because it cannot be reconstructed later. A training crawler hit means your content may inform a future model, with no near-term traffic implication. A search-index crawler hit means you are becoming eligible for citation now. Blending them into one 'AI crawls' number produces a metric that moves for reasons you cannot act on.
Sources & further reading
- 01Overview of OpenAI crawlers and user agents, OpenAI
- 02PerplexityBot and Perplexity-User, Perplexity
- 03Referrer-Policy, MDN Web Docs
- 04AI Insights: crawl and referral traffic, Cloudflare Radar
- 05Google crawlers and fetchers overview, Google Search Central