Ask an AI assistant the same question twice and you will not reliably get the same answer. Ask it on Tuesday and again on Thursday and the difference is larger. This is not a defect in the assistants; it is what a sampling process does. It does mean that any number describing how often you appear in AI answers is an estimate from a sample, and an estimate from a sample without an interval attached is a claim the data does not support.
Here is the number that should end the argument about whether this matters. A presence rate of 20% measured over five runs has a 95% confidence interval running from 3.6% to 62.4%. The point estimate is 20%. The truth is somewhere in a range that spans nearly the whole scale. Reporting "20%" and nothing else is not a simplification, it is a fabrication of precision that does not exist.
Why small samples are the normal case, not the edge case
It would be convenient if the small-sample problem only showed up in a corner of the product. It does not. It is the middle of the distribution.
An answer engine is expensive to poll. Every run is a paid API call, and the number of runs available to you is prompts multiplied by engines multiplied by repetitions multiplied by scans. Even a generous configuration produces tens of observations per prompt in a month, not thousands. Meanwhile the rates being measured cluster near the ends of the scale (a new site is named in almost nothing, an established one in almost everything), and near the ends is exactly where the naive interval breaks down.
So the awkward case is the ordinary case, and any product in this area has to be built for it rather than around it.
The same 20% rate, at four sample sizes
95% Wilson intervals. The point estimate is identical in every row. What changes is how much you are entitled to conclude from it.
presence rate
Wilson, and why not the one from the textbook
The interval every tutorial shows is the normal approximation, usually called Wald: take the rate, add and subtract z × sqrt(p(1-p)/n), done.
It is unusable here, and it fails in precisely the region this product lives in.
At a rate of zero, Wald returns [0, 0]. The interval has zero width. From three runs that happened not to mention you, it announces 0% presence with no uncertainty whatsoever: a statement of perfect confidence derived from almost no evidence. It does the same thing at a rate of one, claiming certainty of universal presence from a handful of lucky runs. It also cheerfully produces bounds below 0% and above 100%, which are not rates.
The Wilson score interval does not have these failures. It stays inside [0, 1] by construction, it stays wide when n is small, and it does not degenerate at the ends. Zero out of five runs gives an interval of 0% to 43.4%, which is the honest answer, and reads as "we have barely looked" rather than "you are invisible."
What zero out of five actually licenses you to say
Not "AI assistants never mention us." Something closer to "our presence rate is somewhere under about 43%, and we need more runs before saying anything sharper." The second sentence is less satisfying and is the only one the evidence supports.
There is a smaller detail here that is worth mentioning because it is the kind of thing that only shows up in production. At a rate of exactly zero the algebra leaves the lower bound at roughly 1.4 × 10⁻¹⁷ rather than exactly zero: floating-point residue, not a real quantity. If you then compare that residue against another interval to decide whether a change is significant, you get comparisons that are technically correct and semantically nonsense. The bound has to be snapped to zero, not merely clamped. Documented behaviour and actual behaviour differ by one epsilon, and one epsilon is enough.
The rule that makes the chart worth reading
Here is where the interval stops being a statistical nicety and becomes a product decision.
A change is only rendered as a change when the two intervals do not overlap. Everything else is drawn flat, with the band visible.
Consider a move from 3-of-10 to 4-of-10. As point estimates that is 30% to 40%, a 33% relative improvement, and it will be reported as one by any tool comparing point estimates. The intervals:
- 3 of 10: 10.8% to 60.3%
- 4 of 10: 16.8% to 68.7%
These overlap across almost their entire length. One additional mention out of ten runs is indistinguishable from noise, and a dashboard that flags it as a win is training you to react to randomness.
The cost of this rule is that fewer things get flagged, and some real improvements will sit un-flagged until the evidence accumulates. That is the correct trade. A dashboard that cries wolf on noise is one nobody is reading by the time something real happens, and the failure mode of the alternative (a marketing team celebrating a coin flip, then explaining a reversal next month) is worse than a delay.
Two ways to compare two periods
Same data, two rules. The left column is what most reporting does. The right column is what the data supports.
| Observation | Point-estimate reading | Interval reading | Which one survives a month |
|---|---|---|---|
| 3 of 10 → 4 of 10 | Up 33%. Ship the case study. | Intervals overlap almost entirely. Flat. | Flat. The next period will be 2 of 10 as often as 5. |
| 10 of 50 → 25 of 50 | Up 150%. | 11.2%–33.0% versus 36.6%–63.4%. No overlap. Real. | Real. This one is worth a case study. |
| 0 of 5 → 1 of 5 | Infinite improvement, from nothing. | 0%–43.4% versus 3.6%–62.4%. Overlapping. | Flat. You have five observations. |
| 20 of 100 → 26 of 100 | Up 30%. | 13.3%–28.9% versus 18.4%–35.4%. Overlapping. | Flat, and this is the painful one to accept. |
That last row is the one people argue about, and it is the honest case. Six more mentions out of a hundred runs is a real-feeling move and the evidence does not separate it from chance. Run more, or wait.
Gating on sample size, on purpose
The consequence of taking all this seriously is that some accounts should not be shown a trended percentage at all, and the product has to be willing to say so.
Scores are gated on sample size, and your plan sets how large a sample you get. At the smallest configuration a percentage is genuinely meaningless (with a single prompt there is no rate worth plotting), and the useful artefact is the answers themselves: here, verbatim, is what an answer engine says about you. At the largest, there is enough evidence for a trended score with an interval that is narrow enough to act on. In between there is a middle ground where per-prompt counts and the competitor leaderboard are defensible and a trended percentage is not.
Two rules make this bearable rather than annoying.
The page always states which of the two reasons applies. "Your plan does not include the trend" and "your plan includes it but too few of your prompts have been scanned yet" are different sentences with different fixes, and collapsing them into one greyed-out panel wastes the customer's time.
A scan below a minimum number of repetitions is refused outright. Below three repetitions every downstream interval is so wide that the resulting number is not a measurement at any confidence level. There is no value in running it, so it is not offered.
We would rather show you nothing than a percentage that moves on noise.
The x-axis is the scan window, not the day
One more piece of arithmetic hygiene, and it is the one most likely to be got wrong by a tool that treats visibility as a time series like any other.
A scan does not fire every call at once. It trickles across its cadence window, partly to spread cost, partly because hammering four engines with a hundred prompts in one minute is a good way to get rate-limited into a gap in your own data.
Plot the results by calendar day and you draw a sawtooth. Days a trickle happened to touch show a rate; days it did not show nothing, or worse, show a rate computed from two runs. The chart reads as violent week-to-week swings and every one of them is an artefact of scheduling.
So each point on the visibility chart is one scan window ("the ten days ending 2 September"), and the bucket boundaries come from the scans that actually ran rather than from a fixed calendar grid. The x-axis is irregular because the underlying process is irregular, and forcing it onto a regular grid would import noise that is not in the data.
95%
Confidence level, fixed. A configurable level would make two screens incomparable, and comparability is the point of a series
not a setting
3
Minimum repetitions for a scan. Below this every downstream interval is too wide for the number to mean anything
refused, not warned
n = 0 → null
No runs returns nothing, not zero. A rate of zero is a measurement; the absence of runs is not
they must never collapse
How to talk about these numbers
Three habits, and they are the difference between a visibility programme that survives scrutiny and one that gets quietly defunded after the first reversal.
Quote the fraction, not just the rate. "Named in 8 of 40 runs" travels better than "20% presence," because the reader can immediately judge how much to believe it. This is the same discipline as publishing the denominator on every funnel rate, and it fails the same way when skipped.
Never compare across different sample sizes without saying so. A 20% rate from last month's 40 runs and this month's 120 runs are not the same kind of object, even before the intervals are drawn.
Treat a flat rendering as information. When the product declines to flag a change, it is telling you something: your evidence does not yet separate this from chance. The right response is more runs or more time, not a different tool that will happily flag it.
None of this makes the measurement weaker. It makes it a measurement, and the moment somebody senior asks how confident you are, the range is the only answer that does not end the conversation badly.
Frequently asked
Because it is an estimate from a sample of engine runs, and the range is the 95% confidence interval around it. At five runs a 20% presence rate spans roughly 3.6% to 62.4%, so quoting the 20% alone would claim a precision the data does not contain. The range narrows as runs accumulate: the same 20% over 100 runs spans 13.3% to 28.9%.
It is an interval for a proportion that stays inside 0 to 100% and stays wide when the sample is small. The usual textbook alternative, the normal approximation, returns a zero-width interval at a rate of exactly zero (announcing 0% presence with total certainty from three runs) and can produce bounds below zero. Visibility rates cluster near the ends of the scale at small sample sizes, which is exactly where that approximation fails, so it is the wrong tool for this measurement.
Because the two confidence intervals overlap, so the move is not distinguishable from noise. Going from 3 of 10 runs to 4 of 10 reads as a 33% improvement on point estimates, but the intervals are 10.8%–60.3% and 16.8%–68.7% and overlap across almost their whole length. Flagging that would train you to react to randomness, so it renders flat with the band visible.
It depends on the rate you are measuring and the size of change you want to detect, but the intervals give you the shape: five runs tell you almost nothing, twenty narrow a 20% rate to roughly 8% to 42%, and a hundred narrow it to roughly 13% to 29%. Detecting a ten-point change reliably takes well over a hundred runs. A scan below three repetitions is refused outright, because every interval it could produce is too wide to be a measurement.
Because scans trickle across their cadence window rather than firing every engine call at once. Bucketing by calendar day would draw a sawtooth reflecting which days a trickle happened to touch, and read as violent week-to-week swings that are entirely scheduling artefacts. Each point is one scan window instead, with boundaries taken from the scans that actually ran.
Sources & further reading
- 01Binomial proportion confidence interval, Wikipedia
- 02Interval Estimation for a Binomial Proportion, Brown, Cai and DasGupta, Statistical Science 16(2), 2001
- 03Sonar API reference, Perplexity
- 04GEO: Generative Engine Optimization, Aggarwal et al., KDD 2024 (arXiv:2311.09735)