Counting AI Crawls: What a Crawl Quota Actually Measures

Crawl allowances have started to appear on analytics pricing pages. One tool includes 100,000 a month, another 650,000, a third pauses at its limit and sells more by the million. The figures look comparable. They are not, because "one crawl" is defined differently from tool to tool, and so is what happens when you run out.

It matters more than it would for page views, because crawl volume is traffic you do not control. A training crawler that decides to re-read your documentation this week moves your count without you shipping anything. This post defines a crawl hit precisely, explains why verified and unverified hits are counted the same way, and sets out the four things that can happen when an allowance runs out.

In Traceten, a crawl hit is one request to your server from a user agent that matches a known AI crawler, reported by a small piece of middleware running on that server. Browser-based analytics cannot do this, because a crawler never runs your tracking script, so the count starts where the request arrives.

Several filters run before anything is counted, and they decide most of what a quota means:

• Methods. Only GET and HEAD requests are considered by default.
• Assets. Requests for scripts, stylesheets, images, fonts and similar files are skipped, as are API and build paths. A crawler fetching your page and its forty assets is one hit, not forty-one.
• Crawler-facing files. robots.txt, llms.txt and sitemaps are always counted, even though a naive text-file rule would skip them. A GPTBot request for llms.txt is one of the most informative events you can log.
• Anchored matching. The user agent must contain the crawler's token at a word boundary, so GPTBot/1.1 matches and a string that merely contains those letters inside another word does not.

You also choose which categories to report: answer fetchers such as ChatGPT-User and Claude-User, search indexers such as OAI-SearchBot and PerplexityBot, training crawlers such as GPTBot and ClaudeBot, and uncategorised others. Switching off a category you have no interest in is the most effective way to keep a count, and a bill, to what you care about.

A user agent is a claim that anyone can type. Verification is how you test the claim, and it is a ladder with several rungs because vendors publish different things.

Every reported hit counts toward the allowance, whichever row it lands in. That can look unfair for spoofed requests until you consider the alternative. Receiving and checking a hit costs the same whether it turns out real or fake. And a sudden wave of requests calling themselves GPTBot from addresses OpenAI does not publish is not noise to be filtered away. It is a finding, and it is stored and shown rather than dropped.

Where the allowance applies changes what a figure means. Traceten's crawl allowance belongs to the account, pooled across all of its sites, and resets with each billing period: 100,000 crawl hits on Starter, 650,000 on Growth and 1,500,000 on Pro. Crawl hits never draw on the event allowance, and events never draw on crawl hits, so a crawler rereading your blog cannot use up the allowance your visitors need.

DataFast's documentation describes a similar shape with a different size: 100,000 accepted requests per account, reset with the billing cycle. The word "accepted" is doing work in that sentence, which is the general point. Before comparing two allowances, find the sentence that defines one unit.

This is where tools differ most, and where the pricing page is least likely to say.

Three of the four keep your crawl totals exact. What varies is how much per-crawl detail survives and whether the excess is billed. Only the free plan genuinely stops counting, because a plan with no revenue behind it cannot carry unbounded crawler traffic. In every state, the middleware's report is accepted in exactly the same way, so nothing about your allowance ever reaches your server's responses or the crawler.

DataFast documents a different contract: reaching its allowance pauses bot traffic ingestion while normal web analytics keep working, and extra capacity costs $9 a month for each additional million requests. Pausing, degrading, metering and capping are all reasonable designs. They answer different questions about what you would rather lose when the crawlers get busy: the counts, the detail or the money.

Five questions turn a crawl allowance into something you can compare.

1. What is one unit? A request, an accepted request, a verified request, or a page?
2. Which requests are skipped before counting? Assets, non-GET methods, crawler-facing files?
3. Per site or per account, and when does it reset? Calendar month or billing cycle?
4. Do unverified and spoofed requests count? And are they shown to you?
5. What happens past the limit? Paused, degraded, metered or capped, and what exactly is lost in each?

Then estimate your own volume before choosing a plan. Filter a month of your server's access log to the AI crawler user agents you care about, excluding asset requests, and you will have a figure close to what any server-side crawl meter will count. It is usually larger than people expect, and it moves with the crawlers' schedules rather than with anything you do.