Anyone Can Claim to Be ClaudeBot: A Verification Runbook

Send this from any machine on the internet:

curl -A "ClaudeBot/1.0 (+claudebot@anthropic.com)" https://example.com/

Congratulations, you are now ClaudeBot. Every log line you generate says so, every user-agent-based analytics tool will count you as such, and every robots.txt rule written for that agent now applies to you, which is to say, applies to whatever you feel like doing.

This is not a flaw anyone is going to fix. HTTP has no native mechanism for a client to prove its identity, so the user agent has always been, and remains, a self-declared string. The question for anyone tracking AI crawlers is not whether that is a problem but what to do about it, in the request path, at scale, without adding latency you cannot afford.

Why bother

If you are only counting crawls for curiosity, user agent matching is fine. Cooperative crawlers from the major labs identify themselves honestly, and they are most of the legitimate volume.

Verification starts to matter the moment a crawl record influences a decision:

• You block or rate-limit by identity. An unverified check is trivially bypassed by the exact traffic you meant to stop, while your robots.txt compliance is honoured by the well-behaved crawlers you were happy to have.
• You report crawl coverage as a leading indicator. If a third of your "OAI-SearchBot" hits are a scraper wearing the name, your crawl-to-citation analysis is measuring a fiction.
• You store the IP. A verified vendor fleet IP is infrastructure metadata. An unverified one might belong to a person, which makes it personal data with all the obligations that follow.

That last point is the one that changes system design, and it is worth getting right before you collect anything.

Method 1: forward-confirmed reverse DNS

The long-standing approach, and still the most broadly applicable.

Skipping the forward lookup is the mistake that shows up in real code. Anyone who controls a block of IP space can publish a PTR record saying whatever they want. The forward confirmation is what proves the operator's own DNS agrees, and without it you have replaced one unauthenticated string with another.

The suffix check needs care too. googlebot.com.evil.example ends with neither googlebot.com as a label boundary nor anything trustworthy, but a naive hostname.includes("googlebot.com") passes it happily. Check that the hostname equals the domain or ends with "." + domain.

The operational cost is the real constraint. Two DNS round trips, in the request path, per request. On a cache miss that is tens of milliseconds, sometimes far worse when a resolver is having a bad day. You cannot do this synchronously on every crawl at any interesting volume.

The way out is that you do not have to. Verify asynchronously, after the response has gone out, and cache the verdict per IP. Crawler fleets reuse addresses heavily, so a cache with a few hours' TTL turns "two lookups per request" into "two lookups per new IP per few hours". Negative results need caching too, and shorter: a failure caused by a DNS blip should not condemn a legitimate crawler for a day.

Method 2: published IP ranges

Most major operators publish the address ranges their crawlers use, as JSON at a stable URL. Fetch, parse, match with a CIDR containment check.

The trade is obvious: no per-request network call, microsecond matching against a prefix tree, and no dependency on DNS at request time. The problem is equally obvious and much easier to underestimate.

Practical requirements for a range-based verifier:

• Refresh on a schedule: daily is defensible, hourly is cheap.
• Keep the last good copy and keep serving from it if a fetch fails. Never fall back to an empty set; that turns a transient network problem into "no crawler is real".
• Handle IPv6 properly. It is not optional, and a v4-only implementation will simply mark a growing share of legitimate traffic unverified.
• Alert on staleness, not just on fetch errors. A 200 response returning yesterday's file forever is a failure mode too.
• Version what you matched against. When a verdict is questioned three weeks later, "which range file was in effect" is the first question.

Method 3: cryptographic signatures

RFC 9421 HTTP Message Signatures let a client sign selected parts of a request so the recipient can verify it came from the holder of a specific key. Applied to bots (the Web Bot Auth work builds on exactly this), it turns identity from an inference into a proof. We covered the mechanics of RFC 9421 in detail previously.

It is strictly better than the other two methods where it is available: no DNS dependency, no range file to go stale, and it survives an operator changing their infrastructure. The catch is coverage. Not every crawler signs today, so a signature-only verifier marks most legitimate traffic unverified. Treat it as the top tier of a ladder, not a replacement for the rungs below it.

The chain, in the order to run it

1. Pre-filter on the user agent. If the string does not name a known crawler, no verification is needed. This keeps the expensive path off the overwhelming majority of requests.
2. Check for a signature. If present and valid, you are done, with the strongest verdict available.
3. Check the IP against published ranges. Cheap, local, no network call.
4. Fall back to forward-confirmed rDNS, asynchronously, and cache the result per IP.
5. Otherwise record it as unverified, and keep it. An unverified crawl claiming to be GPTBot is a real observation about your traffic. It is just not the observation the user agent claims.

Two rules make this durable.

Never let verification block the response. Everything above happens after the response is sent, or on a background path. A DNS resolver having a bad afternoon must never become your site having a bad afternoon.

Never collapse the outcome to a boolean. Record the method, the verdict and (where the method is inferential) a confidence. verified_by: "rdns" and verified_by: "signature" are different claims, and six weeks later, when somebody asks why a particular crawl was counted, the answer needs to be in the record rather than in a maintainer's memory.

The privacy line

Verification status determines what you are allowed to keep, and this is not a small detail.

A verified crawler IP belongs to a documented vendor fleet. It is infrastructure metadata: not a person, not a household, not a device someone browses from. Retaining it is defensible and genuinely useful for debugging.

An unverified IP is an unknown party. It could be a scraper, a misconfigured tool, or somebody's laptop behind a residential connection, which makes it personal data under GDPR, with everything that entails.

Treating those two identically is how a crawler tracker quietly becomes a visitor-IP database. The discipline that avoids it:

• Store the raw IP only when verification succeeded, and store a salted hash otherwise.
• Enforce it in the schema, not only in application code. A database constraint survives a refactor; a convention does not.
• Strip query strings before anything is stored. Crawlers follow whatever links exist, and links in the wild carry session tokens, email addresses and reset tokens. Strip at the edge, at collection time, unconditionally.

A verification health check

Three numbers, reviewed monthly, catch nearly every failure this system has:

The first is the one to watch hardest, because every infrastructure failure in this list expresses itself the same way: as a decline in verified share that looks, on a dashboard, exactly like a decline in crawler interest. One is a bug in your pipeline. The other is a business problem. Telling them apart requires that you were already tracking the difference.