ClaudeBot
Bulk collection of public web content that could potentially contribute to Anthropic's model training. The word Anthropic reaches for when describing the effect of a block is *signals* rather than *blocks*, and that is the real character of this token: a disallow reads as a durable statement of intent about your material, not merely a closed door on today's request. Its scope is forward-looking only: Anthropic says restricting it signals that future materials should be excluded, and says nothing about what has already been collected.
- Operated by
- Anthropic
- Purpose
- Training
- robots.txt token
- ClaudeBot
- Verification
- IP verified
Collects pages into a corpus used to train models.
How to verify ClaudeBot
Anthropic publishes the IP ranges ClaudeBot crawls from, and we fetch that list on a schedule. A request claiming to be ClaudeBot can therefore be checked against 14 published ranges: one that does not match is not this crawler.
Published ranges last confirmed by us on .
User agent
Anthropic has not published a full user-agent string for ClaudeBot. Requests are identified by the ClaudeBot product token appearing in the User-Agent header; we match that token rather than a whole string, because the rest of the header varies and matching it would miss real traffic.
robots.txt for ClaudeBot
Yes, stated as a blanket commitment covering all Anthropic bots, which also honour the non-standard `Crawl-delay` directive. Anthropic further commits not to bypass anti-circumvention measures such as CAPTCHAs. One operational warning: the opt-out must be repeated for every subdomain you want excluded.
User-agent: ClaudeBot
Disallow: /User-agent: ClaudeBot
Allow: /robots.txt is a request, not an enforcement mechanism. It is honoured by convention, and a crawler that ignores it is stopped at your edge, not in a text file.
What blocking ClaudeBot costs you
The cleanest trade of Anthropic's three tokens. No visitor-facing surface changes and no referral traffic disappears, because Anthropic assigns the search and user-retrieval consequences to Claude-SearchBot and Claude-User respectively and describes no downside here beyond training exclusion itself. You forgo corpus inclusion, which pays you nothing measurable, and you lose no citations. The one cost with a real price tag is hidden in that phrase about future materials: a block is not retroactive, and Anthropic documents no mechanism for withdrawing what was already taken.
Vendor documentation
Other Anthropic tokens we track
This page tells you what ClaudeBot is. Traceten tells you whether it came, what it read, and whether any of it turned into revenue.
See how