Anthropic · Training
ClaudeBot
Bulk collection of public web content that could potentially contribute to Anthropic's model training. The word Anthropic reaches for when describing the effect of a block is *signals* rather than *blocks*, and that is the real character of this token: a disallow reads as a durable statement of intent about your material, not merely a closed door on today's request. Its scope is forward-looking only: Anthropic says restricting it signals that future materials should be excluded, and says nothing about what has already been collected.
- Operated by
- Anthropic
- Purpose
- Training
- robots.txt token
- ClaudeBot
- Verification
- IP verified
Collects pages into a corpus used to train models.
How to verify ClaudeBot
Anthropic publishes the IP ranges ClaudeBot crawls from, and we fetch that list on a schedule. A request claiming to be ClaudeBot can therefore be checked against 26 published ranges: one that does not match is not this crawler.
Published ranges last confirmed by us on .
User agent
Anthropic has not published a full user-agent string for ClaudeBot. Requests are identified by the ClaudeBot product token appearing in the User-Agent header; we match that token rather than a whole string, because the rest of the header varies and matching it would miss real traffic.
robots.txt for ClaudeBot
Yes, stated as a blanket commitment covering all Anthropic bots, which also honour the non-standard `Crawl-delay` directive. Anthropic further commits not to bypass anti-circumvention measures such as CAPTCHAs. One operational warning: the opt-out must be repeated for every subdomain you want excluded.
Block
User-agent: ClaudeBot
Disallow: /Allow
User-agent: ClaudeBot
Allow: /robots.txt is a request, not an enforcement mechanism. It is honoured by convention, and a crawler that ignores it is stopped at your edge, not in a text file.
What blocking ClaudeBot costs you
The cleanest trade of Anthropic's three tokens. No visitor-facing surface changes and no referral traffic disappears, because Anthropic assigns the search and user-retrieval consequences to Claude-SearchBot and Claude-User respectively and describes no downside here beyond training exclusion itself. You forgo corpus inclusion, which pays you nothing measurable, and you lose no citations. The one cost with a real price tag is hidden in that phrase about future materials: a block is not retroactive, and Anthropic documents no mechanism for withdrawing what was already taken.