Anthropic · Search index
Claude-SearchBot
Index quality is the whole remit: Anthropic describes this agent as navigating the web to improve search result quality, analysing content specifically to enhance the relevance and accuracy of search responses. The framing is *indexing*, which Anthropic keeps distinct from Claude-User's live retrieval: this token builds the standing picture of your site that later answers draw on, rather than fetching a page in the moment someone asks. It is separated from training as well; ClaudeBot owns that.
- Operated by
- Anthropic
- Purpose
- Search index
- robots.txt token
- Claude-SearchBot
- Verification
- IP verified
Crawls continuously to build and refresh an AI search index.
How to verify Claude-SearchBot
Anthropic publishes the IP ranges Claude-SearchBot crawls from, and we fetch that list on a schedule. A request claiming to be Claude-SearchBot can therefore be checked against 26 published ranges: one that does not match is not this crawler.
Published ranges last confirmed by us on .
User agent
Anthropic has not published a full user-agent string for Claude-SearchBot. Requests are identified by the Claude-SearchBot product token appearing in the User-Agent header; we match that token rather than a whole string, because the rest of the header varies and matching it would miss real traffic.
robots.txt for Claude-SearchBot
Yes, under the same blanket commitment covering all Anthropic bots, with `Crawl-delay` support and the same per-subdomain requirement. Anthropic publishes no propagation-delay figure, so how long a change takes to take effect is undocumented.
Block
User-agent: Claude-SearchBot
Disallow: /Allow
User-agent: Claude-SearchBot
Allow: /robots.txt is a request, not an enforcement mechanism. It is honoured by convention, and a crawler that ignores it is stopped at your edge, not in a text file.
What blocking Claude-SearchBot costs you
Anthropic states the consequence in two parts, and the second is the interesting one: a disallow prevents indexing for search optimisation, which may reduce a site's visibility *and accuracy* in user search results. Accuracy is a cost that blocking a crawler does not usually carry: the implication is not simply that you vanish, but that Claude may still discuss you from a thinner or staler basis. Anthropic publishes no equivalent of OpenAI's navigational-links carve-out, so whether a blocked site keeps bare links in citations is undocumented either way.