"Should we block AI crawlers?" is not one question. It is three, and they have different answers.
The advice circulating online almost universally collapses them. A blog post recommends a robots.txt block, someone pastes it in, and a month later the site has quietly removed itself from the AI search results it was hoping to appear in, because the pasted list named the search-index crawlers alongside the training ones, and nobody checked which was which.
The decision is genuinely yours to make, and reasonable sites land in different places. What is not defensible is making it by accident.
The three bargains
Training crawlers collect content to inform future model weights. GPTBot, ClaudeBot, CCBot, Google-Extended, Meta-ExternalAgent. The exchange is: your content contributes to a model's general capability, and you receive nothing directly and traceably. No link, no citation, no visit. Some argue for diffuse long-term benefit from a model that knows your domain well. It is not measurable, and you should treat claims that it is with suspicion.
Search-index crawlers build the retrieval corpus an assistant consults at answer time. OAI-SearchBot, PerplexityBot. The exchange is direct: you are fetched, and in return you become eligible to be cited (with a link) in answers to questions people are asking right now. This is the closest analogue to letting Googlebot index you, and blocking it has the same character as blocking Googlebot.
User-triggered fetchers retrieve one page because one person asked about it. ChatGPT-User, Claude-User, Perplexity-User. There is no bulk collection here at all. Someone pasted your URL or asked a question that resolved to your page, and the assistant went to read it for them. Blocking this is closest to blocking a specific human visitor, with the difference that the block is visible to them as a failure to answer.
What each category costs you and pays you
Vendor naming changes; verify current agent names against operator documentation before writing rules.
| Category | Example agents | What you give | What you get |
|---|---|---|---|
| Training | GPTBot, ClaudeBot, CCBot, Google-Extended | Content into future model weights | Nothing traceable, ever |
| Search index | OAI-SearchBot, PerplexityBot | Content into the live retrieval corpus | Eligibility to be cited, with a link, today |
| User-triggered | ChatGPT-User, Claude-User, Perplexity-User | One page, on demand, to one person | A real person gets a real answer about you |
The decision, by site type
A defensible default per category
Defaults, not rules. The right answer depends on whether your content is the product or the marketing for the product.
| Your situation | Training | Search index | User-triggered |
|---|---|---|---|
| SaaS / product marketing site | Allow: the content exists to be spread | Allow, emphatically | Allow |
| E-commerce | Allow: product data benefits from being known | Allow: assistants increasingly drive product discovery | Allow |
| Publisher, ad-funded | Block or licence: this is your inventory | Case by case: citations bring traffic, summaries replace it | Allow |
| Paywalled / subscription content | Block | Allow previews only, block the paid corpus | Allow previews only |
| Docs / developer content | Allow: being the source model answers come from is the goal | Allow, emphatically | Allow |
| Original research / proprietary data | Block: this is the asset | Allow the summary pages, block the underlying data | Allow the summary pages |
The pattern across every row: training is where the debate lives, and search indexing is where the traffic lives. If your content exists to bring people to you, blocking the search indexer is the one move that reliably costs you something you can count.
The mistake this post exists to prevent
Copying a "block AI bots" list into robots.txt without checking which category each agent belongs to. The typical result is that OAI-SearchBot and PerplexityBot get blocked alongside the training crawlers, removing the site from AI answers, which is the expensive half and the half nobody had decided to do.
Writing the rules
robots.txt is per-user-agent, and the groups do not cascade. Be explicit.
Allow AI search, decline training (the most common position for a marketing or product site that has been thinking about it):
# Search-index crawlers: allowed, because citation is the point
User-agent: OAI-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /
# User-triggered fetches: allowed, a person is waiting
User-agent: ChatGPT-User
Allow: /
User-agent: Claude-User
Allow: /
User-agent: Perplexity-User
Allow: /
# Training crawlers: declined
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: Google-Extended
Disallow: /
Blocking a section rather than a site (the paywall pattern):
User-agent: GPTBot
Disallow: /members/
Disallow: /research/data/
Allow: /
User-agent: OAI-SearchBot
Disallow: /members/
Allow: /
Three mechanics people get wrong:
Google-Extended is not Googlebot. It governs Gemini and Vertex training use. Disallowing it does not affect Google Search ranking, and disallowing Googlebot to keep content out of AI training removes you from Google Search entirely. These are separate controls and should be reasoned about separately.
A wildcard group is the fallback, and groups do not inherit. User-agent: * followed by Disallow: / applies to every cooperative crawler that does not find a group naming it. A crawler picks the group matching its own name regardless of where that group sits in the file (order is not significant under RFC 9309), but the group it picks is the only one it reads. So a named Allow group must restate every rule it wants, including your sitemap-adjacent paths; it inherits nothing from the wildcard group above or below it.
A Disallow is not a delete. Content already collected stays collected. robots.txt governs future fetching only, and nothing about a block causes anything to be removed from an existing corpus.
What blocking does not do
robots.txt is a request. RFC 9309 codifies the format, and cooperative crawlers from the major labs honour it, but honouring it is a choice they make, and nothing in HTTP enforces it.
A scraper that wants your content will send Mozilla/5.0 and take it. Some do exactly that, and some send a forged AI crawler user agent, which is why your logs cannot be treated as a compliance report. If the content genuinely must not be taken, the control has to be at the network layer: WAF rules, verified-bot enforcement at your CDN, or authentication in front of the content.
Four levels of control, escalating
Each level costs more to operate and stops more traffic. Most sites need level 1; sites whose content is the product often need level 3 or 4.
robots.txt
Free, instant, honoured by cooperative crawlers only. The right starting point for nearly everyone.
Verified-bot rules at the CDN
Block or challenge by verified identity rather than by claimed user agent. Handles forgery.
Rate limiting and challenges
Caps aggressive collection regardless of who claims to be doing it. Risks catching real users.
Authentication
The only actual guarantee. Also removes the content from search and AI answers entirely.
Several CDNs now ship managed AI-bot controls, and there are emerging models for charging crawlers per request rather than choosing between free access and a hard block. If you are a publisher whose content is genuinely the inventory, that is the direction worth watching: the binary allow/deny framing is a poor fit for what is actually a pricing question.
Verify the decision took effect
This step gets skipped almost universally, and it is where the intent and the reality separate.
After changing rules, check server-side (browser analytics cannot see crawler traffic at all) for two things:
Did the blocked agents stop? Cooperative crawlers usually re-read robots.txt within a day or so. If GPTBot is still arriving a week after a Disallow, either the file is not being served correctly on that host, or what is arriving is not GPTBot.
Did the allowed agents keep arriving? This is the failure that hurts, because it is silent. A syntax error, a stray wildcard group, or a Disallow: / inherited from a staging config can eliminate your search-index crawls, and nothing anywhere will tell you. The traffic consequence shows up weeks later, long after the change that caused it.
# Yesterday's crawls by declared agent, from an access log
awk '{print $0}' access.log \
| grep -Eio 'gptbot|oai-searchbot|claudebot|claude-user|perplexitybot|chatgpt-user|ccbot' \
| sort | uniq -c | sort -rn
Run it before the change and after. If you cannot produce those two numbers, you have not made a decision about AI crawlers. You have made a guess and stopped watching.
The question to actually answer
Not "do we block AI?" but "for each of the three categories, what are we exchanging and is the exchange worth it?" Sites that answer it that way end up with different rules per category, which is the shape of a decision rather than the shape of a reflex.
Frequently asked
It depends on whether you want your content in future model training, and it is a separate decision from AI search visibility. GPTBot collects training data: blocking it costs you no citations and no traffic today. If your content is your product, such as paywalled journalism or original research, blocking is reasonable. If your content exists to attract customers, there is little to protect. Critically, blocking GPTBot does not affect whether ChatGPT can cite you; that is controlled by OAI-SearchBot.
GPTBot is OpenAI's training crawler: it collects content that may inform future models, with no traffic returned to you. OAI-SearchBot builds the search index ChatGPT retrieves from when answering questions, and being in it is what makes you eligible to be cited with a link. Blocking GPTBot keeps you out of training while remaining citable. Blocking OAI-SearchBot removes you from ChatGPT's answers.
Blocking dedicated AI crawlers such as GPTBot, ClaudeBot or Google-Extended does not affect traditional search ranking: they are separate agents from Googlebot and Bingbot. The danger is collateral damage: a wildcard rule or an over-broad blocklist that also disallows Googlebot will remove you from Google Search. Google-Extended specifically governs Gemini training and has no effect on Search.
It stops cooperative ones. robots.txt is a request, formalised in RFC 9309, that well-behaved crawlers from the major AI labs honour. It has no enforcement mechanism, so a scraper can ignore it or send a disguised user agent. If a block must be guaranteed, you need network-level controls at your CDN or WAF, or authentication in front of the content.
No. robots.txt governs future fetching only. Content already collected remains in whatever corpus it entered, and a block has no retroactive effect. Removal, where it is possible at all, is a separate request process that varies by operator.
Sources & further reading
- 01RFC 9309: Robots Exclusion Protocol, IETF
- 02Overview of OpenAI crawlers and user agents, OpenAI
- 03Does Anthropic crawl data from the web, and how can site owners block the crawler?, Anthropic Support
- 04Google-Extended and controlling AI training use, Google Search Central
- 05PerplexityBot and Perplexity-User, Perplexity