Skip to content
Traceten

Free tool

AI robots.txt checker

Paste your robots.txt to see whether each of the 60 AI crawler tokens we track may fetch your home page and any path you enter. The checker follows RFC 9309: a group that names the crawler takes priority over the * group, and the longest matching rule decides. It runs in your browser, so nothing you paste is sent.

Runs in your browser. Nothing you paste is sent.

Which AI crawlers can read your site?

Runs in your browser. Nothing you paste is sent.

3 of 60 blocked at /5 blocked at /checkout/cart

Training3/18

Search index0/15

Answers0/15

Other AI0/12

How do I read the result?

Allowed or blocked
Whether the crawler may fetch / and the path you entered, under the group it follows.
The group
A crawler follows the group that names it, then a fallback its vendor documents (Google-CloudVertexBot uses Googlebot's), then *. With no group, everything is allowed.
The deciding rule
The longest matching Allow or Disallow wins, and Allow wins a tie. The line number points at it.
May ignore robots.txt
The vendor says this fetcher may skip robots.txt when a person asks for a page. Block it at your firewall if you must.
Sends no requests
Google-Extended and Applebot-Extended are permission tokens: they control how the vendor may use content, not whether it is crawled.

Changes are not instant: OpenAI says a robots.txt change can take about 24 hours to reach OAI-SearchBot, and Perplexity says up to 24 hours.

Frequently asked questions

01

Does blocking GPTBot remove my site from ChatGPT?

No: GPTBot only collects training data. ChatGPT search draws on OAI-SearchBot's index, and live fetches for a user arrive as ChatGPT-User, each with its own robots.txt group. To opt out of training but stay in ChatGPT search, block GPTBot and allow OAI-SearchBot.
02

Does Google-Extended keep my site out of AI Overviews?

No. Google-Extended only controls whether Google may use your content for Gemini training and grounding. AI Overviews and AI Mode draw on the Search index Googlebot builds, so blocking Googlebot is the only robots.txt route, and it removes you from Search too.
03

Which group does a crawler follow when several match?

Under RFC 9309 a crawler merges every group that names its token, matched case-insensitively. It falls back to the * group only when no group names it, so a named group replaces the * rules rather than adding to them.
04

How are Allow and Disallow conflicts resolved?

The most specific rule wins: the matching rule with the longest path. If an Allow and a Disallow are equally long, Allow wins. In a path, * matches any run of characters and a final $ marks the end of the URL.
05

Do AI crawlers always obey robots.txt?

No. Most training and search crawlers say they do, but several user-triggered fetchers, including Google-Agent, Perplexity-User and meta-externalfetcher, are documented as possibly ignoring it because a person asked for the page. robots.txt is a request, not access control.
06

Does this checker fetch my robots.txt?

No. It makes no network requests: everything runs in your browser against the text you paste. Open yoursite.com/robots.txt and copy the file in.

Sources and further reading

  1. 01RFC 9309: Robots Exclusion Protocol, IETF
  2. 02Overview of OpenAI crawlers, OpenAI
  3. 03Google's common crawlers, Google
  4. 04Google's user-triggered fetchers, Google
  5. 05Perplexity crawlers, Perplexity

Vendor statements checked 11 August 2026.

AI is already sending you buyers. See the receipts.

No calls, no credit card.

Join the waitlistRead the docs