OpenAI · Search index
OAI-SearchBot
Everything ChatGPT quotes, summarises and links in its search answers comes out of the index this token builds. It runs on a crawl schedule of OpenAI's choosing rather than in response to any one person's question, so the fetch that indexes you and the answer that cites you are separated in time: the mirror image of ChatGPT-User, which fetches at the instant of asking. It is strictly a retrieval-side token: OpenAI states that ChatGPT-User plays no part in deciding whether content appears in Search, and directs search opt-outs here instead.
- Operated by
- OpenAI
- Purpose
- Search index
- robots.txt token
- OAI-SearchBot
- Verification
- IP verified
Crawls continuously to build and refresh an AI search index.
How to verify OAI-SearchBot
OpenAI publishes the IP ranges OAI-SearchBot crawls from, and we fetch that list on a schedule. A request claiming to be OAI-SearchBot can therefore be checked against 39 published ranges: one that does not match is not this crawler.
Published ranges last confirmed by us on .
User agent
OpenAI publishes this user agent for OAI-SearchBot. Match on the OAI-SearchBot product token rather than the whole string: vendors revise the surrounding version and URL fragments without notice.
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.4; +https://openai.com/searchbotrobots.txt for OAI-SearchBot
Yes. This is one of the two tokens OpenAI names as its intended robots.txt controls. Two operational details matter: propagation is slow, with OpenAI stating it can take around 24 hours for a robots.txt change to take effect, and robots.txt fetches may arrive under a distinct user agent carrying a `robots.txt` marker, so a matcher keyed on the plain string can miss them.
Block
User-agent: OAI-SearchBot
Disallow: /Allow
User-agent: OAI-SearchBot
Allow: /robots.txt is a request, not an enforcement mechanism. It is honoured by convention, and a crawler that ignores it is stopped at your edge, not in a text file.
What blocking OAI-SearchBot costs you
This is the expensive one, and the loss is graded rather than total. Opted-out sites are not shown in ChatGPT search answers, though OpenAI says they can still appear as navigational links, and the referral stream dies with the snippets, since ChatGPT tags outbound search links with a `utm_source=chatgpt.com` parameter that is exactly the traffic an operator can see in their analytics. There is also a trap worth knowing: if OpenAI learns a disallowed URL from a third party or from crawling other pages, it may still surface the bare link and title, and the documented remedy is a `noindex` tag it can only read if the crawler is allowed to fetch the page.