xAI · Other AI
xAI-Web-Crawler
Rules against this string are already deployed in production robots.txt files on the open web, which is the interesting fact about it, because its vendor has never acknowledged that it exists. No xAI page names it. The secondary descriptions that do are visibly generated rather than observed: one amounts to a fallback line saying it is tracked as another xAI bot that does not fit the other categories, which is a placeholder, not a description. Its function is therefore unestablished. What this entry really documents is a process: an unattributed token propagating between directories until operators write configuration against it, folk knowledge hardening into deployed rules.
- Operated by
- xAI
- Purpose
- Other AI
- robots.txt token
- xAI-Web-Crawler
- Verification
- User agent only
AI-adjacent traffic that does not fit the three purposes above.
How to verify xAI-Web-Crawler
xAI publishes no list of IP addresses for xAI-Web-Crawler, so nobody (us included) can prove that a request carrying this user agent really came from xAI. The name is trivial to copy. Treat it as a claim the visitor is making about itself, not as an identity anyone has checked.
There is no range list to confirm. This entry was last reviewed against xAI's own documentation on .
User agent
xAI has not published a full user-agent string for xAI-Web-Crawler. Requests are identified by the xAI-Web-Crawler product token appearing in the User-Agent header; we match that token rather than a whole string, because the rest of the header varies and matching it would miss real traffic.
robots.txt for xAI-Web-Crawler
Block
User-agent: xAI-Web-Crawler
Disallow: /Allow
User-agent: xAI-Web-Crawler
Allow: /robots.txt is a request, not an enforcement mechanism. It is honoured by convention, and a crawler that ignores it is stopped at your edge, not in a text file.
What blocking xAI-Web-Crawler costs you
A disallow for a token nobody has documented is free to add and proves nothing once added: there is no published behaviour for it to change, and no range or signature scheme against which a later log line could confirm the rule ever applied to a genuine xAI request. The argument for keeping it is defensive breadth. The argument for care is that the line itself becomes evidence to the next person who reads your file, which is how the string spread in the first place. A third-party bot-verification matrix grades it user-agent-string-only, very weak, trivially spoofed, with no published IPs: the same grade every token in this group carries, for the same reason.
Vendor documentation
xAI does not publish documentation for xAI-Web-Crawler that we could find. Everything on this page comes from what they do publish elsewhere and from observed behaviour, so treat it accordingly.