xAI · Other AI
xAI-Bot
Nobody has established what this string does, and the useful version of this page says so rather than picking a side. The secondary directories carrying it disagree at the level of category: one lists it as a training crawler operated by xAI, while descriptions circulating out of the same ecosystem call it the agent that fetches pages for the search-driven retrieval grounding Grok's answers. Those are opposite functions with opposite consequences for a site operator. xAI documents neither, so the defensible position is that the function of this token is unestablished. Its shape is also what a log-parsing heuristic emits when it meets an unfamiliar xAI-ish string, which is a plausible route into a directory that scrapes other directories.
- Operated by
- xAI
- Purpose
- Other AI
- robots.txt token
- xAI-Bot
- Verification
- User agent only
AI-adjacent traffic that does not fit the three purposes above.
How to verify xAI-Bot
xAI publishes no list of IP addresses for xAI-Bot, so nobody (us included) can prove that a request carrying this user agent really came from xAI. The name is trivial to copy. Treat it as a claim the visitor is making about itself, not as an identity anyone has checked.
There is no range list to confirm. This entry was last reviewed against xAI's own documentation on .
User agent
xAI has not published a full user-agent string for xAI-Bot. Requests are identified by the xAI-Bot product token appearing in the User-Agent header; we match that token rather than a whole string, because the rest of the header varies and matching it would miss real traffic.
robots.txt for xAI-Bot
Block
User-agent: xAI-Bot
Disallow: /Allow
User-agent: xAI-Bot
Allow: /robots.txt is a request, not an enforcement mechanism. It is honoured by convention, and a crawler that ignores it is stopped at your edge, not in a text file.
What blocking xAI-Bot costs you
Not estimable, and inventing a figure would be precisely the failure this entry exists to avoid: a cost needs a function, and the function is contested. The two branches point opposite ways: if it collects training data, denying it forfeits nothing a visitor would notice; if it fetches at answer time, denying it forfeits a citation somebody was waiting on. A blanket rule buys whichever turns out to be true at the price of the other, and no verification mechanism exists to tell you which you bought. The move with actual information in it is to log what arrives under this string on your own server before writing a rule about it.
Vendor documentation
xAI does not publish documentation for xAI-Bot that we could find. Everything on this page comes from what they do publish elsewhere and from observed behaviour, so treat it accordingly.