Allen AI · Training
AI2Bot
What this collects becomes a published artefact rather than a private asset. Ai2 is a non-profit research institute, and its crawling notice describes the bot plainly as exploring certain domains to find web content used to train open language models, with the Olmo 3 release committing to make every component of that flow openly available, data and code and weights and checkpoints, over a pretraining corpus of roughly 9.3 trillion tokens. It sweeps on a schedule of its own with no answer surface, no citation slot and no user query anywhere behind it.
- Operated by
- Allen AI
- Purpose
- Training
- robots.txt token
- AI2Bot
- Verification
- User agent only
Collects pages into a corpus used to train models.
How to verify AI2Bot
Allen AI publishes no list of IP addresses for AI2Bot, so nobody (us included) can prove that a request carrying this user agent really came from Allen AI. The name is trivial to copy. Treat it as a claim the visitor is making about itself, not as an identity anyone has checked.
There is no range list to confirm. This entry was last reviewed against Allen AI's own documentation on .
User agent
Allen AI publishes this user agent for AI2Bot. Match on the AI2Bot product token rather than the whole string: vendors revise the surrounding version and URL fragments without notice.
Mozilla/5.0 (compatible) AI2Bot (+https://www.allenai.org/crawler)robots.txt for AI2Bot
Not on the crawler page, and the silence there is conspicuous: its only control guidance is that the user-agent string can be used to filter or reject traffic from the crawler if desired, which points operators at server-side filtering rather than at robots.txt. A commitment does exist, but it lives in the Olmo 3 announcement and is framed at institute level rather than per token: that Ai2 follows established web standards in collecting training data and does not collect from sites that explicitly disallow it, including paywalled content. Worth knowing which document you are relying on when you write the rule.
Block
User-agent: AI2Bot
Disallow: /Allow
User-agent: AI2Bot
Allow: /robots.txt is a request, not an enforcement mechanism. It is honoured by convention, and a crawler that ignores it is stopped at your edge, not in a text file.
What blocking AI2Bot costs you
Nothing a site owner measures moves: no referral traffic, no citations, no link preview. The trade is different in kind from a commercial trainer, because a released corpus makes exclusion permanent and public: your content simply absent from a dataset that other labs and academic researchers go on to reuse, with no counterparty to negotiate later inclusion with, and equally no opacity, since a public corpus is one of the few you can audit yourself. The correction that matters most for anyone writing the rule: blocking this token does not remove you from every Ai2 corpus. The original Dolma corpus was assembled from 24 Common Crawl snapshots and C4 rather than from a first-party crawl, so exclusion from Ai2's Common-Crawl-derived data is governed by CCBot, not by this agent, which makes a block here considerably less complete than it looks.