Common Crawl · Training
CCBot
No product sits at the other end of this one: no search box, no assistant, no answer card. Common Crawl is a non-profit 501(c)(3) that crawls the web and freely provides its archives to the public, and CCBot is the collection mechanism: a Nutch crawler on Apache Hadoop publishing roughly monthly snapshots of some two billion pages, over ten petabytes accumulated since 2008 and hosted free through the AWS Open Data Sponsorship Program. The consequence that makes this token unlike every other entry is that the archive is downloadable by anyone, so one fetch can land in an unbounded number of third-party datasets, models and papers. Verification is well documented in both directions: reverse DNS under `crawl.commoncrawl.org` with `host` and `dig -x` examples, plus a JSON range file that is the only one in this group containing IPv6, with one asymmetry: reverse DNS is not yet supported over IPv6, so IPv6 traffic is verifiable by range alone. Common Crawl also warns that it is aware of crawlers falsely identifying themselves as CCBot, which makes a bare user-agent match especially untrustworthy for this token.
- Operated by
- Common Crawl
- Purpose
- Training
- robots.txt token
- CCBot
- Verification
- IP verified
Collects pages into a corpus used to train models.
How to verify CCBot
Common Crawl publishes the IP ranges CCBot crawls from, and we fetch that list on a schedule. A request claiming to be CCBot can therefore be checked against 5 published ranges: one that does not match is not this crawler.
Published ranges last confirmed by us on .
Common Crawl documents reverse-DNS verification: a genuine request resolves to a hostname under .crawl.commoncrawl.org that forward-resolves back to the same address.
User agent
Common Crawl publishes this user agent for CCBot. Match on the CCBot product token rather than the whole string: vendors revise the surrounding version and URL fragments without notice.
CCBot/2.0 (https://commoncrawl.org/faq/)robots.txt for CCBot
Yes, and this is the only token in this group that obeys `crawl-delay`: "We obey the Crawl-delay parameter for robots.txt," with a worked example of `Crawl-delay: 2` throttling to one request every two seconds. Amazon's three refuse the directive and so does Apple, so an operator whose actual complaint is server load rather than data usage has a remedy here that the commercial crawlers do not offer. The full block is the standard form: ``` User-agent: CCBot Disallow: / ```
Block
User-agent: CCBot
Disallow: /Allow
User-agent: CCBot
Allow: /robots.txt is a request, not an enforcement mechanism. It is honoured by convention, and a crawler that ignores it is stopped at your edge, not in a text file.
What blocking CCBot costs you
The trade here is genuinely two-sided and both sides are large. Against allowing it: Common Crawl reports that its corpus "has become one of the most widely used sources of training data for large language models," citing a 2024 Mozilla Foundation study finding at least 64% of 47 large language models published between 2019 and 2023 were trained on filtered versions of it, with GPT-3 drawing more than 80% of its training tokens from filtered Common Crawl and C4/T5, and LLaMA, BLOOM, Pythia and Falcon all built on it, so a block here reaches a shared upstream rather than one consumer. For allowing it: the same block removes you from a public-good dataset with more than 12,000 research citations, whose stated purpose is to "democratize the data so that everyone, not just big companies, can do high-quality research and analysis," and it cannot distinguish a frontier lab from a linguistics PhD: the well-funded labs mostly run their own crawlers anyway. One practical caveat either way: a block is forward-looking, since snapshots already published are already mirrored worldwide, and Common Crawl publishes no retroactive-removal policy we could find.