ByteDance · Search index
Bytespider
Widely filed as a training crawler, this is the one token in this group with real first-party documentation, and that documentation describes a search engine. ByteDance's Toutiao Search webmaster platform walks through the classic pipeline in its own words: 抓取网页 (fetch pages by following hyperlinks), 处理网页 (extract keywords, deduplicate, segment Chinese text, build the index), 提供检索服务 (serve results with title, URL and snippet), and every surrounding tool is search-operational: index-volume monitoring, sitemap submission, dead-link submission, crawl-rate throttling, mobile landing-page quality rules that explicitly threaten ranking demotion. Nowhere in the platform's 22 documents is model training, an LLM, Doubao or an AI answer mentioned, so a training purpose is something third parties attribute to ByteDance rather than something ByteDance claims. It also carries the strongest verification story of any crawler ByteDance or Alibaba operates: ten published `/24` ranges plus per-platform reverse-DNS instructions, and sampling addresses inside those ranges returns exactly the promised hostnames, in the form `bytespider-<dashed-ip>.crawl.bytedance.com`, with the caveat that one sampled address had no PTR at all, so a verifier must fail closed on a missing record rather than read absence as proof either way.
- Operated by
- ByteDance
- Purpose
- Search index
- robots.txt token
- Bytespider
- Verification
- Vendor-documented
Crawls continuously to build and refresh an AI search index.
How to verify Bytespider
ByteDance publishes no machine-readable range file for Bytespider that we can fetch on a schedule, so this page shows no range count and no verification date. That is not the same as unverifiable: ByteDance documents forward-confirmed reverse DNS, where a genuine request resolves to a hostname under .bytedance.com and that hostname resolves back to the address the request came from and the 10 address ranges ByteDance states this crawler comes from (110.249.201.0/24, 110.249.202.0/24, 111.225.148.0/24, 111.225.149.0/24, 220.243.135.0/24, 220.243.136.0/24, 220.243.188.0/24, 220.243.189.0/24, 60.8.123.0/24 and 60.8.151.0/24), written into its documentation as text rather than as a file, so we transcribe them here and fetch nothing: check them against ByteDance's own page before you rely on them. That check is run against one request at a time, at your edge. It is not something we have confirmed on your behalf here.
We fetch no range list for this crawler, so there is no fetch to confirm. This entry was last reviewed against ByteDance's own documentation on .
ByteDance documents reverse-DNS verification: a genuine request resolves to a hostname under .bytedance.com that forward-resolves back to the same address.
User agent
ByteDance publishes this user agent for Bytespider. Match on the Bytespider product token rather than the whole string: vendors revise the surrounding version and URL fragments without notice.
Mozilla/5.0 (compatible; Bytespider; https://zhanzhang.toutiao.com/) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.0.0 Safari/537.36robots.txt for Bytespider
Yes, and ByteDance documents the matching semantics rather than merely asserting compliance, a stronger signal than most vendors give. A dedicated Robots匹配说明 page states that path matching is 与google匹配方式保持一致 ("kept consistent with Google's matching method"), supports `*` and `$`, and then specifies an extension Google does not have: order-independent matching on query-string parameters, with wildcards permitted in both parameter keys and values. A separate page states the crawler self-throttles against a site's update frequency and server load and offers a console override, though the two pages disagree on timing: 最快次日生效 ("effective the next day at the earliest") against 在1天内 ("within 1 day") for the same change. Against all of that sits a long-standing operator complaint of non-compliance: a 中国企业家 (*China Entrepreneur*) investigation published 2019-11-08 and carried on Sina Tech reported that 该爬虫在抓取时完全不遵守网站的robots协议 ("the crawler completely fails to observe sites' robots protocol when crawling"), with administrators describing 460,000 requests in a single morning and bandwidth saturated to 100% until they blocked ByteDance's IPs. That is credible business media rather than a vendor source, it is seven years old, it is contradicted by ByteDance's own published policy, and we have not independently verified current behaviour, so treat it as a documented dispute, not as a present-tense fact.
Block
User-agent: Bytespider
Disallow: /Allow
User-agent: Bytespider
Allow: /robots.txt is a request, not an enforcement mechanism. It is honoured by convention, and a crawler that ignores it is stopped at your edge, not in a text file.
What blocking Bytespider costs you
The remedy ByteDance itself recommends is a dial rather than a switch, which reframes the whole decision. For a site serving Western readers the surface behind this crawler is Toutiao Search, a Chinese-market product that will not send you visitors you would otherwise have had, so a block gives up little; for anyone courting Chinese-language readers it removes you from a ByteDance-scale index, and the documented crawl-rate control is the proportionate response to a server being hammered. There is a trap in blocking outright: ByteDance treats its crawler's access as a precondition for its own platform: the site-verification troubleshooting page tells owners whose verification failed to check whether they have blocked the published IPs and user agent, so a blanket deny also locks you out of the console you would need to throttle it, which is itself gated behind a Toutiao account and a domain-verification file.