Meta · Search index
meta-webindexer
Citations with links are the stated payback, and this is the only Meta token whose documentation promises anything back at all: Meta says it “navigates the web to improve Meta AI search result quality,” analysing content “to enhance the relevance and accuracy of Meta AI.” That attribution surface is the concrete difference from meta-externalagent, which absorbs the same text into training with no link attached to it. It is also the newest of the family, absent from Meta's January 2025 page and present by January 2026, so any allow-list assembled before 2026 does not mention it and a site that believes it has handled Meta's AI crawlers may have handled only the 2024-vintage pair. What Meta does not offer is any way to confirm a request really came from it.
- Operated by
- Meta
- Purpose
- Search index
- robots.txt token
- meta-webindexer
- Verification
- Network origin only
Crawls continuously to build and refresh an AI search index.
How to verify meta-webindexer
We hold no list of IP addresses that Meta publishes itself for meta-webindexer. What we hold instead is 574 prefixes that a third party (a public routing record, not Meta) reports Meta's network announcing. A request from inside one of those tells you where it came from, not who sent it: anything else on that network can send the same packet, and nothing we hold ties meta-webindexer to those addresses beyond the network they sit on. A request from OUTSIDE them is not thereby a fake either: this describes one network's announcements, not every address Meta can crawl from. Corroboration, not proof.
We hold no range list Meta publishes itself for meta-webindexer, so there is no such fetch to confirm. This entry was last reviewed against Meta's own documentation on .
User agent
Meta publishes this user agent for meta-webindexer. Match on the meta-webindexer product token rather than the whole string: vendors revise the surrounding version and URL fragments without notice.
meta-webindexer/1.1 (+/documentation/sharing/webmasters/web-crawlers)robots.txt for meta-webindexer
Yes, and Meta frames it as an opt-in benefit rather than a restriction: “Allowing Meta-WebIndexer in your robots.txt file helps us cite and link to your content in Meta AI's responses.” No bypass clause is claimed.
Block
User-agent: meta-webindexer
Disallow: /Allow
User-agent: meta-webindexer
Allow: /robots.txt is a request, not an enforcement mechanism. It is honoured by convention, and a crawler that ignores it is stopped at your edge, not in a text file.
What blocking meta-webindexer costs you
Of the five Meta tokens this is the one whose block has a traffic-shaped price. Meta states that allowing it is what lets Meta AI “cite and link to your content,” so a disallow takes you out of the pool of sources an answer can attribute, and removes the referral clicks those citations carry with them. Your presence on Facebook, Instagram and WhatsApp is untouched; the loss is narrowly the Meta AI citation surface and nothing adjacent to it.