You.com · Search index
YouBot
Verification is where this one is genuinely unusual, and it is three methods deep: requests are signed under the Web Bot Auth standard with Ed25519 keys served from a live well-known signature directory, hostnames follow a documented reverse-DNS pattern with a worked forward-confirmation example, and legitimate traffic is stated to originate from a single published /24. That range is written as prose in the documentation rather than served as a machine-readable file, which is exactly the shape an automated range sweep misses and the reason this crawler is so often graded user-agent-only. What it does with the access is index building: discovering and re-crawling public pages on its own schedule, ingesting PDFs, plain text, XML and RSS feeds and structured data in JSON-LD, microdata and RDFa alongside HTML, at a frequency driven by how often your content changes, site popularity, server response time and any crawl-delay you set.
- Operated by
- You.com
- Purpose
- Search index
- robots.txt token
- YouBot
- Verification
- Vendor-documented
Crawls continuously to build and refresh an AI search index.
How to verify YouBot
You.com publishes no machine-readable range file for YouBot that we can fetch on a schedule, so this page shows no range count and no verification date. That is not the same as unverifiable: You.com documents signed requests under the Web Bot Auth scheme, with You.com's public key directory published at https://you.com/.well-known/http-message-signatures-directory, forward-confirmed reverse DNS, where a genuine request resolves to a hostname under search.you.com and that hostname resolves back to the address the request came from and the address range You.com states this crawler comes from (68.67.112.0/24), written into its documentation as text rather than as a file, so we transcribe them here and fetch nothing: check them against You.com's own page before you rely on them. That check is run against one request at a time, at your edge. It is not something we have confirmed on your behalf here.
We fetch no range list for this crawler, so there is no fetch to confirm. This entry was last reviewed against You.com's own documentation on .
You.com documents reverse-DNS verification: a genuine request resolves to a hostname under search.you.com that forward-resolves back to the same address.
User agent
You.com publishes this user agent for YouBot. Match on the YouBot product token rather than the whole string: vendors revise the surrounding version and URL fragments without notice.
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; YouBot/1.0; +https://docs.you.com/youbot; env:prod) Chrome/X.X.X.X Safari/537.36robots.txt for YouBot
Yes, and with more operational detail than most vendors publish. You.com states that YouBot fully respects robots.txt directives including user-agent-specific rules and crawl-delay settings, and gives the number that actually matters in practice: robots.txt is cached for 30 minutes, so a change takes up to half an hour to take effect. Status-code handling is documented alongside it (404 removes the page from the index, 429 reduces the crawl rate, 503 pauses crawling), and crawl complaints have a dedicated mailbox rather than a generic contact form.
Block
User-agent: YouBot
Disallow: /Allow
User-agent: YouBot
Allow: /robots.txt is a request, not an enforcement mechanism. It is honoured by convention, and a crawler that ignores it is stopped at your edge, not in a text file.
What blocking YouBot costs you
You come out of the standing index the You.com answer engine draws on, and the removal is durable rather than per-query: the documentation describes no live fetch at answer time to rescue a blocked page, and a 404 is documented as dropping the page from the index outright. The wider exposure is indirect. You.com's developer catalogue sells a web layer onward as products: a Web Search API returning LLM-optimised results, a Contents API, an Answer API producing synthesised answers with inline citations, and research APIs above them. If those are served from the index this crawler builds, then a block removes you from every third-party AI application built on that layer, a far larger blast radius than you.com's own consumer traffic. You.com does not explicitly connect the two, so treat that second half as a strong inference rather than a documented consequence.