There is a file you are probably being told to add to your site. It takes twenty minutes to write, costs nothing, and sits at /llms.txt. The pitch is that AI assistants will read it and understand your site better. The counter-pitch, delivered with equal confidence, is that it is a placebo nobody consumes.
Both camps are arguing from assertion. The interesting thing about /llms.txt is that whether it is being read is not a matter of opinion at all: it is a request in your logs, or it is not. Almost nobody checks.
What the file actually is
llms.txt was proposed in September 2024 by Jeremy Howard. The reasoning behind it is genuinely sound, and it is worth stating properly because most summaries skip it.
A language model working with your website has a context window, and your website does not fit in it. HTML pages are mostly not content: navigation, cookie banners, footers, analytics tags, marketing chrome. A model retrieving your pricing page has to spend a large fraction of its budget on markup and boilerplate before reaching the sentence it needs.
llms.txt is a Markdown file at your site root that offers a curated answer to "if you could only read a handful of things here, read these." The format is deliberately minimal:
# Traceten
> Analytics that detect AI-referred traffic and attribute it to revenue.
## Docs
- [Install guide](https://traceten.com/docs/install): Add the snippet in under five minutes
- [API reference](https://traceten.com/docs/api): Endpoints, authentication, rate limits
## Optional
- [Blog](https://traceten.com/blog): Long-form writing on AI attribution
An H1 with the site name. A blockquote summary. H2 sections of links, each with a short description of why the link matters. One optional section, explicitly marked as skippable when context is tight.
That last detail is the part people miss. ## Optional is not a leftovers bin. It is an instruction about what to drop first under pressure. It is the only place in the format where you get to make the triage decision instead of the model.
What it is not
Three files at your site root, three different jobs
They are complements, not substitutes. Having a sitemap is not a reason to skip llms.txt, and vice versa.
| File | Audience | Purpose | Status |
|---|---|---|---|
| robots.txt | Any crawler | Permission: what may be fetched at all | RFC 9309, universally honoured by cooperative crawlers |
| sitemap.xml | Search crawlers | Enumeration: every URL, with lastmod | Long-standing convention, consumed by every search engine |
| llms.txt | Language models | Curation: the few things worth reading, and why | Community proposal, no documented consumer |
The confusion worth clearing up: llms.txt does not block anything. It carries no access-control semantics whatsoever. If you want to stop a crawler, that is robots.txt, and nothing you write in llms.txt will substitute for it.
There is a companion convention, llms-full.txt, which contains the actual content in Markdown rather than links to it. It is larger and can be genuinely enormous on a big documentation site, and it saves a model a second round trip. Some documentation platforms generate both automatically.
The honest state of adoption
Here is where the confident blog posts stop being reliable, so let us be exact about what is and is not established.
llms.txt: what actually happened
A short history, stripped of the claims nobody has substantiated.
Sep 2024
The proposal
Jeremy Howard publishes the llms.txt specification at llmstxt.org, framing it as a context-window problem rather than an SEO one.
Late 2024
Documentation platforms adopt
Developer-docs tooling begins generating llms.txt and llms-full.txt automatically, which is where most real-world files come from.
2025
Directories and checkers appear
A small ecosystem of validators and adoption trackers forms. Counts of sites with the file grow quickly, mostly on documentation sites.
2025
Google representatives say it is not used
Google staff state in public commentary (not in documentation) that Google does not consume llms.txt, comparing its trajectory to the keywords meta tag.
Now
Still no documented consumer
No major AI operator publishes documentation stating that llms.txt is read as a retrieval or ranking input. Fetches, however, do appear in server logs.
So the state of play is: a well-motivated proposal, meaningful adoption on the publishing side, and no confirmation from the consuming side. A file that many people write and no vendor admits to reading.
Why 'no documented consumer' is not the same as 'nobody reads it'
Operators document a fraction of what their systems do. A model with browsing capability that fetches a page can also fetch a well-known path at the same origin, and no announcement is required for that to happen. The claim "nothing reads it" is exactly as unevidenced as "everything reads it", unless you look at your own logs.
Turning the argument into a measurement
This is the part that changes the conversation. /llms.txt is a URL. Requests for it hit your server. You can count them.
You will not find them in Google Analytics: crawlers do not execute JavaScript, so browser analytics is structurally blind to the entire question. But your access logs have it, and one line of awk gets you a first answer:
awk '$7 ~ /^\/llms(-full)?\.txt/ {print $0}' access.log \
| grep -Eio 'gptbot|oai-searchbot|claudebot|claude-user|perplexitybot|chatgpt-user' \
| sort | uniq -c | sort -rn
What you want from that, structured properly, is three numbers.
Fetches
Requests for /llms.txt by declared agent, and how many of those verify by rDNS or published IP range
not everything claiming to be GPTBot is
Cadence
Days between fetches by the same agent: one-off probe or recurring visit
recurring implies use
Follow-through
Whether the same agent then fetches the URLs listed in the file
the real signal
The third one is the finding. A single fetch of llms.txt proves only that something enumerated a well-known path. Bots probe for these constantly. What would be genuinely informative is a fetch of /llms.txt followed, in the same session or the same day, by fetches of the specific URLs the file recommends. That is a consumer behaving as though the file were a map rather than a checkbox.
Nobody can hand you that answer for your site. Your logs can.
Two things that will hide the data from you
Most log-analysis and bot-tracking setups exclude .txt by default as static noise, which discards precisely this evidence. And CDNs cache /llms.txt aggressively, so origin logs can show near-zero fetches while the edge is serving hundreds. Check at the edge, or set a short TTL on that path specifically.
Should you write one anyway?
Yes, with the reasoning stated plainly rather than dressed up.
The cost is close to zero. An hour once, minutes to maintain, no risk of harm. There is no penalty from any search engine or model for having the file.
The exercise is worth more than the file. Writing llms.txt forces you to answer "which ten pages actually explain what we do, and can each one be understood without the nine others?" Most teams discover during that exercise that their answer pages are scattered, duplicated, or written to be read in an order nobody follows. That finding improves the site regardless of who reads the manifest.
The downside case is a text file nobody fetched. Which you will know, because you measured it.
What you should not do is treat it as a substitute for the things that are demonstrably working: clean HTML, fast responses, content that answers a question in one self-contained passage, and not blocking the search-index crawlers that decide whether you are citable at all.
Write the file because writing it makes your site clearer. Measure the file because measuring it is the only way anyone will ever settle the argument.
A template you can adapt
Keep it under a few dozen links. The whole point is curation; a file listing four hundred URLs has recreated the sitemap with worse ergonomics.
# Your Company
> One sentence on what you do and who for. Concrete, not aspirational.
## Product
- [What it does](https://example.com/product): Plain description of the core capability
- [Pricing](https://example.com/pricing): Plans, limits and what each tier includes
## Documentation
- [Quickstart](https://example.com/docs/quickstart): Working setup in five minutes
- [API reference](https://example.com/docs/api): Endpoints, auth, errors, rate limits
## Company
- [About](https://example.com/about): Who builds this and why
## Optional
- [Blog](https://example.com/blog): Long-form writing, safe to skip under context pressure
- [Changelog](https://example.com/changelog): Release history
Serve it as text/plain; charset=utf-8, keep it at the root, and reference it in robots.txt the way you would a sitemap. Then log the fetches. Six weeks later you will have something almost nobody in this debate currently has: evidence.
Frequently asked
A Markdown file at your website's root, proposed in September 2024, that gives language models a curated map of your most useful pages. It has a defined structure: an H1 site name, a blockquote summary, H2 sections of annotated links, and an optional section that models can skip when context is limited. It is a curation format, not an access-control file.
No major AI operator documents consuming llms.txt as a retrieval or ranking input, and Google representatives have said in public commentary that Google does not use the file. Fetches of /llms.txt do appear in server logs, so something requests it. Whether that request influences an answer is currently unproven either way, which is why measuring fetches and follow-on requests from your own logs beats arguing about it.
robots.txt controls access (which crawlers may fetch which paths) and is a formal standard under RFC 9309 that cooperative crawlers honour. llms.txt controls nothing. It is a community proposal that curates and describes your best content for language models. Writing an llms.txt will never block a crawler, and nothing in it substitutes for a robots.txt rule.
A companion convention containing the actual content in Markdown rather than links to it, so a model gets the text in one request instead of following a list of URLs. It is much larger (often megabytes on a documentation site) and is usually generated automatically by documentation platforms rather than written by hand.
Check server or CDN logs for requests to /llms.txt, grouped by user agent and date. Browser analytics cannot see these requests because crawlers do not run JavaScript. Look for three things: which agents fetch it, whether they return on a recurring cadence, and whether the same agent subsequently fetches the URLs the file recommends. The third is the strongest evidence that the file is being used rather than merely probed.
Sources & further reading
- 01The /llms.txt file proposal, llmstxt.org
- 02RFC 9309: Robots Exclusion Protocol, IETF
- 03Overview of OpenAI crawlers and user agents, OpenAI
- 04Google crawlers and fetchers overview, Google Search Central
- 05Build and submit a sitemap, Google Search Central