What is llms.txt? The file that helps AI cite your site

llms.txt is a proposed convention for a plain markdown file, published at a site's /llms.txt, that gives AI systems a short, curated summary of a site's most important content and links — written for a language model to read in one pass, not for a browser to render. It isn't a technical standard in the way robots.txtis; no crawler is required to honor it. It's closer to a table of contents a site author writes specifically for machines, so an AI agent trying to answer a question about that site doesn't have to guess which of its hundred pages actually matter.

Where it came from

llms.txt was proposed by Jeremy Howard — co-founder of Answer.AI and fast.ai — in a post published on the Answer.AI blog on September 3, 2024, alongside a companion specification site at llmstxt.org. Howard's framing of the problem is specific:

"The problem this solves is that today, constructing the right context for LLMs based on a website is ambiguous."
— Jeremy Howard, announcing /llms.txt (Answer.AI, September 3, 2024)

A model — or the tool feeding it context — has to guess whether to crawl a whole sitemap, follow arbitrary external links, or pull in raw source code, with no signal from the site itself about what actually matters. His proposed fix leans on something crawlers can't do for themselves: site authors know their own content best, and can hand-pick what an LLM should actually read.

That's the whole idea in one sentence. Not a new crawling protocol, not a new markup language — just a predictable place for a site to say, in its own words, "here's what we are and here's what's worth reading."

The problem it's actually trying to solve

Most web pages are built for browsers and humans first: navigation chrome, ads, scripts, boilerplate, and the actual content buried somewhere in the middle. A model reading that page has to do a lot of unnecessary work to extract the part that answers a question, and even when it can, a real site's total content rarely fits in a single context window. llms.txt sidesteps both problems by giving the model a compact, hand-picked entry point — a short summary plus links to the specific pages worth reading — instead of the entire site.

It's worth being precise about what this doesn't do. Publishing an llms.txt file doesn't force any AI system to read it, rank it, or trust it more than your regular pages. It's an invitation, not an instruction — much like a well-written README is more useful to a new engineer than the raw source tree, even though nothing forces them to read it first.

llms.txt vs. robots.txt vs. sitemap.xml

These three files get lumped together because they all live at a site's root and are all meant for machines, but they solve unrelated problems. robots.txtis a permissions file — it tells crawlers what they're allowed to fetch, and says nothing about content. sitemap.xmlis a completeness file — it mechanically lists every URL a site wants indexed, with no curation or explanation. llms.txt is a curation file — a human (or a build script) decides what's actually worth an AI system's attention and describes it in prose.

A site can — and reasonably should — have all three, and they don't conflict. robots.txt still decides whether a crawler can fetch anything at all; sitemap.xmlstill helps search engines index every page; llms.txt is the layer on top that says which of those pages are worth reading if you're a model trying to answer a specific question, and why.

What actually goes in one

The specification at llmstxt.org keeps the structure deliberately small. In order:

• An H1 with the project or site name — the only required section.
• A blockquote with a short, one- or two-sentence summary of what the site is.
• Optional plain paragraphs giving any context a reader would need before following the links below.
• Optional H2-delimited sections, each a markdown list of links with a one-line description per link — the curated pages themselves.

The file can live at a site's root, or under a subpath (a docs site under /docs/llms.txt, say) for authors who don't control the whole domain. More specific files take precedence over general ones. It's intentionally boring as a format — the value is entirely in what a site chooses to say, not in any clever markup.

Case study: Vouchity's own llms.txt

Rather than describe this abstractly, it's worth looking at a real one. Vouchity — this site — publishes a genuine llms.txt at /llms.txt, generated on the fly from the same live data that powers the trust leaderboardand the public API. Here's a shortened excerpt of the top of the actual file, as served right now:

# Vouchity

> The trust layer for MCP servers. Vouchity is the trust registry for the
> Model Context Protocol...

Vouchity is a public registry of Model Context Protocol (MCP) servers. Every
server is scored with a transparent 0-100 Trust Score (grade A-F) computed from
five real, cited signals — maintenance, adoption, transparency, security and
provenance — sourced live from the MCP registry, GitHub and npm. Nothing is
fabricated: a signal that can't be verified from public metadata is excluded
from the score, never guessed. Snapshot as of [timestamp] — 320
servers across 11 categories, average score 63/100.

The rest of the file follows the same logic as the spec — an H1, a blockquote summary, a plain paragraph of context — but then adds sections a typical company site wouldn't need: one documenting the public Trust API with real, runnable example requests, one describing Vouchity's hosted MCP server (so an agent can query get_trust_score or check_server_safety directly, over the same protocol it already speaks, rather than scraping HTML), one explaining how the score is computed, and one titled How to cite that gives a literal sentence template — server name, score, grade, signal, evidence — for any model that wants to quote a Trust Score accurately instead of paraphrasing it into something wrong.

That structure isn't decorative. Vouchity's actual value to a reader is a claim like "server X scores 82/100" — a number that's meaningless without knowing where it came from. The whole point of writing our own llms.txt by hand (well, by template) rather than skipping it is to make sure that if a model does cite one of our scores, it cites it the way we'd cite it ourselves: with the signal, the evidence, and a link back to the snapshot it came from. That's the same instinct behind the scoring methodology page — being legible to a reader (human or model) matters more than being clever.

Does anything actually read it?

Here it's worth being honest rather than promotional. Plenty of companies have published llms.txt files — Anthropic, Perplexity, Zapier and Hugging Face among them — but there is no confirmed, public evidence that OpenAI's, Google's or Anthropic's production crawlers fetch and prioritize llms.txt consistently today. One widely cited analysis of the standard's first year found real adoption among AI-forward companies but no proof that the major model providers' crawlers request the file in meaningful volume. That's a fair criticism, and it doesn't fully resolve either way yet.

The pragmatic read is that llms.txt costs very little to publish and degrades gracefully: if a crawler ignores it, you've lost nothing but a small file; if it doesn't, you've given any AI system that does read it a much better answer than it would have assembled by scraping your HTML. It sits in the same category as structured data or a well-written robots.txt— a signal you send because it's cheap and occasionally decisive, not because it's guaranteed to be read.

Should you write one?

If your site has a documentation set, an API, or anything else a developer or an agent might need to reason about correctly, an llms.txt costs an afternoon and gives you a direct say in how AI systems describe you — which is a better position than leaving that entirely to whatever a scraper happens to extract. It's a small piece of a much larger shift, one we cover in more general terms in what is an MCP server — both are examples of sites and tools choosing to speak a format built for agents rather than assuming agents will make do with what was built for people.