What is llms.txt?
Also called llms.txt file, llms-full.txt.
llms.txt is a proposed convention in which a website publishes a plain markdown file at the path /llms.txt containing a short, curated description of the site and links to its most useful pages. The intent is to give language models a clean overview instead of leaving them to parse navigation and layout. It is a community proposal rather than an adopted standard.
The proposed format is deliberately simple: a top level heading with the site name, an optional summary paragraph, then sections of annotated links to key documents. A companion file containing the full text of those documents is sometimes published alongside it. Markdown was chosen because it is readable by both people and models and imposes almost no parsing burden.
The argument for it is that a rendered page wraps a small amount of substance in navigation, scripts, and styling, while a curated file states plainly what a site is and where the important material lives. It is often compared to a sitemap, but the audience differs. A sitemap enumerates URLs for crawlers, whereas this file is a written summary for a reader with limited context.
Adoption is the honest weak point. Publishing is easy and many sites have done it, documentation sites especially, but there is no public confirmation that major AI systems fetch it, and search representatives have said publicly that their systems do not use it. Treating it as a reliable way to influence how models describe you is not currently supported by evidence.
It is frequently confused with robots.txt, which it is not. One offers content and the other restricts access, and neither enforces anything on its own. A file that nobody fetches has no effect, so the practical case for publishing one today is that it is cheap, harmless, and useful to your own tooling, rather than that it is known to work.
Key points
- Markdown file at /llms.txt summarizing a site and linking key pages.
- Proposed in 2024 and maintained as a community convention.
- Widely published, especially by documentation sites.
- No public confirmation that major AI systems consume it.
- Offers content, and does not control access the way robots.txt does.
In practice
A documentation site publishes /llms.txt with one line describing the product, then a list of links: getting started, the API reference, the changelog, and a support page, each followed by a short note on what it covers. A tool that fetches this file receives roughly a page of text instead of a rendered site, and can then decide which single document to fetch in full.