llms.txt: what it is and whether you need it

llms.txt is a plain Markdown file placed at the root of a site, at /llms.txt, that gives a language model a short, curated map of the site: what it is, and which pages matter, with a one-line note on each. It was proposed in 2024 as a community convention. It is not a web standard, and it is not something search engines have agreed to read.

That last point decides how much effort it deserves, so it comes first.

What is and is not known about who reads it

• No major search engine has committed to using llms.txt for crawling, indexing or ranking. Google representatives have said publicly that Google Search does not use it, and the other large assistant vendors have not documented support.
• Some tools do read it: coding assistants and documentation helpers that fetch a project's llms.txt to find the right docs pages, and agents that are explicitly pointed at it by a user. This is where the format came from, and where it is most useful today.
• Server logs on many sites show occasional requests for /llms.txt, but a request is not proof that a system uses the content in any particular way.

So llms.txt does not improve rankings and does not make an AI assistant mention a brand. Treat it as a small, cheap courtesy file that may help some tools now and costs almost nothing to maintain. If a consultant sells it as an "AI SEO" fix, that claim is not supported by anything the vendors have published.

What it does not replace

File	Job	Who relies on it
robots.txt	Says which crawlers may fetch which paths	Search engines and AI crawlers that respect the standard
sitemap.xml	Lists every URL you want indexed, with last-modified dates	Search engines
llms.txt	A short, human-written summary and reading list for language models	Some tools and agents, on a voluntary basis

llms.txt has no permission semantics. Listing a page in it does not allow a crawler that robots.txt blocks, and leaving a page out does not hide it. Access control stays in robots.txt; discovery stays in the XML sitemap.

The format

The proposal describes a simple structure in Markdown:

1. An # heading with the name of the site or project. This is the only required part.
2. A blockquote (>) with a one- or two-sentence summary.
3. Optional paragraphs with any context a model should know.
4. ## sections, each a list of links in the form - [Title](URL): short note.
5. Optionally a section named ## Optional for links a tool can skip when it is short on context.

A realistic example for a small business site:

# Northfield Dental

> Family dental clinic in Northfield with two dentists and a hygienist.
> Open Monday to Saturday; accepts most major insurance plans.

## Services
- [General check-ups](https://northfielddental.example/check-ups): what a visit includes and how long it takes
- [Teeth whitening](https://northfielddental.example/whitening): in-chair and take-home options, typical price range
- [Emergency appointments](https://northfielddental.example/emergency): same-day slots and what counts as an emergency

## Practical
- [Prices](https://northfielddental.example/prices): fees for common treatments, updated each January
- [Insurance](https://northfielddental.example/insurance): plans accepted and how billing works
- [Contact and directions](https://northfielddental.example/contact)

## Optional
- [Blog](https://northfielddental.example/blog): oral-health articles

The full specification also suggests offering Markdown versions of key pages at the same URL with .md appended. That is useful for documentation sites; for a typical business site it is usually not worth the extra build step.

Writing one in fifteen minutes

1. Write the summary as you would explain the site to a new colleague: who it serves, what it offers, where. Facts, not slogans.
2. Pick ten to thirty pages that answer the questions customers actually ask: services or products, prices, policies, contact, the best guides. Do not dump the whole sitemap; a curated list is the point.
3. Give each link a note that says what the page answers, so a tool can choose without opening every link.
4. Use absolute https:// URLs on your canonical host.
5. Serve it as plain text. Check that /llms.txt returns status 200 with Content-Type: text/plain or text/markdown, not your HTML homepage. Single-page apps often answer unknown paths with the app shell, which looks like a valid page in a browser but is not the file.

On WordPress or a similar CMS, the simplest route is to upload a static file to the web root. Several plugins generate one automatically; check what they produce, because auto-generated files tend to list every post, which defeats the purpose of a short curated map.

Keeping it honest

An outdated llms.txt is worse than none, because it points tools at prices, services or pages that no longer exist. Add it to whatever checklist you use when prices change or pages are removed. Do not put anything in it you would not publish on the site itself, and do not use it to push claims the visible pages do not support; tools that read it can also read the pages.

Should you bother?

• Documentation sites, APIs, developer tools: yes. That audience uses assistants that read it.
• Content sites and businesses: optional. It takes little time, and the summary exercise often exposes that the site itself never states plainly what it does, which is worth fixing on the homepage anyway.
• If time is limited: make sure AI crawlers are not blocked (robots.txt for AI crawlers), that the main content is in the HTML without JavaScript (what crawlers see on JavaScript sites), and that key pages carry structured data. Those affect what crawlers can read; llms.txt does not.

The free AI bot checker reports whether /llms.txt exists, how many lines it has and how it starts. A missing file is shown for information only and never lowers the score on the SEO checker.