Do any AI systems actually read llms.txt today?

Adoption is real but narrow, and concentrated in developer documentation where coding agents follow it to find API references. We publish one and would not claim it has measurably changed anything for us. Anyone promising traffic from it is selling something.

How is llms.txt different from robots.txt?

Opposite purposes. robots.txt tells crawlers what to avoid; llms.txt points them at what matters. One is a restriction written for machines in a bespoke syntax, the other is a curated reading list written in markdown.

Does publishing one give AI systems permission to train on my site?

No, and this worries people unnecessarily. It grants nothing and changes no licence. It is a signpost, not a consent form. If you want to restrict training use, that conversation happens in robots.txt and your terms, not here.

What is the difference between llms.txt and llms-full.txt?

The first is an index — links with short descriptions. The second inlines the entire content of those pages into one file. Large sites publish both, though a full file can run to megabytes and few agents fetch something that size.

Where exactly does the file go?

At the root, served as plain text or markdown at /llms.txt. Serving it with the wrong content type is the most common implementation mistake — a file returned as text/html gets treated as a page rather than a resource.

Should it link to HTML pages or markdown versions?

Markdown versions where you have them, because that removes a conversion step for whatever reads it. The convention is to serve the same URL with a .md extension. Linking normal HTML pages is fine and still better than nothing.

How long should the file be?

Short enough that a person could read it. Twenty well-chosen links beat two hundred, because the entire value is curation. A file listing every page is just a sitemap in a different format and adds nothing.

Does it help with Google AI Overviews?

There is no evidence that Google uses it, and we are not going to imply otherwise. Overviews draw on the ordinary search index. Treat llms.txt as being aimed at agents that fetch your site directly.

Is there a schema or validator?

The specification is loose by design — a heading, an optional summary, then sections of links with descriptions. That looseness means most files pass any check, so validators offer less reassurance than they appear to.

Is it worth the effort right now?

For documentation sites and tool directories, yes — writing one takes an hour and forces useful thinking about what your best pages are. For a small marketing site, honestly it is optional and nobody will notice either way.

What Is llms.txt? The Proposed Standard, and Whether It Does Anything Yet

An AI agent lands on your site and has to guess which of four hundred pages matters. It picks your pricing page and a blog post from 2019. Meanwhile the documentation you spent months on goes unread, because nothing told it where to look.

What Is llms.txt, in Practical Terms?

A markdown file at your site root that lists your most useful pages with a sentence explaining each. That is the whole proposal.

It was put forward by Jeremy Howard in 2024 as a convention rather than a formal standard. The structure is deliberately minimal — an H1 with your project name, an optional summary, then sections of links with short descriptions.

The idea is that an agent fetching your site gets a curated starting point instead of crawling blindly. It is a reading list, not a restriction.

We publish one at /llms.txt. Being straight about the outcome: we cannot point to a measurable change from doing so. It took an hour, it forced a useful conversation about which pages actually matter, and that was worth it regardless.

llms.txt vs robots.txt — Opposite Jobs

One says stay out, the other says start here. They are frequently confused because of the shared naming, and they have almost nothing in common.

	robots.txt	llms.txt
Purpose	Restrict crawling	Point at good content
Format	Bespoke directive syntax	Plain markdown
Status	Long-established convention	Recent proposal
Audience	All crawlers	AI agents specifically

A useful clarification: publishing llms.txt grants nothing. It does not permit training, it does not change your licence, and it does not waive any right. It is a signpost.

If restricting AI training is your goal, that conversation belongs in robots.txt and your terms of service. Our guide to the file it is often compared to covers what that side actually governs.

How to Create llms.txt Without Overthinking It

Start with the pages you would send a new colleague to. That instinct produces a better file than any generator.

The structure is straightforward:

# Your Project

> One sentence on what this is and who it serves.

## Documentation

- [Getting started](https://example.com/docs/start.md): Install and first run.
- [API reference](https://example.com/docs/api.md): Every endpoint with examples.

## Guides

- [Deployment](https://example.com/guides/deploy.md): Production checklist.

Two details that matter. Serve it as plain text or markdown — returning it as text/html is the most common implementation mistake, and it gets treated as a page rather than a resource.

And link to markdown versions where you have them, conventionally the same URL with .md appended. That saves whatever reads it a conversion step. If you do not have markdown copies, producing the markdown files it points at takes a few minutes per page.

Keep it short. Twenty well-chosen links beat two hundred, because curation is the entire point. A file listing every page is a sitemap wearing a different hat.

Who Honours AI Crawler Standards Today

Adoption is real, narrow, and concentrated in documentation. That is the honest position.

Developer documentation platforms have taken it up most visibly, because coding agents genuinely benefit from being pointed at an API reference rather than a marketing page. Several documentation hosts now generate the file automatically.

Outside that niche it is quieter. There is no published evidence that the major search engines use it, and we are not going to imply otherwise to make the case sound stronger.

What we would say: the cost is an hour, the downside is nothing, and the thinking it forces about which pages matter has value on its own. That is a reasonable trade even if adoption stalls.

What It Cannot Do

It does not make your content reachable if the content is not reachable.

A site that renders everything client-side still returns an empty shell to anything that fetches it, llms.txt or not. Pointing an agent at a URL it cannot read helps nobody — the mechanics are in why some pages come back empty.

It also does not improve pages that are thin. A curated list of weak content is a curated list of weak content, and no amount of signposting changes that.

And it carries no enforcement whatsoever. An agent is free to ignore the file entirely, exactly as crawlers are free to ignore robots.txt.

Fitting It Into Something Larger

Treat it as one small piece of making your content machine-readable.

Structured data still does more work than llms.txt for most sites, and it has a decade of proven support behind it. Running your pages through an on-page SEO analyzer will usually surface higher-value fixes first.

The broader question of shaping content for machines rather than people is covered in getting a page into a model's context, which is the consumer side of the same problem.

Our position, stated plainly: publish one if you maintain documentation or a tool directory. Skip it if you run a small marketing site and nobody will notice. Anyone telling you it will bring traffic is guessing.