What llms.txt is
A Markdown file at the root of a website, proposed by Jeremy Howard in September 2024, that tells language models what the site is and which pages are worth reading. It is to AI systems roughly what robots.txt is to crawlers, except it says what to read rather than what to avoid.
Adoption by the engines is uneven and none of them has said it affects what they cite. Treat it as a ten-minute job that removes one reason to be misread, not as a ranking lever.
What a good one contains
A first line with the site's name as an H1, a one-sentence summary as a blockquote, and then short sections of links with a few words on each. Ours is at /llms.txt and follows exactly that shape.
# Northwind Hire
> Applicant tracking software for teams hiring at volume.
## Product
- [Features](https://example.com/features): what the product does
- [Pricing](https://example.com/pricing): plans and prices
## Learn
- [Docs](https://example.com/docs): setup and integrationsllms.txt and robots.txt are different files
robots.txt tells crawlers what they may fetch. llms.txt tells a model what your site is. A site can welcome GPTBot in the first and have nothing to say in the second, and the reverse: a perfect llms.txt behind a site-wide block is a file nobody reads. The checker tests both, and the AI crawler checker goes through the crawlers one by one.
Common mistakes
- Serving it as an HTML page. It must come back as plain text.
- Listing every page. Twenty good links beat four hundred.
- Marketing copy in the summary. One factual sentence: what, for whom.
- Blocking it for AI crawlers in robots.txt.
- Leaving the links alone after a redesign, so half of them 404.
Questions about llms.txt
Does llms.txt help SEO?
Not Google rankings, and nobody has shown it does. It is for language models and the products built on them: a short, reliable description of your site they can read in one request. It is cheap insurance, and it is one of the thirteen checks in our report because the sites that have it tend to have thought about the rest.
Do I need llms-full.txt as well?
No. llms.txt is the index; llms-full.txt is an optional single file with the full text of your key pages for models that want everything in one fetch. Add it if your documentation is large.
Where does the file go on WordPress, Webflow or Shopify?
At the root: yourdomain.com/llms.txt, served as plain text. On WordPress a static file in the web root or a small plugin; on Webflow, a redirect to a hosted text file or the site settings' custom files; on Shopify, a redirect to a file in Files. It has to answer with a text content type, not a page.
How often should it change?
When your main pages change. Check it after a redesign: the links in it are the first thing to rot.
That was one of thirteen checks
The full report runs the other twelve and asks four AI engines whether they name you at all.