Llms.txt explained: the benefits, the limits, and our advice for the future of SEO

Knowing what the file is not matters just as much as knowing what it is. Llms.txt blocks no crawlers and no bots: every bot that wants to read your site can carry on doing so, with or without an llms.txt. Nor does it decide whether your content is used for AI training. There are other, properly documented mechanisms for that: OpenAI describes the user agent GPTBot with which you keep training crawlers out in robots.txt, and Google documents Google-Extended for the same purpose with Gemini.

So anyone who wants to shield their content arranges that in robots.txt and not in llms.txt. What we allow AI systems to do with the content of this site is a separate choice, and it is written out in our AI policy. The two files get lumped together in article after article, and that is exactly how site owners end up with a false sense of control.

It has no effect on your rankings in Google either. There is no indication at all that llms.txt counts in ordinary search results, and Google has hinted more than once that it does nothing with it. For AI Overviews, the AI answers at the top of Google, there is no documented relationship with llms.txt either: those answers come about through the ordinary crawling and indexing systems. What AI Overviews mean for your visibility, and what you can and cannot steer there, we have set out separately in the article on AI Overviews and local SEO.

Finally, the file replaces no SEO fundamentals. Structured data remains the way you tell machines what a page is, your metadata still decides how you appear in results, and your sitemap remains the file that is demonstrably read every day. A handy self test: our AI snippet previewer shows what your page looks like when a machine summarises it. If that picture is already messy, an llms.txt will not fix it; the page itself is next in line.

Start with the selection, because it decides everything else. Walk through your site and pick the pages that answer a real question or document real knowledge: guides, explainers, tools, documentation. A good rule of thumb is ten to twenty five links. Product pages, campaign pages and anything that will be out of date in three months you leave out.

For every link you write a one line description that says factually what a reader will find there. No marketing language: a language model has no use for the word leading, but it does have use for being told that a page holds a step by step plan for local visibility. The blockquote at the top describes your site the way an outsider would summarise it, not the way you sell it yourself.

Then the building. You can do that by hand in any text editor, but our free llms.txt generator is quicker: you enter your subjects and categories and the tool sets up the structure for you, including the right order of H1, blockquote and category blocks. Working in Claude or Codex? The downloadable skill at the top of this page does the same in your own environment: it asks you for your site details, selects strictly and hands over the file together with placement and maintenance instructions.

You save the result as llms.txt in the root of your domain, straight after your domain name. Then check in your browser that the file is shown as plain text and not as a download or an error page.

And then the part everybody skips: upkeep. An llms.txt is a promise about what your best content is, and promises go stale. Plan a ten minute check every quarter: do all the links still work, are the pages described still current, and is there new content that deserves a place. If in the meantime you want to know whether anything is happening at all with your file, look in your server logs to see whether the path /llms.txt is being requested and by which user agents. That is not science, but it is the only real measurement that exists at this moment. Everything beyond it is speculation, ours just as much as that of anyone who tells you otherwise.

The question behind llms.txt is bigger than the file itself: how do you make sure AI systems know your company, name it and cite it? More and more people put their question to an assistant instead of a search box, and the answer they get comes from training data and from sources fetched live. On the first you have hardly any influence, on the second you do: with content that really answers a question, a clean technical base and a consistent brand story.

Whether a model already knows your name today you can test in thirty seconds with the free AI brand checker: it puts a question about your industry to a language model and looks at whether your company name appears in the answer.

Want to follow it more systematically? Then look at the AI Index: our running measurement in which we track per industry which Dutch companies a language model names in answer to common questions, with the movement over the months. There you see the playing field of which llms.txt is a tiny part. Would you rather have that measured for your own company than read it off an index? That is what our service AI visibility does: first the measurement, then the work, and no promise of a place in an answer.

Because let us keep the proportions honest: whether AI names you depends on what has been written about you, how consistently your name and story appear online and how good your content is. Google also works with signals such as brand trust, which we wrote about in the article on preferred sources and brand trust. A small text file in your root does not change that balance of power, and we would rather say so than pretend otherwise.

What is shifting: AI is beginning not only to answer but also to act. Google is experimenting with agents that search and even buy on behalf of a customer, and that puts entirely different demands on your site than a classic visitor does, as we set out in agentic commerce.

In that playing field a readable, structured and honest web of pages is your best preparation. Llms.txt fits that direction, and that is why we think it is worth the ten minutes of work, even without proof that it is read today. Anyone who wants to go deeper, or wants to teach their own team how AI deals with web content, can turn to our AI training or the service AI and automation.

Is llms.txt an official standard?

No. It is a proposal that Jeremy Howard of Answer.AI published on llmstxt.org in September 2024. There is no standards body that maintains it and there is no obligation to support it. That does not make the file pointless, but it does mean you should treat it as an experiment and not as a fixed part of technical SEO.

Do ChatGPT, Claude or Google use my llms.txt?

There is no confirmation of that. OpenAI, Google, Anthropic and Perplexity have none of them made public that their production systems fetch and use the file. John Mueller of Google even compared the file to the keywords meta tag in 2025. Want to know whether anything is happening on your own domain? Look in your server logs to see whether the path /llms.txt is being requested and by which user agents.

Does llms.txt block AI bots or AI training?

No. Think of llms.txt as an invitation, not a lock. To keep crawlers or training bots out you use robots.txt, with user agents such as GPTBot from OpenAI or Google-Extended from Google. Those are documented mechanisms described by the makers themselves. Llms.txt has no part in that: it points at content, it shields nothing at all.

What is the difference between llms.txt and llms-full.txt?

Llms.txt is an index: a short markdown file with links to your best pages, each with a one line description. Llms-full.txt is the extended variant that holds the full content of those pages in one file. That second form mainly makes sense for documentation sites, where a language model can read the entire manual in one go. For an ordinary company website the index is enough.

Does llms.txt help my ranking in Google?

No. The file is not a ranking factor and Google has nowhere said that it counts. Expecting it to lift your rankings is the wrong reason to start. The things that demonstrably do count stay exactly where they were: technology, content that solves the search question, internal structure and links. Start there, and see llms.txt as a small extra for the long term.

How do I write an llms.txt myself?

Pick ten to twenty five pages with lasting value, split them into logical categories and write a factual one line description for each link. Above that put an H1 with your site name and a blockquote with a summary in two sentences. Save the lot as llms.txt in the root of your domain. Our free generator builds the structure for you, and the downloadable skill on this page does the same in Claude or Codex.

Our summary in three sentences. Making one is a decent, cheap experiment: do it if your content is worth it, maintain it like a promise, and expect nothing from it today. The real gain is in what you do along the way: selecting your best content forces you to see where your site is strong and where it is weak, and that insight is worth more than the file itself.

Start with the generator or the skill at the top, check your basics with the free scan, and read on in the knowledge base. And if the question is what all this costs when we look over your shoulder: every rate is out in the open on the pricing page, and for everything beyond that there is the contact page.