What llms.txt is
It is a proposal, published by Jeremy Howard in September 2024 and revised in August 2026. The idea: put a Markdown file at /llms.txt that gives a short overview of the site and links to clean text versions of its pages (llmstxt.org).
The proposal is clear about the purpose. It says the file is "used on demand, when an agent needs information about a topic while assisting a user", and that it has "different purposes" from robots.txt. It is a map for an AI agent that is already on your site. The proposal describes a resource for agents, not a ranking mechanism.
What Google says
Google's guide to AI features in Search lists llms.txt among the things you can ignore (Google Search Central, AI optimisation guide). In its words:
- "You don't need to create new machine readable files, AI text files, markup, or Markdown to appear in Google Search (including its generative AI capabilities), as Google Search itself doesn't use them."
- "It's completely fine if you decide to create and maintain LLMS.txt files (or other similar files) for other services or systems that use these files. Doing so will neither harm nor help your site's visibility or rankings in Google Search, as Google Search ignores them."
Google's wording includes "its generative AI capabilities", so this is not only about the classic results.
What the other AI companies say
The crawler pages from OpenAI, Anthropic and Perplexity document robots.txt controls for their automated crawlers. OpenAI says those rules may not apply to ChatGPT-User, and Perplexity says Perplexity-User generally ignores them. None of the three says its crawler reads a site's llms.txt.
Two of those companies do publish an llms.txt of their own. OpenAI's and Perplexity's developer documentation both point AI agents to one as an index of their docs. That fits the proposal: a guide for an agent reading documentation.
What our own logs show
We publish both /llms.txt and a longer /llms-full.txt on this site. We summed Vercel's request counts for both files, and for robots.txt, on searchintel.tech across four seven-day windows, from 12 October to 9 November 2026 (UTC). The bot names are Vercel's labels. Every response status is included, redirects too.
| File | Requests | Vercel bot label |
|---|---|---|
| /llms.txt | 146 | 101 with no bot label, PetalBot 26, Meta's external agent 13, Amazonbot 4, GPTBot 1, Claude-User 1 |
| /llms-full.txt | 21 | 19 with no bot label, GPTBot 1, Meta's external agent 1 |
| /robots.txt | 3,659 | Includes OAI-SearchBot 300, ClaudeBot 201, Googlebot 186, PerplexityBot 136, Claude-User 122 |
Three things stand out.
- Several AI bot labels are missing from the llms.txt requests. No request for /llms.txt carried the label of OAI-SearchBot, ChatGPT-User, ClaudeBot, PerplexityBot, Googlebot or Bingbot. In the same four weeks, 300 requests for robots.txt carried the OAI-SearchBot label.
- Three labels make up most of the labelled requests. PetalBot, Meta's external agent and Amazonbot account for 43 of the 45.
- Most requests had no bot label. 101 of the 146 have an empty label, so the sender is not identified. The label is Vercel's, and a user-agent can be faked.
This is one website over four weeks. The logs record requests, response statuses and Vercel's labels. They do not show what a model did with the file, and they do not measure visibility.
Should you publish one?
This is our view.
- If you publish developer documentation, yes. The proposal says the files are used most heavily for software documentation, and OpenAI's and Perplexity's documentation pages link to their own llms.txt indexes.
- If you want to be named in AI answers, put the effort elsewhere. Check that robots.txt lets the AI search crawlers in. Our four-week sample records requests for it labelled OAI-SearchBot and PerplexityBot. Then work on pages that state one thing clearly (how to write a page AI will cite) and on the third-party sources AI engines cite (AI citation strategy).
- If you already have one, keep it. It costs little. Our longer file is rebuilt by a script when the site changes.
How to check your own site
- Open your server or hosting logs for the last four weeks.
- Filter for requests to /llms.txt and count them by user-agent.
- Do the same for /robots.txt.
- Compare the two lists. Our guide to tracking AI crawlers has the bot names to look for.
Frequently asked questions
Does Google use llms.txt?
No. Google says Google Search does not use these files, including for its generative AI features, and that having one will neither harm nor help.
Does ChatGPT read llms.txt?
OpenAI's crawler documentation tells site owners to use robots.txt and does not say its crawlers read a site's llms.txt. In four weeks of our logs, one request for /llms.txt carried the GPTBot label and none carried OAI-SearchBot or ChatGPT-User.
Is llms.txt the same as robots.txt?
No. robots.txt tells crawlers what they may access. llms.txt is a proposed overview of a site for an AI agent to read. The proposal itself says the two have different purposes.
Will llms.txt hurt my site?
Google says it will neither harm nor help in Google Search. Keep it accurate, so an agent that reads it gets current information about your site.