llms.txt Explained: What It Is, Whether AI Search Engines Use It, and How to Add One
llms.txt is a proposed convention, not a standard: a Markdown file placed at a site’s root (for example, yoursite.com/llms.txt) that gives large language models a short description of the site and a curated list of links to its most useful pages. Jeremy Howard published the proposal in September 2024. As of 2026, no major AI search engine has said it uses the file to decide what to retrieve, cite or rank. Google’s documentation says explicitly that no AI text file is needed to appear in AI Overviews or AI Mode, Google’s John Mueller has compared it to the keywords meta tag, and an Ahrefs analysis of 137,210 domains in May 2026 found that 97 percent of published llms.txt files received no requests at all. The file is cheap to add, harmless when accurate, and occasionally useful to coding agents and tools that read documentation. It is not a lever for AI visibility. If you add one, do it in ten minutes with a plugin, keep it accurate, and spend the rest of your time on the things that do decide whether AI engines cite you.
Key takeaways
- llms.txt is a proposal from September 2024: an H1, a summary blockquote, and H2 sections of Markdown links, with an “Optional” section for links an agent can skip.
- Google states there are no new machine-readable or AI text files needed for AI Overviews or AI Mode. OpenAI, Anthropic and Perplexity have not announced support.
- Ahrefs’ May 2026 server-log study: 28 percent of domains publish one, 97 percent of those files got zero requests, and AI retrieval bots such as OAI-SearchBot and PerplexityBot accounted for 1.1 percent of the requests that did arrive.
- Yoast SEO can generate the file from a settings toggle; dedicated WordPress plugins do the same with more control. Hand-writing it takes a few minutes.
- What moves AI citations is crawler access, answer-first content, consistent entity data and independent coverage. Treat llms.txt as housekeeping.
What is llms.txt?
llms.txt is a plain-text Markdown file that a website publishes at its root to tell AI systems what the site is and where its most important content lives. The llmstxt.org proposal, written by Jeremy Howard and first published on September 3, 2024, describes it as “a proposal to standardise on using an /llms.txt file to provide information to help agents use a website.” The motivation is practical: language models have limited context windows, and HTML pages carry navigation, scripts and boilerplate that waste that space. A concise index in Markdown lets an agent find the right pages without crawling everything.
The proposal also suggests that each important page offer a Markdown version at the same URL with .md appended (page.html.md) or substituted (page.md), discoverable through a rel=”alternate” link with type text/markdown. Many documentation sites also publish a companion /llms-full.txt that concatenates the full content of every linked page into one file; that practice grew up around the proposal rather than being defined in it.
It is worth being precise about what the file is not. It is not robots.txt, which tells crawlers what they may fetch. It is not a sitemap, which lists every URL for indexing. And it is not a ranking signal: nothing in the proposal claims that, and no engine has claimed it either. It is a reading guide, written by the site owner, that an agent may or may not consult.
Do ChatGPT, Google, Perplexity or Claude actually use it?
The short answer is that none of them has said so, and the available server-log evidence says their search crawlers rarely ask for it.
Google’s documentation on AI features and your website is direct: “You don’t need to create new machine readable files, AI text files, or markup to appear in these features,” and “There are no additional requirements to appear in AI Overviews or AI Mode, nor other special optimizations necessary.” A page needs to be indexed and eligible for a standard snippet. How those features choose what to cite is covered in how to rank in Google AI Overviews and AI Mode.
John Mueller of Google went further in a Reddit thread reported by Search Engine Journal in April 2025: “AFAIK none of the AI services have said they’re using LLMs.TXT, and you can tell when you look at your server logs that they don’t even check for it. To me, it’s comparable to the keywords meta tag.” His point is that the file is a site owner’s claim about the site, which an engine has no reason to trust over the site itself. Gary Illyes of Google was more direct at a Search Central Deep Dive session in July 2025, saying Google does not support llms.txt and is not planning to, as reported by Search Engine Land, and we keep the file anyway because it costs nothing and other crawlers can read it.
OpenAI, Perplexity and Anthropic
OpenAI’s crawler documentation describes OAI-SearchBot, GPTBot and ChatGPT-User and how robots.txt controls each; it does not mention llms.txt. Perplexity’s bot documentation covers PerplexityBot and Perplexity-User and likewise says nothing about the file. Anthropic’s crawler article describes ClaudeBot, Claude-User and Claude-SearchBot without reference to it. Anthropic does publish an llms.txt for its own developer documentation, as do many software companies, which is the one context where the format has clear adoption: documentation meant to be read by coding assistants.
What the server logs show
In June 2026 Ahrefs published the largest log analysis to date, covering 137,210 domains that received traffic in May 2026. Twenty-eight percent of those domains published an llms.txt file, which the authors note is an upper bound given their customers skew technical. Of the files that existed, 97 percent received zero requests in the month. Among the requests that did arrive, 96 percent came from bots, and the largest categories were SEO audit tools (21.7 percent), unidentified bots, general crawlers and tech-profiling tools, each outranking any single AI bot. AI bots combined accounted for 19.5 percent of requests; AI retrieval bots, the group that includes OAI-SearchBot and PerplexityBot, accounted for 1.1 percent. Slackbot fetched llms.txt files more often than PerplexityBot. Requests for files that did not exist were about 98 percent human, with zero AI bot requests, which the authors read as evidence that AI tools do not go looking for the file when it is absent.
The study measured fetches, not use, and could not see what Gemini or any model does with content once fetched. But the direction is clear enough: the search-side crawlers that decide citations are not reading this file in any meaningful volume.
Why does the idea keep spreading if engines ignore it?
Three reasons, none of them bad faith.
It works for its original audience. The proposal came out of the developer-tooling world, where coding assistants read documentation in bulk. Ahrefs found that a coding agent out-fetched every AI search bot except GPTBot in its sample. If your product has technical docs that developers paste into AI assistants, an llms.txt and .md versions of pages are a real service to those users.
It is cheap and visible. A toggle in an SEO plugin produces a file you can see. That makes it an easy line item for agencies and an easy box to tick, in a field where the signals that matter (entity consistency, independent coverage, extractable content) are slower and harder to show in a screenshot.
Correlation gets mistaken for cause. Sites that add llms.txt early tend to be technically well-run sites with good documentation, and those sites get cited. The file did not cause the citations. The original generative engine optimization research, which measured what actually lifts visibility in generated answers, found the gains in statistics, quotations and cited sources on the page, not in metadata about the page. Our pillar on what generative engine optimization is sets out that evidence.
What goes in an llms.txt file?
The format defined at llmstxt.org is short and strict about order. The only required element is the H1.
| Element | Markdown | Required? | What to write |
|---|---|---|---|
| Site or project name | # Name |
Yes | The entity name exactly as it appears on your site and profiles |
| Summary | > One or two sentences |
Recommended | What the organization does, for whom, where; the same description you use everywhere else |
| Detail paragraphs | Plain Markdown, no headings | Optional | Context an agent needs to use the links well: scope, audience, how to interpret the sections |
| File lists | ## Section then - [Title](URL): note |
Optional | Your most useful pages grouped by purpose: services, guides, pricing, policies, press |
| Optional section | ## Optional |
Optional | Secondary links an agent can skip when it needs less context |
A minimal, correctly formed file for a service business looks like this:
# Example Agency
> Example Agency is a digital PR firm in New Jersey that secures editorial coverage, manages online reputation and improves AI search visibility for founders and brands.
Pages are listed in order of usefulness. Service pages describe what is included and how engagements work; guides are long-form explanations written for buyers.
## Services
- [Guaranteed press placements](https://www.example.com/guaranteed-pr-placements/): how placements work, what is guaranteed, approval process
- [Reputation management](https://www.example.com/reputation-management/): removal, suppression and authority content
- [AI search visibility](https://www.example.com/ai-visibility/): how we measure and improve AI citations
## Guides
- [What is generative engine optimization](https://www.example.com/what-is-geo/): definition, how it differs from SEO
- [How to get a knowledge panel for a business](https://www.example.com/knowledge-panel-business/): entity signals and claiming
## Optional
- [Contact](https://www.example.com/contact/)
- [Publications list](https://www.example.com/publications/)
Keep it to the pages you would want an assistant to read first. A file that lists every URL on the site is a worse sitemap, not a better index.
How to add llms.txt on WordPress
You have three options, in increasing order of control.
Option 1: Yoast SEO
Yoast’s help article gives the path: Yoast SEO, then Settings, then Site features, then AI tools, then LLMS.txt. Turn on the toggle and save. The plugin generates a file highlighting what it judges to be your most important, up-to-date content, and you can switch to manual page selection in the customization settings to choose the pages yourself. Yoast notes the feature is not available on multisite installations and that a “couldn’t be auto-generated” notice usually means a filesystem permissions problem.
Option 2: A dedicated plugin
Plugins built for this one job give more control over sections, descriptions and update frequency. The Website LLMs.txt plugin, with more than 50,000 active installations at the time of writing, builds the file from the post types you select, pulls titles and meta descriptions, excludes anything marked noindex and anything restricted to members, respects the settings of Yoast, Rank Math, SEOPress and AIOSEO, and regenerates on a schedule or when content changes. Check that whichever plugin you choose does not list private, thin or outdated pages; the file is only useful if everything in it is something you would show a prospect.
Option 3: Write it by hand
Create a file named llms.txt in the root of your web server (the same folder as robots.txt), paste the Markdown, and confirm it loads at yoursite.com/llms.txt with a text/plain or text/markdown content type. For a small site with a dozen important pages, this takes less time than configuring a plugin and produces a cleaner result. Add a line to your editorial checklist to update it when you publish or retire a key page.
Whichever route you take, do not block the file in robots.txt, do not put anything in it that is not on the public site, and do not use it to describe the site in terms that differ from the site itself. The one way this file could hurt you is if an engine ever did compare it with the pages and found the claims did not match.
What llms.txt will not do
Set expectations before anyone on your team reports it as a win.
| Expectation | Reality | What actually does this |
|---|---|---|
| Gets you into Google AI Overviews or AI Mode | Google says no AI text file is needed or used; eligibility is indexing plus snippet eligibility | Indexed, snippet-eligible pages that answer sub-queries directly on a site Google trusts for the topic |
| Gets you cited by ChatGPT or Perplexity | Neither has announced support; their search crawlers accounted for 1.1% of llms.txt requests in Ahrefs’ study | Allowing OAI-SearchBot and PerplexityBot, ranking for the rewritten queries, extractable passages |
| Controls what AI says about your brand | Models reconcile many sources; a self-description carries little weight against them | Consistent entity facts across your site, LinkedIn, Wikidata, Crunchbase, reviews and press |
| Blocks AI training on your content | llms.txt grants nothing and forbids nothing | robots.txt rules for GPTBot, ClaudeBot and similar training crawlers |
| Gets your brand recommended for “best X” questions | Recommendation answers cite third parties, not the brand’s own files | Independent coverage, reviews and comparisons that name you |
| Helps coding assistants use your documentation | This is the one use with real adoption | llms.txt plus .md versions of docs pages |
The last row is the honest case for the file. If you publish developer documentation, add it. If you run a service business, add it because it costs nothing, then move on to the work that decides citations. That work is described in how ChatGPT and Perplexity choose their sources, and the reason on-site signals alone are not enough is in why strong SEO no longer guarantees AI answer visibility.
What to do instead, in order
If the goal is to be cited and recommended by AI engines, this is the sequence that has evidence behind it.
- Open the crawler path. Allow OAI-SearchBot, PerplexityBot and Claude-SearchBot in robots.txt and at the firewall or CDN. Keep training-crawler decisions separate; they do not affect search appearance.
- Make the answer the first sentence. Every important page and every H2 should open with a complete, specific answer. Add the number, the date and the source. This is what the controlled research found moves visibility.
- Publish structured data that matches the page. Organization markup with sameAs links to your profiles, Article markup with author and dates, FAQPage where you have real FAQs. Google says structured data must describe visible content.
- Make the entity consistent. Same name, description, founding details and leadership on your site, LinkedIn, Crunchbase, Google Business Profile, review sites and Wikidata where eligible.
- Earn independent corroboration. Coverage, interviews and expert quotes in recognized outlets are what a model cites when it names you as an option. The mechanics are in how press coverage gets your brand recommended by ChatGPT.
- Measure monthly. A fixed prompt set across engines, logging mentions, citations and accuracy. See AI visibility tools and metrics for the method.
- Then add llms.txt. Ten minutes, accurate, maintained. Last on the list because it is the least likely to change anything.
Where Viral Spike fits
Viral Spike Marketing’s AI search visibility service covers the technical layer above as part of a baseline audit, including crawler access, structured data and whether an llms.txt file is worth maintaining for your site, but the engagement is built around what the evidence says moves citations: extractable content, consistent entity data and independent coverage. The coverage layer uses guaranteed editorial placements in recognized outlets, where you approve the draft, the article is published and indexed on Google, and the engagement is refunded in full if it never runs. For brands that want press, search and AI visibility run as one program, the brand authority service combines them. If you want to know where you stand before deciding, contact us and we will run the baseline prompts.
Add the file, then forget about it
Turn on the toggle in your SEO plugin or write the twenty lines by hand, check that the file loads, and put a reminder in your publishing checklist to keep it current. Then open your robots.txt and your firewall logs and confirm the search crawlers that actually decide citations can reach you. That second check will do more for your AI visibility than the first, and it takes about as long.
Frequently asked questions
What is an llms.txt file?
llms.txt is a Markdown file placed at a website’s root that gives AI systems a short description of the site and a curated list of links to its most useful pages. Jeremy Howard proposed it in September 2024. The format requires an H1 with the site name, and recommends a summary blockquote, H2 sections of Markdown links, and an Optional section of secondary links. It is a proposal, not an adopted standard.
Does Google use llms.txt for AI Overviews?
No. Google’s documentation on AI features states that you do not need to create new machine-readable files, AI text files or markup to appear in AI Overviews or AI Mode, and that there are no additional requirements beyond being indexed and eligible for a snippet. Google’s John Mueller has compared llms.txt to the keywords meta tag, a self-description with no verification.
Do ChatGPT or Perplexity read llms.txt?
Neither OpenAI nor Perplexity has announced support, and their crawler documentation does not mention the file. Ahrefs’ May 2026 analysis of 137,210 domains found that 97 percent of published llms.txt files received no requests at all, and that AI retrieval bots including OAI-SearchBot and PerplexityBot made up 1.1 percent of the requests that did arrive. Slackbot fetched the file more often than PerplexityBot.
Is llms.txt the same as robots.txt?
No. robots.txt tells crawlers which paths they may or may not fetch, and the major AI companies document how their bots honor it. llms.txt grants and forbids nothing; it is a reading guide that lists pages an AI system might find useful. Blocking AI training requires robots.txt rules for agents such as GPTBot or ClaudeBot. An llms.txt file has no effect on that.
How do I create an llms.txt file in WordPress?
In Yoast SEO, go to Settings, then Site features, then AI tools, and turn on the LLMS.txt toggle; you can switch to manual page selection to choose which pages appear. Dedicated plugins such as Website LLMs.txt build the file from chosen post types, exclude noindex and members-only content, and regenerate on a schedule. Or write the Markdown by hand and upload it next to robots.txt.
Can llms.txt hurt my site?
Only if it is inaccurate. The file has no known ranking effect, so an accurate, maintained one is harmless. Problems arise if it lists private or outdated pages, describes the business differently from the site itself, or is treated as a substitute for the work that does decide AI citations: crawler access, answer-first content, consistent entity data and independent coverage. Keep it truthful and keep it short.
Sources
Primary sources referenced in this article, in order of first use.
- llmstxt.org proposal (llmstxt.org)
- AI features and your website (developers.google.com)
- Search Engine Journal (searchenginejournal.com)
- crawler documentation (developers.openai.com)
- bot documentation (docs.perplexity.ai)
- crawler article (support.claude.com)
- an llms.txt for its own developer documentation (platform.claude.com)
- covering 137,210 domains (ahrefs.com)
- help article (yoast.com)
- Website LLMs.txt plugin (wordpress.org)
More in AI Search
