llms.txt file icon: a document page with a sparkle, labelled LLMS, on a soft pink gradient

97% of llms.txt Files Get No Requests, So Publish One for Coding Agents

llms.txt will not put your pages into AI answers. Google has said in writing that it ignores the file. I would still publish one, because a narrow group of agents does read it and the cost is twenty minutes.

What llms.txt is

A Markdown file at the root of your site that lists the pages worth reading, with one line of description each. Jeremy Howard of Answer.AI proposed it in September 2024. The spec at llmstxt.org was revised to v2 in August 2026. It is a map for a machine that has already arrived at your domain, and it does nothing to bring that machine to you.

What the numbers say

Two studies in 2026 describe the same picture. Adoption is real, and almost nobody is reading the files. Ahrefs checked 137,210 domains that had traffic in May 2026 and found that 97% of the valid llms.txt files received zero requests that month. On domains without a file, no AI bot asked for one.

SourceWhat it measuredResult
Ahrefs, 15 June 2026137,210 domains with traffic in May 2026About 38,000 served a valid file, a 28% figure the authors call an upper bound. 97% of those files got zero requests. AI retrieval bots were 1.1% of requests.
SEOmator, 13 September 2026Top 100,000 domains8,598 publish one, or 8.6%. 20,171 answered 200 at /llms.txt, so a status code proves nothing. 16.4% of the real files contain no links at all.
Google Search Central, June 2026Its own crawler and rankingIgnores the file, and says so in a note added in June 2026.

Google’s June 2026 changelog added a note to its AI optimisation guide, in wording that leaves no room for interpretation.

“It’s completely fine if you decide to create and maintain LLMS.txt files (or other similar files) for other services or systems that use these files. Doing so won’t harm (nor help) your visibility or rankings in Google Search, as Google Search ignores them.”

The AI features documentation says the same thing in fewer words. Beyond being indexed with an eligible snippet, there are no additional technical requirements for a page to appear as a supporting link in AI Overviews or AI Mode. So a sales page promising AI Overviews through llms.txt is selling you a file that a Google crawler will not open. What decides whether a passage gets quoted is how well it answers one question, which is why query fan-out is the thing to study instead. More arguments about getting cited sit in AI search and getting cited.

Who does read the file

Coding agents, mostly. In the Ahrefs data, Claude-Code out-fetched every AI retrieval bot, assistant and training crawler except GPTBot, which did the most fetching of any AI crawler. A developer agent opens your file to find the three pages it needs without crawling four thousand. That is a real use case, and it is the one I would write for: API references, changelogs, support pages, one line of plain description each. If your site is 40 blog posts and a contact page, nothing in that list applies and your sitemap already does the job. The file is worth writing when you have documentation somebody would want to quote, or when your own tools read it during a task.

What a useful file looks like

  1. An H1 with the site name. 95.1% of published files get this right, so it is not the hard part.
  2. A one-line summary underneath, inside a blockquote, saying what the site is for.
  3. Grouped sections with absolute URLs, each carrying a description of five to ten words.
  4. Under about 200 lines. A file that lists every post is a sitemap with extra steps.
  5. A date or a version line, so a reader knows how stale it is.

3 uses that backfire

  • Let a plugin generate it and never open it. Shopify accounts for 1,561 of the files in the SEOmator sample, Yoast 463 and Rank Math 146. Generation is cheap. Review is not.
  • Treat it as an access control. Crawler permissions live in robots.txt and headers, and the ChatGPT crawler rules are where that fight is happening.
  • Put it in a client report as a visibility win. There is no measurement that supports the claim.

A 200 status proves nothing and 30 days of logs settles it

Check any WordPress site at /llms.txt and you may get a 200 back with an HTML page in the body. That is a soft 404, not a file. SEOmator found 10,775 domains doing exactly that in September, which is more domains than the 8,598 that publish a real file. Any audit that counts status codes without reading the body will report adoption at more than twice the true figure.

Read the response, not the code. If the body opens with an H1 and ends with a list of links, you have a file. If it opens with a doctype, you have your homepage wearing a different hat.

Publish a short file and check the server log after 30 days. If the only hits are your own monitor, you have reproduced the study on your own domain and lost nothing. Passages get cited for answering a question with a date, a number and a source, which is the argument in why blog posts never show up in AI answers. A text file does not fix a page that says nothing.

If you already have one, open it and count the links. A map with no destinations is a title card.

The same limit turns up in analytics. AI referral traffic arrives in GA4 as a floor rather than a count, because assistants Google does not list land in Referral and app clicks land in Direct. A file with no recorded hits may be a file nobody fetched, or a site whose hits were never counted.

1 thought on “97% of llms.txt Files Get No Requests, So Publish One for Coding Agents”

  1. Pingback: AI Referral Traffic in GA4 Misses 4 Kinds of Visit - Siddharth Vira

Comments are closed.

Scroll to Top