llms.txt has become one of the most discussed ideas in AI search. It is a short Markdown file placed at the root of a website that summarizes the site and points AI systems to its most important pages. Many agencies and software vendors now recommend it as a way to improve visibility in AI answers. Some claim it will help a brand get cited more often.
The question is simple: does it work? The honest answer, based on the data available in 2026, is that there is little evidence it changes how AI systems read or cite websites. That does not make the file useful for nothing, but it does mean it should not be the centerpiece of an AI visibility strategy.
This article reviews what the studies measured, what they found, what they cannot prove, and where your time is better spent.
The file was designed to help AI systems read a site. The studies so far have mostly measured whether anyone reads it at all.
What llms.txt Was Designed to Do
llms.txt is a proposed convention, described at llmstxt.org, for giving language models a curated, readable overview of a website. The file uses Markdown, lives at /llms.txt, and lists the pages a site owner considers most useful. The aim is to reduce the effort a model needs to understand a site, much as a sitemap helps search engines discover URLs.
The proposal has three parts that are worth separating, because the evidence differs for each:
- Adoption: how many sites publish the file.
- Fetching: whether AI systems request the file.
- Impact: whether publishing the file changes how often or how accurately a site is cited.
Most of the public discussion has focused on the first part. The more important questions are the second and third.
The Adoption Picture
Adoption has grown quickly from a small base. A study by Originality.ai, which monitored more than 3 million websites, recorded 4,088 llms.txt instances in June 2025. By May 2026, the count had reached 36,120, an increase of about 8.8 times over twelve months. Over the same period, the number of sites publishing llms-full.txt, a companion file with more complete content, grew from 23 to 2,463.
Adoption is still concentrated among certain kinds of sites. A study by ProGEO.ai reported in April 2026 found that 7.4 percent of Fortune 500 companies, 37 of 500, had implemented llms.txt as of March 2026. In the same group, robots.txt was present at 92.8 percent. Other crawls of the top ranked domains have reported similar or lower figures, with the exact number depending on the list used and the date of the crawl.
Growth in adoption tells you that many people are willing to try the file. It does not tell you that the file works.
Adoption has risen sharply from a small base, but it remains far below the near-universal presence of robots.txt among large companies.
The Fetch Gap: Do AI Crawlers Request the File?
This is where the data is most consistent. An Ahrefs study published in June 2026 analyzed server log data from 137,210 domains that received traffic in May 2026. It found that about 28 percent of those domains had a valid llms.txt file, and that 97 percent of those files received zero requests during the month.
The study looked at the index file only, so it does not measure every possible way a system might use the content. Even so, the result is striking. Most published files were not requested at all, by bots or by people.
A separate 12-week study, reported in July 2026, examined 83 sites that had deployed llms.txt and logged the requests made by crawlers. The figures for OpenAI's crawler illustrate the pattern. Over that period, GPTBot requested robots.txt 3,990 times and requested llms.txt 7 times. The difference is large, and it suggests that when a crawler does visit a site, it is far more interested in the rules that govern its access than in a curated summary.
These results should be read carefully. They describe specific crawlers in specific periods, and crawler behavior can change. But the pattern has held across several independent analyses, which makes it harder to dismiss.
In one twelve-week study, a single crawler requested robots.txt thousands of times and llms.txt only a handful of times.
The Citation Question: Does It Increase Visibility?
The most important question for most businesses is whether llms.txt increases citations in AI answers. Here the evidence is thinner and more mixed.
A study by SE Ranking, published in July 2026, analyzed around 300,000 domains and found no measurable citation advantage for sites that had the file. A study by Trakkr Research, covering 37,894 domains in 2026, reported the same conclusion. Both studies compared sites with and without the file and did not find a meaningful difference in citation frequency.
A single-site observation reported by OtterlyAI reached a more positive view. Single-site observations are useful for generating hypotheses, but they cannot separate the effect of llms.txt from everything else a site changed at the same time, such as better content, new pages, or improved technical health.
Google has also been clear about its own position. In its May 2026 guidance on AI features, Google listed llms.txt among tactics that are not needed for appearance in AI Overviews and AI Mode. Google's public statement is a statement about its own systems, not a proof about every AI platform, but it removes one of the most common assumptions behind the file.
Taken together, the citation evidence so far suggests that llms.txt does not produce a consistent, measurable lift. It is possible that a real effect exists and has not yet been detected. It is also possible that the file has little effect on citations at all.
The citation studies differ in sample size and method, but none so far has shown a consistent, measurable benefit.
What the Data Cannot Prove
An honest review has to include the limits of the evidence.
Most studies are observational. They compare sites that have the file with sites that do not, and sites that adopt new standards often differ in other ways. Sites that publish llms.txt may also have better content, stronger technical foundations, or more active teams. Observational data cannot fully separate those factors.
Crawler behavior changes. A crawler that ignores llms.txt today may read it tomorrow. Companies can change their policies, and new agents can appear. The results describe a moment in time.
Different systems behave differently. Some AI products search the live web, some rely on training data, and some use their own indexes. A file that is never requested by one system might be used by another, though the evidence for any specific system remains limited.
Absence of evidence is not proof of absence. The fair conclusion is not that llms.txt definitely does nothing. It is that the available evidence does not show it does what its advocates claim.
Some sources have commercial interests. Many articles about llms.txt are published by companies that sell GEO tools, audits, or consulting services. That does not make their findings wrong, but it is a reason to check primary data and be cautious about strong claims.
Why the Gap Exists
The gap between adoption and impact is not surprising once you consider how the file is supposed to work. For llms.txt to help, several things have to happen in sequence. A crawler has to know the file exists, decide to request it, decide to trust its contents, and then use those contents when generating an answer. Each step depends on the system's design, and none of them is guaranteed by the proposal.
robots.txt, by contrast, has a clear and widely respected role. Crawlers check it before they request pages, and most well-behaved crawlers follow its rules. llms.txt has no comparable enforcement, and no major AI provider has committed to reading it. Without that commitment, a site owner is publishing a document for an audience that may never open it.
This does not make llms.txt useless. Some developer tools and documentation platforms may read it, and it may be helpful to developers who use AI tools to explore documentation. The value there is practical, but it is different from the claim that the file improves general AI search visibility.
So Should You Publish One?
The decision depends on cost and purpose.
If you already maintain clear documentation, a well-structured site, and a short list of key pages, creating an llms.txt file takes little time and carries little risk. The file is simple, and writing it can force useful clarity about what your site offers and which pages matter most.
If your goal is to improve AI search visibility, the file should not be your main investment. The evidence does not support that expectation. A better use of your time is the work that studies and practitioners consistently point to: making sure AI crawlers can access your content, keeping pages fast and clearly structured, stating facts directly, and building a consistent description of your business across the web.
If you do publish the file, keep it accurate. An outdated llms.txt that points to retired pages is worse than no file at all.
Crawler access, clear content, and consistent entity information matter more than llms.txt for most sites.
Where to Spend Your Time Instead
If llms.txt is not the lever, what is? The consistent answer across the research and practitioner work is a set of fundamentals.
Access. Confirm that the crawlers you want to allow can reach your important pages. Check robots.txt, meta robots tags, server responses, and CDN bot rules. A blocked page cannot be cited.
Structure. Use clear headings, direct answers near the top of each page, and accurate titles and descriptions. Pages that state their topic and conclusion early are easier for any system to use.
Content depth and credibility. Named authors, dated updates, attributed statistics, and specific, verifiable claims give readers and systems reasons to trust a page.
Consistency. Describe your company, products, and location the same way across your site, your profiles, and third-party directories. Inconsistent descriptions make it harder for any system to represent your business accurately.
Performance and stability. Slow or broken pages reduce crawling efficiency and user satisfaction. These problems affect both search engines and AI systems that rely on the same web infrastructure.
Measurement. Track which crawlers visit your site, which pages they request, and how your brand appears in answers to the queries that matter to you. Repeat the measurements over time.
How to Evaluate Claims About llms.txt Yourself
When you read an article or a vendor pitch about llms.txt, ask a few questions before you accept the conclusion.
What was measured: the presence of the file, requests for the file, or citations in AI answers? These are different outcomes.
How was the comparison made: was there a control group of sites without the file, and were other changes controlled for?
Over what period: a single month or a single site is a weak basis for a general claim.
Who published it: is the author selling a product that benefits from the claim?
Can you reproduce it: does the study describe its data and method well enough for you to check it?
Ask what was measured, how the comparison was made, over what period, by whom, and whether the result can be reproduced.
Frequently Asked Questions
Do AI crawlers read llms.txt? Based on the server log studies available in 2026, most AI crawlers request it rarely or not at all. Some tools and platforms may use it, but there is no broad confirmation from the major providers that it shapes their answers.
Does llms.txt improve citations in ChatGPT, Claude, or Perplexity? The studies so far have not found a consistent, measurable citation benefit. That does not rule out an effect in the future, but it is not the expectation to plan around today.
Is it harmful to publish llms.txt? Publishing an accurate, well-maintained file is low risk. The main downside is that a neglected file can point to outdated pages and mislead anyone who reads it.
Is llms.txt the same as robots.txt? No. robots.txt controls crawler access and has broad adoption and respect. llms.txt is a curated summary with no comparable enforcement or confirmed use by major AI providers.
Does Google use llms.txt? Google has stated that llms.txt is not needed for appearance in AI Overviews and AI Mode. Its guidance is about its own systems.
What should I prioritize instead? Crawler access, clear page structure, accurate and credible content, consistent business information, site performance, and regular measurement of how your brand appears in AI answers.
Conclusion
llms.txt is a reasonable idea that has spread faster than the evidence supporting it. Adoption has grown sharply, but most published files are not requested by crawlers, and the studies so far have not shown a consistent lift in citations. The honest conclusion is that the file is, at best, a low-cost supplement, not a proven way to improve AI visibility.
Treat it accordingly. Publish it if it is easy and accurate for your site, but put your effort into the fundamentals that the evidence points to: access, structure, credible content, consistency, and measurement. Let the data guide the next decision, and revisit the question as the research improves.
Key Takeaways
- Adoption of llms.txt has grown rapidly, but it remains concentrated and far below robots.txt among large companies.
- Log-based studies found that most published llms.txt files are rarely or never requested by AI crawlers.
- Citation studies so far have not shown a consistent, measurable benefit from publishing the file.
- Many claims about llms.txt come from vendors with commercial interests, so check what was measured and how.
- The file is low cost and can be useful, but crawler access, content structure, credibility, and consistency matter far more.
