llms.txt: What It Is, Who Respects It, and How to Write One

llms.txt is a proposed plain-markdown file, placed at a site's root, that curates links to a site's most important content specifically for large language models — but despite rapid adoption growth, no major AI search engine (Google, OpenAI, or otherwise) has confirmed using it to inform search results or citations, and server-log data shows the overwhelming majority of published files are never requested by any AI system at all. That gap between adoption and actual use is the single most important thing to understand before spending time on one. This guide covers where the file came from, the exact format, what the evidence shows about who really reads it, and a step-by-step method for writing one correctly if you decide it's worth doing.

Key Facts at a Glance

Question	Finding	Source
Who proposed it, and when?	Jeremy Howard of Answer.AI, published September 3, 2024	Answer.AI [1]
Does Google use it?	No. Gary Illyes confirmed at Google's Search Central Deep Dive, Asia Pacific (July 23, 2025) that Google has no plans to support it	Reported by Search Engine Land [9]
Does any major AI search engine confirm using it for citations?	No major provider — OpenAI, Google, Meta, Mistral — has publicly confirmed using it as a search or citation signal	SE Ranking, Nov. 2025 [4]
How many files actually get requested by AI systems?	Just 3% of published llms.txt files received any request at all, from any source, in a monthly sample; of that traffic, AI retrieval bots made up only about 1%	Ahrefs, via Search Engine Journal, June 2026 [5]
Overall website adoption	10.13% of ~300,000 domains sampled had a file, as of November 2025	SE Ranking, Nov. 2025 [4]
Top-10,000-site adoption, and its growth	5.61% of the top 10,000 sites as of June 2026, up from 1.04% in July 2025 (~5.4x in 11 months) — Shopify stores at 78.1%, pushed by the platform itself; WordPress sites at 8.7%, reflecting actual owner choice	HTTP Archive analysis, via Casey Burridge, June 2026 [7]
Fortune 500 adoption	7.4% (37 of 500 companies) have an llms.txt, versus 92.8% that have a robots.txt	ProGEO.ai, March 2026 [8]
Where it demonstrably does get used	Coding agents and IDE tools (Cursor, Claude Code, Windsurf, GitHub Copilot, Cline, Aider) fetch it when pointed at software documentation sites	Multiple vendor docs [3]

What Is llms.txt, and What Problem Was It Meant to Solve?

llms.txt is a plain-markdown index file, meant to sit at yoursite.com/llms.txt, that gives a large language model a short, curated map of a site's most important pages — created because a model's context window is too small to ingest a whole website, and raw HTML (navigation, ads, JavaScript) is a poor format for an LLM to parse cleanly. Jeremy Howard, founder of the AI research lab Answer.AI (and previously of fast.ai and Kaggle), published the proposal on September 3, 2024, framing the core problem this way: when an LLM needs to use information from a website, it's ambiguous whether it should crawl the entire sitemap, follow external links, or — for something like software documentation — attempt to ingest the underlying source code as well.^[1] llms.txt was Howard's answer: let the site owner decide what matters and hand the model a short, deliberately curated list instead of making it guess.

The proposal bundled a second idea alongside the file itself: that any page on a supporting site should have a clean markdown version available by appending .md to its URL — for example, example.com/docs/api.md alongside example.com/docs/api. Together, the two pieces were meant to give an LLM both a table of contents (the llms.txt file) and a low-noise version of the content itself (the .md variant), without the model having to strip HTML boilerplate on the fly.^[1]

It's worth being precise about what llms.txt was not designed to do. It is explicitly an inference-time tool — something a model consults when it's actively trying to answer a question using a specific website, similar to how a developer might open a project's README before diving into its codebase — not a training-time signal that shapes what a model learns during pretraining.^[1][2] That distinction matters enormously for the "who respects it" question later in this guide: a file that's designed for a coding agent to fetch on demand is solving a fundamentally different problem than a file meant to influence what ChatGPT or Google's AI Overviews cite in a general web search.

What Does an llms.txt File Actually Look Like?

The specification at llmstxt.org defines a strict but minimal markdown structure: exactly one required element (an H1 with the site or project name), an optional blockquote summary, optional free-text context paragraphs, and zero or more H2-headed sections that each contain a markdown-formatted list of links.^[2] Here is the shape, in order:

# Project or Site Name

> A one-to-two sentence blockquote summarizing what this site or project is.

Optional additional context paragraphs go here — background the model needs
before it starts following links.

## Docs

- [Getting Started](https://example.com/docs/getting-started.md): A first-time setup guide.
- [API Reference](https://example.com/docs/api.md): Full endpoint documentation.

## Optional

- [Changelog](https://example.com/changelog.md): Release history, safe to skip in short contexts.

A few format rules matter in practice. Each link entry must use standard markdown syntax — [name](url) — with an optional colon and short note after it describing what the link contains.^[2] Links should be absolute URLs (https://example.com/docs/api, not /docs/api), because a model consuming the file in isolation, outside a browser context, has no reliable way to resolve a relative path back to your domain. The spec also defines a special H2 section literally titled ## Optional — a signal that the links inside it are lower-priority and can be safely dropped if a model is working with a constrained context budget.^[2] Beyond the required H1, everything else is genuinely optional, which is part of why implementations in the wild vary so much in length and structure.

llms.txt vs. robots.txt vs. sitemap.xml: What Each One Actually Controls

robots.txt (introduced in 1994 as the Robots Exclusion Protocol) tells compliant crawlers what they're allowed to request; sitemap.xml tells search engines what pages exist and roughly how important or recent each one is; llms.txt curates what a site owner thinks an LLM should prioritize — and, critically, unlike the other two, it carries no enforcement mechanism and no confirmed consumer among general-purpose AI search engines. These three files are frequently lumped together in "AI SEO" advice, but they solve genuinely different problems and sit at different levels of maturity.

robots.txt is a permission gate — a well-established, widely-honored protocol (now formalized as IETF RFC 9309) that tells a crawler which paths it may or may not fetch. It says nothing about content quality or priority; it's purely access control, and major crawlers from Googlebot to GPTBot to ClaudeBot generally respect it because ignoring it carries real reputational and, increasingly, legal risk. sitemap.xml is a discovery aid — a structured, machine-readable list of every public URL a site wants indexed, submitted directly to search engines via Search Console-style tooling, that helps a crawler find pages efficiently rather than relying on link-following alone.

llms.txt occupies a different, newer, and far less settled position: it's neither a permission system nor a discovery mechanism in the sitemap sense, but a curated recommendation — closer in spirit to a README file placed at the root of a GitHub repository than to either of the older web-crawling standards. Nothing compels a model to read it, nothing verifies its accuracy, and — as the adoption data in the next two sections shows — most of the systems it was written for don't appear to be reading it in practice. The three files are complementary rather than substitutes when they do work as intended: robots.txt for access control, sitemap.xml for discovery, llms.txt for curation. But only two of the three have a track record of actually being consumed at scale by the audiences they were built for.

Who Actually Respects llms.txt Today? The Honest, Engine-by-Engine Answer

As of mid-2026, no general-purpose AI search or answer engine — not Google, not OpenAI/ChatGPT — has confirmed using llms.txt as a signal for search results, AI Overviews, or citations, and Google has explicitly and repeatedly said it does not and will not. This is the question the title of this guide promises to answer directly, so here is the state of play, engine by engine, sourced to the most recent public statements available.

Google: an explicit no. Google's Gary Illyes confirmed at Google's Search Central Deep Dive event in the Asia Pacific region, on July 23, 2025, that Google has no plans to support llms.txt.^[9] Google's John Mueller went further, publicly comparing the file to the deprecated "keywords" meta tag — a field that once let site owners self-declare what a page was about, which search engines stopped trusting decades ago because it was trivially easy to game and told them nothing they couldn't verify by reading the page directly.^[10] Mueller's stated reasoning was blunt: if a crawler is already capable of downloading and parsing the full page and its structured data, a separate self-authored summary file adds an unverifiable, easily-gamed layer rather than new information.^[10] Independent server-log research corroborates this — Google's crawlers show no distinct pattern of requesting /llms.txt beyond the general web-scanning behavior standard crawlers like Googlebot already exhibit for any file at a predictable root path.^[5]

OpenAI/ChatGPT: no public commitment, and server logs show minimal activity. OpenAI has not made a public statement adopting llms.txt as a search or retrieval signal, and the most detailed server-log study available — Ahrefs' June 2026 analysis of request patterns to published llms.txt files — found that AI-attributable retrieval bots (a category covering ChatGPT-linked crawlers alongside others) accounted for only about 1% of the file requests it observed, with the overwhelming majority of traffic instead coming from SEO audit tools, generic web crawlers, and tech-profiling scanners checking whether the file exists at all rather than reading it for content.^[5] (For what does have measured evidence of moving ChatGPT citation specifically — content structure, technical eligibility, earned authority — see RankGarage's ChatGPT citation playbook.)

Anthropic and Perplexity: narrower, more credible support — but for a specific use case. Anthropic maintains its own developer documentation using the format, and Mintlify's documentation platform names Anthropic among the early sites that adopted llms-full.txt support.^[3] Perplexity is separately and widely reported, though without a single citable primary statement, to have its crawler check for llms.txt where available to help prioritize which pages on a site to pull from. Even taking both at face value, these are meaningfully narrower confirmations than "our general answer engine cites pages differently because of your llms.txt file" — they describe developer-documentation contexts and page-prioritization within a retrieval pass, not a demonstrated citation or ranking advantage in general AI search answers, and no controlled study to date has isolated a citation lift specifically attributable to the file's presence.^[4][5]

Coding agents and IDE tools: the one place adoption clearly translates to real usage. Cursor, Claude Code, Windsurf, GitHub Copilot, Cline, and Aider all look for /llms.txt and /llms-full.txt when a developer points them at a documentation site, using the file as a table of contents before deciding which specific docs pages to fetch in full.^[3] This is the use case the original 2024 proposal was arguably best suited for from the start — a developer tool with a small, well-scoped task (understand this specific library's API) rather than a general web-search engine trying to decide what to cite out of the entire open web.

The short version: if your mental model of "llms.txt helps AI recommend my brand" is based on it functioning like a modern, AI-era sitemap.xml that search engines actively consult, the evidence as of mid-2026 does not support that model for any general-purpose AI search engine. If your use case is developer documentation consumed by coding agents, the evidence is considerably more supportive.

What Do the Adoption Numbers Actually Show?

Website adoption of llms.txt has grown substantially over the past year — SE Ranking's November 2025 study found 10.13% adoption across nearly 300,000 domains, and a separate HTTP Archive-based analysis of the top 10,000 sites found adoption rose from 1.04% in July 2025 to 5.61% by June 2026, roughly 5.4x in eleven months — but that growth in presence has not translated into any measurable relationship with AI citation frequency, and a large share of it is attributable to platform-level defaults rather than individual site-owner decisions.^[4][7] A separate raw count of published llms.txt instances (rather than a percentage of a fixed sample) shows the same directional trend even more sharply: from 4,088 in June 2025 to 36,120 by May 2026, an 8.8x increase.^[11]

SE Ranking's November 2025 study, which combined Spearman correlation analysis with an XGBoost regression model trained with SHAP feature analysis across the nearly 300,000-domain sample, found adoption was surprisingly uniform across traffic tiers — 9.88% for low-traffic sites (0–100 monthly visits), 10.54% for mid-traffic sites, and 8.27% for high-traffic sites (100,001+ visits) — but, more importantly, found no statistically meaningful relationship between having the file and how often a domain got cited in the AI outputs it measured. The researchers reported that removing the llms.txt presence/absence variable from their predictive model actually improved its accuracy, meaning the variable was closer to noise than signal for predicting citation frequency.^[4]

Adoption looks different depending on which slice of the web you sample, and the differences are revealing. An HTTP Archive-based analysis of the top 10,000 sites globally, published in June 2026, found overall adoption of 5.61% — but with an enormous platform-driven skew: Shopify-hosted stores showed 78.1% adoption, explained almost entirely by Shopify silently pushing the file to every store on its platform by default in late April/early May 2026, while WordPress sites — where adding the file requires an actual, deliberate owner or developer decision — sat at 8.7%.^[7] That gap is the clearest evidence available that raw adoption percentages conflate two very different things: genuine site-owner intent, and a hosting platform quietly flipping a default switch for millions of sites at once.

Enterprise adoption remains particularly low. ProGEO.ai's March 2026 scan of all Fortune 500 company websites — using a Python-based HTTP client to check for the presence of robots.txt, sitemap.xml, and llms.txt across each site — found only 37 companies, or 7.4%, had implemented an llms.txt file, versus 92.8% that had a robots.txt (a standard now more than three decades old).^[8] That 85-point gap is a useful sanity check on how far llms.txt still is from being treated as baseline technical infrastructure, even among the organizations with the resources to implement it easily.

Is Anyone's AI Crawler Actually Requesting These Files?

Server-log data is the most direct evidence available, and it's stark: Ahrefs' June 2026 analysis of request logs across 137,000 domains — roughly 38,000 of which had a valid llms.txt file — found that 97% of those files received zero requests at all in the sampled period, and of the roughly 1,100 files that did get traffic, AI-attributable retrieval bots made up only about 1% of the requests.^[5] This is the single most important data point for anyone deciding whether to invest time in the file, because it moves the question from "does an AI company say it uses this file" to "do the request logs show anyone's AI system actually fetching it."

The breakdown of what did request the ~3% of files that got any traffic is itself informative: roughly 21% of requests came from SEO audit tools checking whether the file exists (not reading it for content signal), 14% from unidentified bots, 13% from standard web crawlers like Googlebot doing routine indexing sweeps (not a distinct llms.txt-aware behavior), 11% from technology-profiling tools, and around 19% from AI bots across all categories combined — of which the narrower subset of AI retrieval bots specifically tied to answer-generation (as opposed to, say, a general-purpose training crawler) accounted for only about 1 percentage point.^[5] Google's John Mueller's own characterization, in the same reporting, was that llms.txt currently functions as, at best, "a temporary crutch, perhaps to save some tokens" for AI coding tools — a narrow, specific use case, not evidence of broad adoption by search-facing AI systems.^[5]

It's worth flagging the methodological caveat honestly: server logs can only show what got requested, not what an AI system might have consumed through some other channel (a cached crawl, a third-party index, a partner data-licensing arrangement). But absent evidence of those alternate channels, the direct, observable signal — actual HTTP requests to the file's URL — is the closest thing available to ground truth, and it points the same direction as the engine-by-engine public statements in the previous section: minimal, narrow usage concentrated in coding-agent contexts, not broad usage by general AI search engines.

If Search Engines Ignore It, What Is llms.txt Actually Good For?

The evidence supports one clear, defensible use case — giving coding agents and AI-assisted developer tools a fast, structured entry point into software documentation — plus a smaller set of lower-cost, forward-looking reasons that don't require believing any specific search engine reads the file today. Given the adoption and server-log data above, it would be dishonest to recommend llms.txt as a citation-boosting AEO tactic on the strength of current evidence. But "it doesn't move AI Overview citations today" isn't the same conclusion as "it's worthless," and the distinction matters for deciding where it fits in a broader technical SEO and AEO program.

Documentation and developer-tool discoverability is the strongest case. If your site has software documentation, an API reference, or an SDK — the exact scenario the original 2024 proposal was written to address — llms.txt gives tools like Cursor, Claude Code, and GitHub Copilot a fast way to find your most relevant docs pages without those agents having to crawl your entire site structure blind.^[3] Mintlify's launch of /llms.txt and /llms-full.txt support in November 2024 — starting with early-adopter documentation sites including Anthropic, Windsurf, and Bolt.new, and later extended as a standard option across its hosted documentation platform^[3] — is early, concrete evidence that this specific use case has real, practical adoption rather than speculative value.

Low implementation cost relative to uncertain future upside. A well-scoped llms.txt file for a marketing or content site typically takes under an hour to write and costs nothing to host. Given the format is genuinely simple and adoption is still rising sharply — roughly 5.4x growth among top-10,000 sites in eleven months, and a similar 8.8x by raw instance count over roughly the same span^[7][11] — there's a reasonable argument for implementing a lightweight version now as a low-cost hedge, on the premise that if any major engine does eventually adopt it as a signal, sites that already have a well-formed file won't be starting from zero. This is a hedge, not a current-state citation strategy, and should be communicated to stakeholders as exactly that.

It forces useful internal clarity. The exercise of deciding what belongs in a curated, 15-to-50-link summary of "the most important things to know about this site" often surfaces gaps in a site's own information architecture — pages that don't clearly map to a coherent hierarchy, or a site that genuinely doesn't have a clean answer to "what are our five most important pages." That's a real, if secondary, benefit independent of whether any AI system ever reads the resulting file.

What it should not be treated as: a replacement for the content-structure and citation-density work that the available research does show moves AI-answer visibility — inline sourcing, statistics, direct quotations, and self-contained answer passages. (RankGarage's guide to what actually changes between SEO and GEO covers the levers with a measured, evidenced effect on AI citation in far more depth than this guide's narrower scope.)

How to Write an llms.txt File, Step by Step

Writing a spec-compliant llms.txt takes four steps: pick the required H1 and summary, decide which content genuinely deserves inclusion, format the links correctly, and publish the file at your site's root so it returns a plain-text 200 response. Given the file's job is curation, the hardest part is usually the "what to include" decision, not the syntax.

Step 1 — Write the header. Start with a single H1 naming your site or project (# RankGarage), followed immediately by a blockquote with a one-to-two sentence summary of what the site does. Both are the closest thing the spec has to required fields, and a parser built to the llmstxt.org standard expects to find them at the top.^[2]

Step 2 — Add context paragraphs only if they earn their place. The spec allows free-text paragraphs between the summary and the first H2 section, but these should carry information a model genuinely needs before it starts following links — background on your industry, terminology specific to your product, or context that would otherwise require the model to infer from scattered pages. If you don't have anything that specific to say, skip this section entirely; padding it out works against the file's own purpose.

Step 3 — Organize your most important content into H2-headed sections. Use headers like ## Documentation, ## Product, or ## Guides to group related links, and under each, list entries in the format - [Title](https://example.com/page): a short note on what the page contains. Keep the total file lean — most practical guidance converges on 15 to 50 curated links and well under 500 words total, because a file that tries to list everything gives a model no actual signal about what matters most; that's the same failure mode as an unfiltered sitemap, just reformatted as markdown. Use absolute URLs throughout, since a model consuming this file outside a browser context has no way to resolve a relative path.

Step 4 — Add an ## Optional section for anything lower-priority. This is the spec's built-in mechanism for telling a context-constrained model "you can skip these if you're tight on space" — good for things like a changelog, an older version of documentation, or supplementary reference material that isn't core to understanding the site.

Step 5 — Publish it at your domain root and verify it. The file needs to live at yourdomain.com/llms.txt, return an HTTP 200 status, and serve as plain text/markdown (not wrapped in HTML by your CMS template). After publishing, run it through a spec validator — several free tools check a file against the llmstxt.org format and flag missing required elements or malformed links — and spot-check that every linked URL actually resolves rather than 404ing, since a curated list of broken links undermines the file's one job.

Do You Need llms-full.txt Too?

llms-full.txt is a companion file, popularized alongside the original proposal and now widely supported by documentation platforms like Mintlify, that includes the complete, expanded content of every linked page inline in one file — appropriate for smaller documentation sets, but not a direct substitute for the curated llms.txt for larger sites. Where llms.txt is a table of contents with links out, llms-full.txt is closer to printing the entire book into a single file: every page's full markdown content concatenated together, so a model (or a coding agent) can ingest everything in one fetch without following individual links.

The tradeoff is straightforward: llms-full.txt is more convenient for a small documentation set where the full content fits comfortably within a typical context window, but it scales poorly for a large site — a full-content file for a sprawling documentation set can run to hundreds of thousands of words, defeating the context-efficiency goal the original llms.txt proposal was built around in the first place. The practical convention that's emerged: offer llms.txt as the default curated index for any site, and add llms-full.txt only if your total documentation is small enough that dumping it all into one file stays genuinely useful rather than becoming its own unwieldy artifact.

Common Mistakes to Avoid

1. Treating it as an AI-era sitemap.xml that search engines will actively consult. The engine-by-engine evidence above shows this expectation isn't currently supported for any general AI search engine — Google has explicitly declined, and OpenAI has made no public commitment.^[4][9]
2. Listing everything instead of curating. A file with 200+ links defeats the entire purpose — it gives a model the same undifferentiated pile of URLs a sitemap already provides, just in a different format.
3. Using relative URLs. A model consuming the file outside a full browser context can't reliably resolve /docs/api back to your domain — always use the full https:// address.
4. Skipping validation. A file with malformed markdown, a missing required H1, or broken links signals low effort to any system that does check it, and defeats the purpose for the coding-agent use case where it demonstrably does get used.
5. Letting it go stale. An llms.txt that links to pages you've since removed, renamed, or deprecated is actively worse than having no file — for the coding-agent use case where the file is genuinely consulted, broken links waste an agent's limited fetch budget.
6. Over-promising its SEO or AEO value to stakeholders. Given the adoption and server-log data, framing llms.txt internally as a near-term citation-boosting tactic sets up a credibility problem when the promised lift doesn't materialize — the honest framing is "low-cost hedge and developer-tool aid," not "AI visibility strategy."
7. Exposing content you'd rather not curate for scraping. Because the file is a plain-text, unauthenticated list of your most important pages, don't include links to gated, unreleased, or sensitive content just because it seems "important" — treat it the same way you'd treat anything else published in public, unauthenticated plain text.

Should You Still Implement llms.txt in 2026? A Decision Framework

Implement it if you run developer-facing documentation, have spare capacity for a low-cost, low-risk technical addition, or want a hedge against future adoption — deprioritize it if you're choosing between it and higher-evidence AEO work like citation-dense content structure, schema markup, or brand-mention building, all of which have measurably moved AI-answer visibility in independent research where llms.txt has not.^[4][5]

Concretely: a SaaS company with an API and a docs subdomain should very likely add both /llms.txt and, if the docs set is small enough, /llms-full.txt — the coding-agent use case is the one place the evidence directly supports it, and the format costs little to implement well. A local service business, an e-commerce brand, or a marketing-led company with no developer audience gets far less from the exercise; for that profile, the better use of the same hour is a well-structured FAQ section with schema markup (see RankGarage's schema markup guide for the implementation specifics) or an answer-capsule rewrite of an existing high-traffic page, both of which sit on firmer evidentiary ground for actually moving AI citation.

If you do decide to implement it, treat it the way this guide's evidence supports: a small, well-maintained piece of technical hygiene with a clear, narrow beneficiary (coding agents and doc-savvy AI tools) rather than the centerpiece of an AI-visibility strategy. RankGarage's AEO Starter plan includes llms.txt setup and maintenance as one component within a broader, ongoing program — alongside schema implementation, answer-block content, and monthly visibility tracking — precisely because it's cheap to maintain correctly once, not because it's the highest-leverage lever available. If you're unsure where llms.txt should rank against everything else competing for your team's time, a free AEO audit benchmarks your current AI visibility and prioritizes the levers with the strongest evidence behind them for your specific site.

What's Next: Will Google or OpenAI Ever Adopt It?

Neither the IETF nor the W3C has ratified llms.txt as a formal web standard, and nothing in Google's or OpenAI's public statements as of mid-2026 suggests that's imminent — but the file's role as de facto infrastructure for coding agents gives it a plausible path to continued relevance even without general-search adoption. Unlike robots.txt, which was eventually formalized as RFC 9309 roughly three decades after informal adoption began, llms.txt remains what Jeremy Howard originally called it: a proposal, maintained on GitHub, that any site can choose to implement without any standards body's involvement.^[1][2]

The more interesting near-term question isn't whether Google reverses its position — its stated reasoning (a crawler that can already read the full page has little use for an unverifiable, self-authored summary) is a durable objection, not a temporary one — but whether the coding-agent ecosystem's usage keeps compounding independently of general AI search. Given that Cursor, Claude Code, Windsurf, and similar tools already treat the file as a working convention, and given that AI-assisted software development shows no sign of slowing, llms.txt's most likely long-term outcome may be a stable, permanent niche — genuinely useful developer-tooling infrastructure that never becomes the general AI-search equivalent of a sitemap its early proponents may have hoped for.

---

FAQ

Does adding an llms.txt file improve my chances of being cited by ChatGPT or Google's AI Overviews?

No — based on the best available evidence, it doesn't. SE Ranking's November 2025 study of nearly 300,000 domains found no measurable relationship between having an llms.txt file and AI citation frequency, and removing the variable from its predictive model actually improved accuracy.^[4][6] Google has explicitly said it doesn't use the file for search,^[9] and server-log data shows AI retrieval bots account for only about 1% of the requests the file receives at all.^[5]

If no major AI search engine confirms using it, is there any reason to bother with llms.txt at all?

Yes, for a specific audience: if your site has software documentation or an API, coding agents like Cursor, Claude Code, and GitHub Copilot do fetch the file when working with your docs, making it genuinely useful developer-tooling infrastructure.^[3] For a marketing or content-led site with no developer audience, the case is much weaker, and time is better spent on content structure and schema work with stronger evidence behind it.

What's the difference between llms.txt and llms-full.txt?

llms.txt is a curated index — an H1, a summary, and a short, hand-picked list of links to your most important pages. llms-full.txt includes the complete content of every linked page concatenated into one file, so a model can ingest everything in a single fetch rather than following links individually. llms-full.txt works well for smaller documentation sets; for larger sites, a full-content dump can run to hundreds of thousands of words and defeats the context-efficiency goal the format was built around.

Should I use llms.txt to block AI crawlers from sensitive content, the way robots.txt blocks search crawlers?

No — llms.txt isn't an access-control mechanism and has no enforcement behind it; use robots.txt for that. llms.txt is an unauthenticated, plain-text list of links you're actively choosing to surface, so don't include anything gated, unreleased, or sensitive in it just because it feels important to mention.

How long should an llms.txt file be?

Most practical guidance converges on roughly 15 to 50 curated links and well under 500 words total. A much longer file — one that tries to list everything on the site — gives a model no more signal than an unfiltered sitemap already provides, just reformatted as markdown, and defeats the curation purpose the format exists for.

Does WordPress or Shopify add llms.txt for me automatically?

Shopify does — the platform silently pushed the file to every store by default in late April/early May 2026, which is why Shopify-hosted sites show roughly 78.1% adoption in top-10,000-site data. WordPress does not add it automatically; WordPress adoption sits at only about 8.7% in the same dataset, reflecting genuine, individual site-owner decisions rather than a platform default.^[7]

Could having a low-quality or inaccurate llms.txt file hurt my SEO?

There's no evidence of a direct search-ranking penalty tied specifically to llms.txt content, since Google has stated it doesn't use the file for search in the first place.^[9] The more realistic risk is reputational and practical rather than algorithmic: a file full of broken links or inflated claims undermines trust with the one audience (coding agents and developer tools) that does appear to read it.

Is llms.txt an official, ratified web standard like robots.txt?

No. Robots.txt was eventually formalized as IETF RFC 9309, decades after informal adoption began. llms.txt remains a community proposal maintained on GitHub since its September 2024 publication, with no ratification from the IETF or the W3C as of mid-2026.^[1][2]

How often should I update an llms.txt file once I've published one?

Update it whenever the set of pages it links to changes meaningfully — new major documentation sections, deprecated or removed pages, or a restructured information architecture. A stale file with broken or outdated links is arguably worse than no file at all for the coding-agent use case, since it wastes an agent's limited fetch budget on dead ends.

Should a software documentation site prioritize llms.txt more than a typical marketing website?

Yes, meaningfully so. Documentation sites are the one context where the evidence directly supports real-world usage — by coding agents and AI-assisted developer tools specifically.^[3] A marketing-led site with no developer audience should treat llms.txt as a low-priority, low-cost addition at best, and invest its limited AEO effort in schema markup, citation-dense content, and answer-block structure instead, where the evidence of AI-visibility impact is considerably stronger.

---

llms.txt is a real, well-intentioned proposal solving a real problem — but as of mid-2026, it's solving that problem for a narrower audience than most of the coverage around it suggests. The honest summary: Google has explicitly declined to use it, no other major AI search engine has confirmed doing so, and server logs show almost nobody's AI system is actually requesting the files that do get published. Where it clearly does work is developer documentation read by coding agents — a real, if narrow, use case worth serving well if it applies to your site. Everywhere else, it's a low-cost, speculative hedge at best, not a substitute for the content-structure and citation work with real evidence behind it. For the broader picture of what does move AI-answer visibility, see RankGarage's guide to what actually changes between SEO and GEO and the complete guide to Answer Engine Optimization.

See where you stand in AI search. Run a free AEO audit of your site — free with a 3-month plan minimum.

References

[1] Jeremy Howard, Answer.AI, "/llms.txt — a proposal to provide information to help LLMs use websites," September 3, 2024. https://www.answer.ai/posts/2024-09-03-llmstxt.html

[2] llmstxt.org, "The /llms.txt file" (specification). https://llmstxt.org/ — defines the required H1, optional blockquote/context, and H2-delimited link-list sections; accessed August 2026.

[3] Mintlify, "Simplifying docs for AI with /llms.txt," published November 20, 2024. https://www.mintlify.com/blog/simplifying-docs-with-llms-txt — announced /llms.txt and /llms-full.txt support, launching with early-adopter documentation sites including Anthropic, Windsurf, and Bolt.new. Coding-agent support for the format (Cursor, Claude Code, Windsurf, GitHub Copilot, Cline, Aider fetching /llms.txt when pointed at a documentation site) is widely reported across developer-tooling coverage as of 2026, though this guide does not cite a single primary source for each individual tool.

[4] SE Ranking (Yulia Deda and Svitlana Tomko), "LLMs.txt: Why Brands Rely On It and Why It Doesn't Work," published November 7, 2025. https://seranking.com/blog/llms-txt/ — Spearman correlation and XGBoost/SHAP analysis of ~300,000 domains; 10.13% overall adoption as of the study date; no measurable correlation between file presence and AI citation frequency.

[5] Ahrefs data, as reported by Search Engine Journal, "97% Of llms.txt Files Got No Requests, Ahrefs Data Shows," published June 16, 2026. https://www.searchenginejournal.com/97-of-llms-txt-files-got-no-requests-ahrefs-data-shows/579478/ — server-log analysis of 137,000 domains (~38,000 with valid llms.txt files); 97% received zero requests; AI retrieval bots made up ~1% of requests among files that did get traffic; includes John Mueller's "temporary crutch" characterization.

[6] Search Engine Journal, "LLMs.txt Shows No Clear Effect On AI Citations, Based On 300k Domains," 2025 reporting on the SE Ranking study cited in [4]. https://www.searchenginejournal.com/llms-txt-shows-no-clear-effect-on-ai-citations-based-on-300k-domains/561542/

[7] Casey Burridge, "Does anyone actually have an llms.txt? I checked millions of websites," published June 20, 2026, using HTTP Archive's BigQuery dataset. https://caseyrb.com/blog/state-of-llms-txt-adoption/ — 5.61% adoption among the top 10,000 sites; Shopify 78.1% (platform-pushed default beginning late April/early May 2026); WordPress 8.7% (owner-driven adoption).

[8] ProGEO.ai (Clinton Karr), Fortune 500 website scan, conducted March 2026, published March 31, 2026, as reported by ppc.land, "Only 7.4% of Fortune 500 have an llms.txt file, study finds." https://ppc.land/only-7-4-of-fortune-500-have-an-llms-txt-file-study-finds/ — Python-based HTTP client scan of all Fortune 500 sites; 7.4% (37/500) had llms.txt versus 92.8% with robots.txt.

[9] Search Engine Land, "Google says normal SEO works for ranking in AI Overviews and LLMs.txt won't be used," reporting on Gary Illyes' (Google Search Relations) statement at Google's Search Central Deep Dive, Asia Pacific, July 23, 2025. https://searchengineland.com/google-says-normal-seo-works-for-ranking-in-ai-overviews-and-llms-txt-wont-be-used-459422 — Illyes stated Google doesn't support llms.txt and isn't planning to, and that ranking in AI Overviews requires standard SEO practices, not a separate file.

[10] Search Engine Journal, "Google Says LLMs.Txt Comparable To Keywords Meta Tag," 2025 reporting on statements from Google's John Mueller. https://www.searchenginejournal.com/google-says-llms-txt-comparable-to-keywords-meta-tag/544804/ — Mueller: "AFAIK none of the AI services have said they're using LLMs.TXT... To me, it's comparable to the keywords meta tag."

[11] ppc.land, "llms.txt adoption rises 8.8x but 97% of files get zero AI requests," 2026, corroborating reporting on the Ahrefs dataset in [5]. https://ppc.land/llms-txt-adoption-rises-8-8x-but-97-of-files-get-zero-ai-requests/ — llms.txt instance count grew from 4,088 in June 2025 to 36,120 by May 2026 (8.8x); includes companion llms-full.txt/ai.txt formats for a combined ~38,980 adopting sites.