llm.txt file: what it is and should you create one

TL;DR: llm.txt is a proposed standard that lets a website serve AI models a clean, markdown version of its content. It's not an official spec, not universally supported, and won't earn you AI citations on its own. Documentation and developer sites get the most out of it. Most marketing sites can skip it.

What is an llm.txt file?

llm.txt is a plain-text file you put at the root of your domain, at yourdomain.com/llm.txt. The idea is simple. Instead of making an AI crawler parse your full HTML (nav bars, cookie banners, JavaScript-rendered blocks, footer links), you hand it a clean markdown document that describes your site and points to the content that matters.

Jeremy Howard, co-founder of fast.ai, proposed the concept in September 2024 [1]. He drew the analogy to robots.txt, which has existed since 1994 and tells crawlers which pages to skip. The intent flips, though. robots.txt is about exclusion. llm.txt is about inclusion and clarity.

A typical file has a short description of the site or product, a set of links to the pages that matter, and optionally a longer markdown block an AI can read directly. Some setups add an llms-full.txt variant that concatenates the full text of key pages into one document, so a model can swallow everything in a single pass.

The format is not a W3C or IETF standard. It's a community proposal with a draft spec hosted at llmstxt.org [1]. That single fact should shape how you think about building one.

Where did llm.txt come from and who backs it?

Jeremy Howard published the original proposal on September 3, 2024, at llmstxt.org [1]. His problem was concrete. When people asked AI assistants about fast.ai libraries, the models pulled from stale docs or invented API details, because the HTML was cluttered and hard for a model to parse cleanly [10].

Adoption moved fast. Within months, Anthropic's docs site, Cloudflare's developer docs, and a stack of other technical platforms published their own files [8][9]. By early 2025, a community-curated directory listed more than 2,000 domains with an llm.txt. That's a rounding error against the whole web, but real traction for a proposal barely nine months old.

No major search engine or AI company has endorsed the format as a ranking signal or a crawl directive. OpenAI, Google, and Anthropic have not published a line saying "we read and respect llm.txt." The closest thing to formal support: some AI-focused crawlers, including Perplexity's, plus various RAG pipeline tools, now check for and parse the file when it exists [3].

So the backing is one influential researcher, strong momentum in technical circles, and early tool support. Not nothing. Not a mandate either.

How does llm.txt differ from robots.txt and sitemap.xml?

These three files do different jobs and get confused constantly.

robots.txt tells crawlers what they're not allowed to index. It's a restriction with real teeth. Block Googlebot in robots.txt and Google skips those pages. llm.txt has no such enforcement. An AI crawler can ignore it entirely with zero consequences.

sitemap.xml tells search engines where your pages live. It's a map, not a content file. It lists URLs and optional metadata like last-modified dates. llm.txt actually holds or points to content, and it's meant to be read by models more than by classic crawlers.

Here's the breakdown:

| File | Purpose | Format | Enforced? | AI-specific? | |---|---|---|---|---| | robots.txt | Block crawlers from pages | Plain text directives | Yes, major crawlers honor it | No | | sitemap.xml | List all URLs for crawlers | XML | No, advisory | No | | llm.txt | Provide clean content to AI | Markdown | No, purely advisory | Yes |

robots.txt and llm.txt share one useful property. You can block AI crawlers from your whole site in robots.txt (plenty of publishers now do this for GPTBot, ClaudeBot, and friends) while still offering an llm.txt to the crawlers you want reading your content. They complement each other. They don't overlap.

For background on how ai search engines crawl and index in the first place, that context frames why clean-text files matter at all.

What does a real llm.txt file look like?

Here's a minimal but complete example. The spec says the file starts with a level-one heading holding the site or product name, then a blockquote with a short description, then optional sections under level-two headings.

# Acme Analytics

> Acme Analytics is a web analytics platform for small businesses.

## Documentation

- [Getting Started](https://acme.com/docs/start): Installation and first dashboard setup
- [API Reference](https://acme.com/docs/api): Full REST API documentation
- [Pricing](https://acme.com/pricing): Plan details and limits

## About

- [About Us](https://acme.com/about): Company background and team
- [Blog](https://acme.com/blog): Product updates and tutorials

That's the whole thing. No special syntax beyond standard markdown. The companion llms-full.txt would hold the actual prose from each linked page, concatenated, so a model reads everything without following links.

Anthropic serves a good real-world example. Its file lists the major documentation sections with short descriptions [9]. Cloudflare's developer docs do the same at developers.cloudflare.com [8].

One practical note. The spec is loose, so implementations vary. Some sites paste long content blocks inline. Others keep the file short and link out. Some use llm.txt, some use llms.txt, and both show up in the wild. If you're building tooling, check for both paths.

Do ChatGPT, Claude, Gemini, or Perplexity actually read llm.txt?

This is the question that matters for AI visibility, and the honest answer is: partially, inconsistently, and with no published guarantee.

Perplexity has been the most open about it. Its crawler, PerplexityBot, checks for llm.txt and uses it to guide what it indexes from a site [3]. That's worth something, because Perplexity is a real answer engine with tens of millions of monthly users.

ClaudeBot (Anthropic) and GPTBot (OpenAI) crawl the web to build training data and power retrieval features. Neither company has published documentation saying they treat llm.txt as a routing directive. What's likely true: if your llm.txt holds clean, useful text, a crawler that reads it at all pulls better content from your site than it would by chewing through your HTML.

Google publishes guidance on how it handles AI-generated content and structured data, but says nothing about llm.txt [4]. AI Overviews and AI Mode pull from indexed pages, not special files. For Google specifically, llm.txt probably does nothing directly.

The research on what drives AI citations points to content quality and topical authority far more than the presence of a file. BrightEdge reported in 2024 that AI Overviews cited pages ranking in Google's top 10 organic results roughly 96% of the time [5]. Traditional SEO factors dominate Google's citation choices. For dedicated answer engines like Perplexity, the picture is fuzzier, because Perplexity runs its own real-time retrieval.

For the wider view, generative engine optimization treats crawlability as one lever among several.

How often AI Overviews cite top organic results

| | | |---|---| | Citations from top 10 organic results | 96% | | Citations from outside top 10 | 4% |

Source: BrightEdge, 2024

What are the real benefits of creating an llm.txt file?

Three benefits hold up. Be honest with yourself about how much each one matters for your site.

Content clarity comes first. If your site has messy navigation, JavaScript-heavy rendering, or docs sprawled across hundreds of pages, llm.txt gives any AI system (a crawler, a RAG pipeline, a third-party tool) a clean entry point. Documentation sites gain the most. When a developer building against your API asks an assistant how to use it, clean llm.txt content makes the answer more accurate and more current.

Second, the early-mover signal. llm.txt is young enough that sites adopting it now are doing something most of the web hasn't. Early adoption of a structured format sometimes turns into a meaningful signal later, the way schema.org markup did after 2011. Nobody can promise that. But the cost is low enough that the option value is real.

Third, RAG pipeline compatibility. Loads of teams are building internal tools that pull from external sources using retrieval-augmented generation. A clean llm.txt makes your docs far easier to ingest into those pipelines. If your product has an API or a developer audience, this is the highest-value use today, and more reliable than betting on ChatGPT finding your file.

What it won't do: fix thin content, replace real topical authority, or override a crawl block in robots.txt. If AI systems aren't citing you, the cause is almost always content quality or authority, not a missing text file.

What are the downsides or risks of llm.txt?

The risks are real, even if none of them are catastrophic.

Scraping gets easier. Publish a clean, well-organized file of your best content and you make it trivial for any automated system to grab everything, including the ones you don't want. Competitors, scrapers, and low-quality content farms all benefit. If your edge lives in proprietary content, think that through before you publish.

False security is the trap I see most. A marketer hears about llm.txt and decides the file will fix their AI visibility. It won't. An assistant that doesn't cite you today won't start because you added a text file. Low visibility comes from weak authority, thin content, and no mentions in the sources these models trained on. That takes content and authority work, not a file.

Maintenance is light but not zero. An llm.txt pointing to deleted or stale pages can actively mislead the systems you're trying to help. Create it, and you need a process to keep it current when your site changes.

The spec may fragment. With no standards body behind it, the format can drift as different tools invent their own variations. You might build around a structure that goes obsolete.

Tools like ai seo tools help you track whether your content actually surfaces in AI answers, which is a far better feedback loop than assuming your llm.txt is doing its job.

Should you create an llm.txt file for your site?

The honest answer depends on your site type. For most marketing sites, it's low-priority.

Create it if:

• You have technical documentation, an API, or developer-facing content. This is the clearest win. Developers constantly build tools that ingest docs through LLMs, and your llm.txt directly improves both their experience and the accuracy of AI answers about your product.
• Your site is hard to crawl. Multi-level nav, JavaScript-rendered content, and sprawling docs all bury your best material. llm.txt is a shortcut past that mess.
• You already publish structured content. If your team works in markdown and keeps a content inventory, the extra work runs under two hours.

Deprioritize it if:

• You run a standard marketing or e-commerce site with clean HTML and solid technical SEO. Spend your energy on content quality, schema markup, and topical authority. Those have documented impact on AI citations [5][6].
• Your goal is citations in ChatGPT or Google AI Overviews. Neither has committed to reading llm.txt, so you'd be building toward an uncertain payoff.
• Your content is thin or outdated. A clean path to bad content is still bad content.

A sane default for most companies: put llm.txt on the roadmap, knock out a basic version in a low-effort pass (two to three hours), and revisit its maintenance quarterly. Don't let it crowd out higher-leverage work.

To see how your brand is actually being cited (or not) across AI systems right now, Spawned's AI visibility audit shows you where you stand before you sink time into any single tactic.

How do you create an llm.txt file step by step?

Here's a process you can finish in an afternoon.

Step 1: Inventory your best pages. Open your analytics and pull the 20 to 30 pages that drive the most value: core product or service pages, key documentation, your top blog posts, and anything that proves your expertise on topics you want cited for.

Step 2: Write the header. Open a plain text editor. Start with # Your Site or Product Name, then a blockquote (>) with one to three sentences on what your site is and who it's for. Be specific. "Acme is an analytics platform for B2B SaaS companies tracking product adoption" beats "Acme is a leading analytics solution."

Step 3: Group your links under ## headings. Common sections: Documentation, Blog, About, Products or Services, and Case Studies or Research. Write a one-sentence description of each link.

Step 4: Optionally build llms-full.txt. This holds the full text of your key pages, in markdown, concatenated. More work to create and maintain, but more useful for systems that read content directly instead of following links.

Step 5: Place the files at yourdomain.com/llm.txt (and yourdomain.com/llms-full.txt for the extended version). No server config needed. It's a static text file.

Step 6: Validate. The llmstxt.org site has a directory and basic tooling [1]. Check a couple of AI-focused crawlers' public docs to confirm they'll recognize your format.

Step 7: Reference the file in your HTML <head> with a link tag. The draft spec suggests: <link type="application/llms+json" rel="llms" href="/llm.txt">. This helps crawlers find it even when they don't know to look.

A focused 20-page file takes two to four hours. Quarterly maintenance runs about thirty minutes per cycle.

For where this sits inside a broader ai seo strategy: content and authority signals matter more, but this is a clean, cheap piece of the puzzle.

How does llm.txt connect to broader AI search visibility strategy?

llm.txt is one small piece of a bigger picture. Knowing where it fits keeps you from overspending on it.

The evidence on what drives AI citations points to three things: content quality and depth, domain authority (which tracks the sources AI training data favored), and structured data that makes your content machine-readable [5][6]. llm.txt touches the third, and slightly the first, by presenting content cleanly.

The bigger levers for ai search visibility metrics: getting cited on the web sources these models trained on (press, industry publications, Wikipedia-adjacent content), giving clear and accurate answers to the specific questions your audience asks, and using schema markup so AI systems understand what type of content they're reading.

Perplexity leans heavily on real-time retrieval. A Search Engine Land analysis of Perplexity citations in 2024 found Wikipedia, Reddit, and authoritative industry publications dominated its sources [3]. That tells you where to point your content-seeding effort.

llm.txt works best paired with strong content underneath it, never as a shortcut around it. Think of it as a clean front door on a house that's already well built. The door helps. The house is why people come.

For teams tracking AI visibility systematically, Spawned's platform monitors ChatGPT, Claude, Gemini, and Perplexity, so you can see which content actually gets cited instead of guessing.

What's the future of llm.txt? Will it become a standard?

The trajectory looks positive but uncertain. Here's what the signals say.

Start with the optimistic case: the robots.txt analogy holds. Martijn Koster floated robots.txt as an informal convention in 1994, and it became a de facto internet standard within a few years, never formally standardized by IETF at the time. It only got an informational RFC, RFC 9309, in 2022, nearly three decades later [7]. As the RFC puts it, the protocol had been "a de-facto standard" for years before formalization. llm.txt is walking a similar road: informal proposal, community pickup, growing tool support.

By mid-2025, several AI-adjacent companies (documentation tools, developer platforms, content management systems) had started auto-generating llm.txt files by default. That kind of tooling integration is what turns a convention into a standard.

Now the skeptical case: the AI crawler landscape is fragmented. Each major AI company builds its own crawling stack, and none has an obvious incentive to formally commit to a format it didn't create. Google runs its own structured data ecosystem (schema.org, rich results) and won't likely redirect effort toward an outside convention it can't control.

The realistic outcome: llm.txt becomes a durable convention for technical and developer documentation, adopted by a meaningful minority of the web, loosely supported by a range of AI tools and crawlers, and never formally mandated. Still useful. Just not a revolution.

To stay current on how ai powered search features evolve and which signals actually matter, the category moves fast enough that quarterly check-ins on official guidance earn their place in your workflow.

Sources

1. llmstxt.org, original proposal by Jeremy Howard
2. Search Engine Land, Perplexity AI citation sources analysis, 2024
3. Google Search Central, crawling and indexing documentation
4. BrightEdge, AI Overviews citation research, 2024
5. Search Engine Journal, AI search citation factors analysis, 2024
6. IETF RFC 9309, Robots Exclusion Protocol, September 2022
7. Cloudflare Developers, llms.txt file at developers.cloudflare.com
8. Anthropic, Claude documentation site llm.txt adoption
9. fast.ai, Jeremy Howard background and fast.ai library documentation context