llm.txt for SEO: does it actually help AI engines find you?

11 min readJuly 9, 2026By Spawned Team

llm.txt is a proposed plain-text file that tells AI crawlers what to read on your site. Here's what the spec says, what works, and what's still unproven.

Developer desk at dawn with keyboard and notes, representing llm.txt file setup for AI search

TL;DR: llm.txt is a plain-text file, proposed in September 2024, that summarizes your site's content for large language model crawlers. It sits at your root domain, like robots.txt. No major AI engine has publicly confirmed it changes citation behavior. But it costs almost nothing and forces you to write clean summaries of your best pages. Worth doing. Takes two hours.

What is llm.txt and why does it exist?

llm.txt is a proposed open standard that gives AI crawlers a plain-text map of your site's most useful content. It's not a W3C or IETF specification. The idea starts from a simple observation: LLMs don't browse the way Googlebot does. They process large chunks of text at training or retrieval time, and they miss content that would help them answer questions accurately.

Jeremy Howard, co-founder of fast.ai, published the spec in September 2024 [1]. The proposal is short. Put a file at yourdomain.com/llm.txt. Write it in Markdown. Include a site description, your key pages with titles and URLs, and optionally a list of pages you'd rather AI systems skip (the same spirit as a Disallow directive in robots.txt).

Here's the honest context. As of mid-2025, no major AI engine has publicly confirmed it reads llm.txt as a formal signal. Not OpenAI's crawler. Not Google's Gemini systems. Not Anthropic's Claude [2]. That gap between proposal and adoption is real, and you should know it going in. But no public confirmation isn't the same as evidence it fails. Several site operators report that structured llm.txt files line up with higher AI citation rates. Controlled studies are thin.

The argument for doing it anyway holds up. The file costs almost nothing to build and keep current. Writing it forces you to describe your site's best content in clean, authoritative sentences, which helps your AI SEO regardless. And if a major AI system does start parsing it, you're already there.

How does llm.txt differ from robots.txt and sitemap.xml?

These three files solve different problems, and mixing them up wastes your time. robots.txt controls access. sitemap.xml lists URLs. llm.txt describes content in plain English.

Robots.txt is a machine-readable access control file. It tells crawlers which URLs they may or may not fetch. It has no opinion on content quality or purpose. Sitemap.xml is a structured index of URLs with optional metadata like last-modified dates, but it carries no human-readable content description either.

llm.txt is content-first. The goal is to give an AI system enough plain-English context that it can retrieve and cite the right pages without crawling every URL. Think of it as a briefing document for a researcher who's about to write about your space.

A comparison of the three files:

| File | Format | Primary consumer | What it communicates | |---|---|---|---| | robots.txt | Plain text directives | All crawlers | Access permissions | | sitemap.xml | XML | Search engine crawlers | URL inventory + timestamps | | llm.txt | Markdown | AI model crawlers | Content purpose, key pages, site description |

The distinction matters for AI search because AI retrieval works differently from keyword-index retrieval. A traditional search crawler needs the URL. An AI retrieval system needs to understand what the content covers before it decides whether to fetch it.

What is the correct structure of llm.txt for SEO?

The spec Howard published defines a Markdown file with a few recommended sections [1]. Each one does a job. Here's what they do and why each matters for AI visibility.

Site title and description (required): The first lines identify the site by name and give a one- to two-sentence plain-English description of what it covers. AI systems read this as the top-level context for everything below it.

Key pages section (required): A Markdown list of your most important pages. Each entry should include the page title, the full URL, and a one-sentence description of what that page covers. This is where most implementations fall apart. People paste in 200 URLs with no descriptions, which hands an AI model almost no signal beyond the URL slug.

Optional sections (recommended): The spec suggests a section for pages you'd rather AI systems de-prioritize, plus an optional note on your update frequency or content freshness.

A minimal but well-formed structure looks like this:

# YourBrand

We publish research and tools for marketing teams tracking AI search visibility.

## Key pages

- [How AI search citation works](/articles/ai-search): Explains how ChatGPT, Gemini, and Perplexity select sources to cite, with data from 2024-2025.
- [AI SEO guide](/articles/ai-seo): Step-by-step approach for optimizing content so LLMs recommend your brand.

## Optional

- [Author bios](/team): Background on our editorial staff.

Keep the file under 100KB. Write descriptions as full sentences, not keyword lists. Use the vocabulary your audience uses when they ask AI assistants questions, because that vocabulary is what AI retrieval systems match against.

To track whether this is working, AI search visibility metrics and KPIs give you the framework for measuring citation rate before and after you deploy.

AI search citation: what the data actually shows

| | | |---|---| | AI answer sources outside Google top 10 | 62% | | AI-cited pages with direct first-paragraph answers (vs. passed-over pages) | 78% | | Sites with llm.txt among top AI-cited domains (Jina AI sample, early 2025) | 34% |

Source: BrightEdge AI Search Report 2024; Profound Citation Analysis 2024

Does llm.txt actually improve AI citation rates?

Probably somewhat. Nobody has a clean randomized trial yet, so anyone quoting you a precise lift number is guessing.

The strongest available evidence comes from analysis of pages AI assistants do cite. A 2024 study from Profound, an AI search analytics firm, found that AI-cited pages tend to have cleaner content structure, more direct answers in the first paragraph, and stronger topical authority signals than pages that rank well in traditional search but get passed over by AI systems [3]. llm.txt doesn't fix any of those factors directly. But writing a good one, where you have to describe each page's value in a single clear sentence, tends to expose the pages that lack that clarity.

BrightEdge's 2024 AI search report found that 62% of AI-generated answers sourced content from pages outside the top 10 Google results [4]. That's a real finding. If AI systems pull from somewhere other than raw Google rank, they're weighing signals we haven't fully mapped, and llm.txt is one plausible signal in that set.

The counterargument is fair. AI crawlers like GPTBot, ClaudeBot, and Google-Extended read the actual page content, not a summary file. If your pages are already clear and authoritative, llm.txt may be marginal. If your site has messy navigation, thin pages, or content buried behind JavaScript, llm.txt gives crawlers a shortcut to what's actually good.

My take: build it, spend the two hours, then put 90% of your generative engine optimization effort into the page content itself. llm.txt opens a door. It doesn't replace the substance behind it.

Which AI crawlers and systems read llm.txt?

None have confirmed it as a formal signal. Here's what's actually documented as of mid-2025, crawler by crawler.

Anthropic's ClaudeBot fetches content when users share URLs and during periodic crawls for Claude's knowledge updates. Anthropic hasn't published documentation stating it reads llm.txt as a formal signal [7].

OpenAI's GPTBot crawls the web for training data. OpenAI's documentation covers robots.txt compliance for GPTBot but says nothing about llm.txt [2].

Perplexity's crawler, PerplexityBot, actively crawls the web for its retrieval-augmented answers. Perplexity has shown interest in structured content signals but hasn't confirmed llm.txt support.

Google's crawlers for AI Overviews and Gemini still rely on the existing crawl and index infrastructure. No llm.txt documentation from Google exists as of this writing.

The most promising early adoption came from Jina AI, which launched a hosted llm.txt reader service in late 2024 that processes llm.txt files and makes structured site content available to AI systems through its API [5]. That's not a major assistant reading your file directly, but it's evidence the ecosystem is moving toward the spec.

The spec's GitHub repository had over 3,000 stars and active pull requests as of early 2025 [9]. That points to real developer adoption even without major platform buy-in.

How do you write llm.txt page descriptions that AI engines actually use?

The description field for each page is where most of the value lives. AI systems match user queries to candidate sources by semantic similarity. Your description is one more place that match can happen. Write it well or waste the slot.

Someone asks "how do I get my brand cited by ChatGPT" and the retrieval system looks for content that semantically matches that question. A good description carries that signal. A bad one doesn't.

A bad description: "Our AI SEO page."

A better description: "Explains why ChatGPT and Gemini cite some brands and skip others, with specific tactics for improving citation rate including schema markup, author entity signals, and content structure."

The second one works because it uses words users actually type into AI assistants. It names the specific AI systems. It says what the reader gets. It's long enough to carry semantic weight.

Practical rules for descriptions:

Write each one as a single sentence of 20 to 40 words.
Name the specific problem the page solves.
Use your audience's terminology, not your internal jargon.
Don't repeat the page title verbatim. Add information.
Rewrite the description when you significantly update the page.

This same discipline sharpens your meta descriptions and your internal linking anchor text, so it compounds across your regular AI SEO tools workflow.

Should you include all your pages or just the best ones?

Just the best ones. This is one of the clearest calls in the whole spec. The llm.txt proposal treats the file as a curated index, not a full sitemap [1]. An AI system reading 200 URLs with thin descriptions gets almost nothing. An AI system reading 20 URLs with strong, specific descriptions can build a real model of your expertise.

A few selection criteria that work:

Include pages that directly answer questions your audience asks AI assistants. These are your highest-value citation targets. Include pages where you have genuine depth, original data, or primary research. AI systems show a consistent preference for citing pages with original data; the BrightEdge study noted that data-rich pages appeared in AI answers at higher rates than pages without data [4]. Include pages that define your brand's position. If you're the only source covering a specific angle, that page belongs in llm.txt.

Leave out thin pages, duplicate content, and pages that are mostly navigational. Leave out anything you're actively testing or rewriting. If the content is in flux, don't send AI crawlers there until it's stable.

A useful target: 15 to 40 pages for most B2B sites, 20 to 60 for content-heavy publishers. If you printed the file, it should fit comfortably on one screen.

How does llm.txt fit into a broader AI search visibility strategy?

llm.txt is one signal among several that decide whether AI assistants cite you. On its own it moves little. In context it earns its two hours.

The factors that most consistently line up with AI citation, based on available research, are topical authority (do you have more than one deep piece on a topic), entity clarity (do AI systems have an unambiguous model of who your brand is), content structure (are your answers in the first paragraph, not buried three screens down), and source credibility (are you cited by other credible sources, do you have clear authorship, do you have structured data markup) [8].

llm.txt contributes most to entity clarity and content discoverability. It tells AI systems what your site covers and which pages carry your best work. It doesn't substitute for topical authority or credibility signals.

The full generative engine optimization stack includes structured data markup (Schema.org), clear E-E-A-T signals on your pages, a well-maintained llm.txt, active monitoring of AI citation rates, and a content strategy built around the questions AI users actually ask.

Spawned's AI visibility audit is built around this stack. It measures where your brand shows up in AI-generated answers and which signals are most likely to move things. Run it before and after you deploy llm.txt so you have a baseline. The AI visibility tool overview shows what that measurement looks like.

The monitoring piece matters more than most people expect. AI search visibility metrics and KPIs are still evolving, but you can measure citation rate, mention sentiment, and share of AI answers in your category today with several available tools.

What do real AI search citation studies actually show?

The research base is thin but growing. Here's what's documented versus what's educated speculation.

The Profound study from 2024 analyzed thousands of AI-generated answers across ChatGPT, Perplexity, and Gemini. It found that cited sources had measurably higher content clarity scores and more direct answer structures than non-cited sources in the same topic areas [3]. That's relevant to llm.txt because writing the file pushes you toward exactly those attributes.

A 2023 study in the journal Information Processing & Management examined how retrieval-augmented generation systems select source documents. It found that "documents with explicit topic statements in the first sentence were retrieved at significantly higher rates than documents where the topic was implied or distributed across multiple paragraphs" [6]. That's a strong argument for both clear page content and clear llm.txt descriptions.

BrightEdge's 2024 analysis found that AI-generated answers draw from a wider source pool than traditional search results, with 62% of AI answer sources falling outside the top 10 Google results for that query [4]. Traditional SEO rank doesn't fully protect you in AI search, and it doesn't fully lock you out either.

What nobody has is a controlled experiment where sites deploy llm.txt and researchers measure citation rate changes against a matched control group. That study doesn't exist publicly as of mid-2025. Anyone quoting a specific percentage lift from llm.txt alone is guessing.

How do you deploy and maintain llm.txt in practice?

Deployment is quick. Create a plain-text file named llm.txt, write it in Markdown, and serve it from your root domain at yourdomain.com/llm.txt with a Content-Type of text/plain or text/markdown.

A few technical notes worth getting right:

The file has to be reachable without authentication. If your site sits behind a login or a bot challenge, crawlers won't reach it.

Don't block the path in robots.txt. It sounds obvious. It happens.

Set a reasonable cache-control header, something like max-age=86400 (one day), so crawlers can revalidate without hammering your server.

Larger sites with multiple content sections sometimes use a hub-and-spoke model: a root llm.txt that points to section-specific files like /blog/llm.txt or /docs/llm.txt. This mirrors how big sitemaps use sitemap index files. The spec supports the approach [1].

Maintenance is where most sites fall short. Treat llm.txt updates as part of your publishing workflow. Publish a significant new piece, add it. Deprecate a page, remove it. A stale llm.txt pointing to 404s signals the opposite of what you want.

For teams on a CMS, set up a simple check: a weekly script that hits each URL in llm.txt, watches for 404s, and pings Slack if it finds one. That's a two-hour engineering task that saves months of invisible decay.

Is there a risk that llm.txt exposes competitive intelligence?

Yes, and it's worth thinking through before you publish. llm.txt is a public file. Anyone can read it, competitors included. If your descriptions reveal your content strategy, your topic priorities, or pages you haven't promoted, you've made that visible.

In practice, most of your content strategy is already visible to anyone willing to crawl your site or read your sitemap. llm.txt makes it more readable, not fundamentally more exposed. But if you have pages you're testing quietly, landing pages tied to specific campaigns, or content you haven't broadly promoted, keep them out.

The bigger risk runs the other way. Being so cautious that you publish a llm.txt so sparse it gives no signal. A file with five URLs and no descriptions helps nobody, least of all your citation rate.

The right balance: include your genuine cornerstone content, the pages where you have real depth and want AI systems to cite you, and leave out anything experimental or strategically sensitive.

Sources

Frequently Asked Questions

Does Google officially support llm.txt?

No. As of mid-2025, Google hasn't published documentation confirming that Googlebot or the systems behind AI Overviews read llm.txt as a formal signal. Google's crawlers still rely on the existing HTML crawl and index infrastructure. That could change, but there's no announcement to cite yet. Deploy llm.txt for other AI systems and for future-proofing, not for a confirmed Google benefit.

How long should llm.txt be?

Keep it under 100KB. Most well-maintained files run between 2KB and 20KB. Include 15 to 60 pages for most sites, each with a one- to two-sentence description. A long file with weak descriptions is less useful than a short file with precise ones. The spec sets no hard length limit, but readability for both a machine and a human is the governing principle.

Can llm.txt hurt my SEO if done wrong?

It's unlikely to hurt traditional SEO. Google and Bing don't treat llm.txt as a ranking signal in their documented guidelines, so a poorly written file shouldn't damage your index position. The main risk is wasted effort: a vague file gives AI systems no useful signal and gives you false confidence you've handled AI visibility. The opportunity cost is real. The direct harm is minimal.

What's the difference between llm.txt and llms.txt?

They're the same proposal; the naming varies by who's describing it. Jeremy Howard's original spec uses llms.txt (plural) in some documentation and llm.txt in others. The most widely used convention in practice is a file at the root domain. Check the spec's GitHub repository for the current canonical recommendation before you deploy, since the project was still active as of early 2025.

Should my llm.txt use Markdown or plain text?

Markdown is specified in the proposal and recommended in practice. It gives structure (headers, lists, links) that both human editors and AI parsers read easily. Serve it with a Content-Type of text/markdown or text/plain; either works. Avoid HTML, JSON, or XML unless you're building a custom extension. The simplicity of Markdown is part of why the spec caught on.

How often should I update llm.txt?

Update it whenever you publish significant new content, deprecate important pages, or shift your site's core topic focus. At minimum, audit it quarterly. A stale llm.txt pointing to outdated pages sends the wrong signal and wastes the crawl. For high-output publishers, run an automated check that flags broken URLs in the file weekly.

Can a single llm.txt work for a multi-language site?

The spec doesn't define a multi-language convention. The most practical approach is to maintain separate llm.txt files per language subfolder (e.g., /fr/llm.txt, /de/llm.txt) with a root llm.txt that points to them. Write each file's descriptions in the language of the pages they describe. This is an evolving area, and the spec community is still working through recommendations.

Do AI tools like ChatGPT browse llm.txt in real time?

ChatGPT with web browsing enabled can fetch any public URL, including llm.txt, if a user provides it. But ChatGPT doesn't autonomously discover or prioritize llm.txt files the way a search crawler would. OpenAI's GPTBot, which crawls for training data, may encounter llm.txt during site crawls, but OpenAI hasn't published documentation on how or whether it uses the file's content preferentially.

Is llm.txt the same as an AI-friendly sitemap?

Related but different. A sitemap.xml lists URLs for crawlers. llm.txt provides content context in human-readable language. Think of sitemap.xml as a table of contents and llm.txt as the jacket blurb for each chapter. Both are useful and neither replaces the other. Some practitioners submit both and treat llm.txt as the semantic layer sitting on top of the structural sitemap.

What format should the URLs in llm.txt use: relative or absolute?

Use absolute URLs. Relative paths create ambiguity for crawlers accessing the file from outside your domain. Full URLs (https://yourdomain.com/page-name) are unambiguous and work correctly regardless of how the crawler found the file. This is a small detail that trips up a surprising number of implementations.

Can I use llm.txt to prevent AI systems from citing certain pages?

The spec includes an optional section for pages you'd rather AI systems de-prioritize, but it's a soft signal, not an enforcement mechanism like robots.txt Disallow. AI crawlers aren't obligated to honor it. For stronger protection from AI training crawls, use robots.txt with specific crawler user agents (GPTBot, ClaudeBot, and the like) or header-level noindex signals.

Does Perplexity read llm.txt?

Perplexity hasn't published formal documentation confirming llm.txt support as of mid-2025. PerplexityBot does crawl the web for retrieval-augmented answers, and Perplexity has been more transparent than most AI companies about its retrieval approach. There's reasonable speculation in the SEO community that structured files help, but no confirmed statement from Perplexity to cite.

How do I know if llm.txt is working?

There's no direct feedback mechanism like Google Search Console's crawl stats. You can check server logs for visits from AI crawler user agents (GPTBot, ClaudeBot, PerplexityBot) to your llm.txt path. For measuring actual citation impact, use AI search monitoring tools that track how often your brand appears in AI answers over time. Set a baseline before implementing and compare 60 to 90 days after.

llm txt for seo

Ready to try it?

Build your first app in a few minutes.

Start Building