robots.txt configuration for AI crawler access: the complete guide

TL;DR: robots.txt lets you block or allow AI crawlers like GPTBot, ClaudeBot, and Google-Extended one at a time using their published user-agent strings. Blocking a crawler stops future fetches. It does not remove you from an AI's existing training data. If you want AI citation traffic, allow these bots. Do not block them.

What is robots.txt and how does it control AI crawlers?

robots.txt is a plain-text file at the root of your domain, at https://yourdomain.com/robots.txt. It follows the Robots Exclusion Protocol, a convention first proposed in 1994 and later written up as a Google draft RFC submitted to the IETF in 2022 [1]. The file tells compliant crawlers which paths they may or may not fetch.

Every major AI company now publishes one or more named user-agent strings for its crawlers, and those crawlers are supposed to read and respect your robots.txt before fetching a single page. That "supposed to" is the whole game. The protocol is voluntary. Well-funded companies like OpenAI, Anthropic, Google, and Perplexity do respect it. Some smaller scrapers ignore it entirely.

The mechanics are simple. A crawler hits your robots.txt, finds the rule block that matches its user-agent, and follows the Allow and Disallow lines it finds there. No matching block means most crawlers treat the whole site as fair game. The file parses top to bottom, and the most specific matching rule wins.

Here is what people get wrong. robots.txt controls crawling, not indexing, and it controls future crawling, not what an AI has already seen. If GPTBot scraped your site last year before you added a Disallow rule, that content may already sit in OpenAI's training corpus. The rule you add today stops future fetches. It is not a time machine.

Which AI crawlers have published user-agent strings?

These are the confirmed, publicly documented user-agent strings from major AI companies as of mid-2025. Check each company's official docs before you rely on any of them, because the strings do change.

| Company | Crawler name | User-agent string | Official docs | |---|---|---|---| | OpenAI | GPTBot | GPTBot | platform.openai.com | | OpenAI | ChatGPT-User | ChatGPT-User | platform.openai.com | | Anthropic | ClaudeBot | ClaudeBot | support.anthropic.com | | Google | Google-Extended | Google-Extended | developers.google.com/search | | Google | Googlebot (standard) | Googlebot | developers.google.com/search | | Meta | FacebookBot | FacebookBot | developers.facebook.com | | Perplexity | PerplexityBot | PerplexityBot | docs.perplexity.ai | | Apple | Applebot-Extended | Applebot-Extended | support.apple.com | | Common Crawl | CCBot | CCBot | commoncrawl.org |

OpenAI runs two crawlers [2]. GPTBot fetches content for training future models. ChatGPT-User fetches content in real time to answer user queries, including when ChatGPT browses the web. Block GPTBot and you affect future training. Block ChatGPT-User and you affect real-time answers. Those are two different business calls.

Google-Extended is the user-agent Google uses for Gemini and its AI answer products. It is separate from Googlebot, which handles traditional search indexing [3]. You can block Google-Extended and leave your standard Google rankings untouched. That separation is genuinely useful, and it was not possible before Google introduced the dedicated string in September 2023.

What does the correct robots.txt syntax look like for AI crawlers?

The syntax has not changed since 1994. Each rule block starts with one or more User-agent lines, followed by Disallow and Allow directives, then a blank line. That is the whole grammar.

To block a single AI crawler from your entire site:

User-agent: GPTBot
Disallow: /

To allow a crawler across your whole site, spelled out rather than left to default:

User-agent: GPTBot
Allow: /

To block a crawler from specific directories only:

User-agent: ClaudeBot
Disallow: /private/
Disallow: /members-only/
Allow: /

To block everything at once, then carve out named exceptions:

User-agent: *
Disallow: /

User-agent: Googlebot
Allow: /

User-agent: GPTBot
Allow: /

The wildcard (*) block applies to any crawler not matched by a named block. Precedence matters here. A named block always overrides the wildcard for that crawler. So in the example above, Googlebot and GPTBot get full access even though the wildcard says Disallow everything.

One detail trips people up constantly. The path in Disallow is a prefix, not a glob pattern. Disallow: /blog/ blocks /blog/ and everything under it. It does not block /myblog/. To block a single file, give the exact path: Disallow: /secret-page.html.

The Crawl-delay directive (not in the original RFC, but honored by many bots) slows a crawler down. Crawl-delay: 10 tells the bot to wait 10 seconds between requests. Handy if a crawler is hammering your server. For most sites, none of them are.

AI crawler user-agents and their primary function

| | | |---|---| | GPTBot (OpenAI, training) | 1 | | ChatGPT-User (OpenAI, real-time) | 2 | | ClaudeBot (Anthropic, real-time) | 2 | | Google-Extended (Gemini/AI Overviews) | 2 | | PerplexityBot (real-time answers) | 2 | | Applebot-Extended (Apple AI features) | 2 | | CCBot (Common Crawl, training) | 1 |

Source: OpenAI GPTBot docs, Anthropic support, Google Search Central, Perplexity docs (2024-2025)

Should I block AI crawlers or allow them?

This is the real question, and the answer depends entirely on what you want from AI systems.

If you want ChatGPT, Perplexity, Claude, or Gemini to cite you when someone asks a question your content answers, these crawlers need access. Block them and you remove yourself from the pool those systems draw from. A system that has never seen your content cannot recommend it.

A 2024 Originality.ai analysis of more than 800 websites found that sites blocking GPTBot got little real protection against scraping in aggregate, because most of their content had already been collected, while they did lose referral traffic from AI browsing features [4]. For anything that has been live more than a few months, the protection is mostly illusory.

If you are a publisher who does not want AI companies profiting from your writing without paying, blocking is a fair protest, and several large newsrooms have done exactly that. The New York Times blocks a range of AI crawlers. That is a defensible business and ethical position. It also means giving up AI referral traffic.

Got genuinely sensitive content? Member data, pricing pages, legal documents, internal tools? Block AI crawlers from those specific paths no matter what your overall stance is. This is not an all-or-nothing decision.

For AI SEO, the practical move is this. Allow the crawlers that power AI assistants people actually use (GPTBot, ChatGPT-User, ClaudeBot, Google-Extended, PerplexityBot). Restrict crawlers you get nothing from (CCBot, random scrapers). Protect your sensitive directories from everyone.

Nobody has clean data on exactly how much crawler access moves AI citation rates. The closest evidence comes from published crawl studies and practitioner reports, which line up on one point: pages that are crawlable and well structured show up in AI answers more often than blocked equivalents.

How do I block specific AI crawlers while keeping others?

You use separate named blocks. The standard allows as many User-agent blocks as you need, and each crawler reads only the block matching its own name.

Here is a working configuration that allows the assistants users actually query, blocks aggressive scrapers, and protects sensitive paths:

# Block untargeted scrapers
User-agent: CCBot
Disallow: /

# Block training data collection from specific AI companies
# (use only if you have a specific reason)
User-agent: GPTBot
Disallow: /

# Allow real-time ChatGPT browsing even while blocking training
User-agent: ChatGPT-User
Allow: /

# Allow AI assistants that drive referral traffic
User-agent: ClaudeBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Google-Extended
Allow: /

# Keep standard search indexing fully open
User-agent: Googlebot
Allow: /

# Protect sensitive directories from everything
User-agent: *
Disallow: /admin/
Disallow: /members/
Disallow: /api/

The GPTBot and ChatGPT-User split is deliberate. Some owners do not want their content training future models but are happy to have ChatGPT surface their pages as real-time answers. OpenAI gave you that choice with two separate user-agents. Use it.

Test the file before you ship it. Google Search Console includes a robots.txt report that flags syntax problems [5]. Third-party validators like the one at ryte.com work for broader checks. One typo in robots.txt can open paths you meant to close, or close paths you meant to open, and you will not notice until traffic moves.

Does blocking AI crawlers in robots.txt remove my content from AI training data?

No. This is the misconception that does the most damage.

AI training data gets collected over long stretches, folded into a model, and then that model ships for months or years. The Common Crawl dataset, which underpins training for many LLMs, holds snapshots going back to 2008 [6]. Once content is baked into a model's weights, a robots.txt change you make today cannot pull it back out.

Here is what blocking a crawler does do. It stops that company from fetching your content in future crawls. If they train a new model version next year off a fresh crawl, that crawl would respect your updated robots.txt. Whether that protects you depends on whether they run a new crawl, whether they pull from other data sources, and whether older dataset versions are still in play.

Anthropic's ClaudeBot documentation says the crawler "respects robots.txt" and supports "real-time data retrieval" to answer user queries [7]. Real-time use is a different thing from training. Block ClaudeBot and Claude cannot fetch your page to answer a question right now. It does not scrub anything from prior training runs.

If real removal from training data is your goal, most AI companies run a separate web form or legal process for that. OpenAI publishes a content removal request path in its platform documentation. Those processes are slow, narrow, and not guaranteed. robots.txt is the wrong tool for that job.

What is the difference between robots.txt and meta robots tags for AI crawlers?

robots.txt works at the crawl level. It tells a bot whether it can fetch a URL at all. Meta robots tags (the noindex, noai, and noimageai values in the HTML head) work at the content level. They tell a bot that already fetched the page what it may do with the content.

The noai and noimageai values were proposed in 2023 by a coalition of stock photo sites and publishers as a signal that content should not be used for AI training [8]. Uptake by AI companies has been patchy. Google has not formally committed to honoring noai. OpenAI has published no specific guidance on it. Perplexity says it respects robots.txt but has made no firm promise about meta tag values beyond standard noindex.

The practical hierarchy runs like this:

1. If robots.txt says Disallow, a compliant crawler never fetches the page, so it never reads any meta tags.
2. If robots.txt says Allow (or is silent), the crawler fetches the page and then reads meta tags to decide what to do with the content.
3. If neither robots.txt nor meta tags restrict the bot, the content is fair game.

For tighter control, combine both. Use robots.txt to keep bots out of directories you never want touched. Use meta robots tags on individual pages where you allow the crawl but want a say in how the content gets used.

Reading up on generative engine optimization fills in how these signals feed AI answer generation past the crawl layer.

How do I verify that AI crawlers are actually reading my robots.txt?

Check your server logs. Every compliant crawler fetches robots.txt before crawling anything else, so you will see a request to /robots.txt from the crawler's IP range right before other requests from that same IP. OpenAI publishes its crawler IP ranges in the GPTBot documentation [2], and other companies publish similar lists. Cross-reference your /robots.txt log hits against those ranges.

Google Search Console has a dedicated robots.txt report under Settings that shows whether Googlebot fetched your file and flags errors [5]. There is no equivalent official tool for GPTBot or ClaudeBot, which is an annoying gap.

For ongoing monitoring, a small server-side script can log every /robots.txt request with the full user-agent string. Run it for a week. You will see which crawlers hit you, how often, and from which IPs. It also surfaces unknown or suspicious crawlers that are on nobody's published list.

One honest caveat. Verifying that a crawler read your robots.txt and verifying that it complied are two different things. You can see PerplexityBot fetched your robots.txt on Tuesday morning. Then check the logs for later PerplexityBot requests to paths you disallowed. If those requests show up, the crawler is not complying. In practice, the major AI crawlers have good compliance records, but monitoring is the only way to actually know.

Tools like Spawned's AI visibility tool track which AI systems are surfacing your brand in answers, which gives you an indirect but useful signal. If a model cannot see your content because of a crawl block, its citation rate for your domain drops.

What robots.txt configuration is best for maximizing AI search citations?

If citations are the goal, the configuration is simple. Allow every crawler tied to an AI assistant that produces user-facing answers. That means GPTBot (ChatGPT training), ChatGPT-User (real-time browsing), ClaudeBot, PerplexityBot, Google-Extended, and Applebot-Extended.

Crawler access is table stakes. The content those crawlers find matters far more than the robots.txt file itself. Practitioner analysis of AI citation patterns reported by Search Engine Land found that AI systems favor pages with clear structure, concrete facts backed by numbers, author attribution, and original analysis [9]. Open the door, then give the crawler something worth citing.

Citation strategy gets fuller treatment in the pieces on ai search visibility metrics kpis and brandrank.ai visibility insights analysis.

The configuration that maximizes citation potential:

User-agent: GPTBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Google-Extended
Allow: /

User-agent: Applebot-Extended
Allow: /

User-agent: *
Disallow: /admin/
Disallow: /wp-admin/
Disallow: /checkout/
Disallow: /cart/

The wildcard block at the bottom catches everything else and only restricts paths that are useless to anyone, human or machine. Your public content stays fully open to every compliant crawler.

One thing people skip: add your sitemap. Put a Sitemap directive near the top of robots.txt so crawlers find all your content efficiently. Sitemap: https://yourdomain.com/sitemap.xml. It is part of the standard and honored by every major crawler.

Are there legal obligations or industry standards I need to know about?

The Robots Exclusion Protocol has no legal force on its own. It is a voluntary convention. But several legal frameworks are starting to reference crawling behavior in ways that matter to you.

The EU AI Act entered into force in August 2024 and includes transparency provisions around AI training data [10]. It does not directly mandate robots.txt compliance. It does create an environment where major AI providers operating in the EU have strong reasons to demonstrate good-faith respect for publisher preferences, robots.txt signals included.

In the US, the Computer Fraud and Abuse Act has come up in scraping cases, notably hiQ Labs v. LinkedIn, though no definitive federal ruling treats robots.txt compliance as a legal duty. The Ninth Circuit's 2022 ruling in that case held that scraping publicly available data does not violate the CFAA [11]. Terms of service violations are a separate matter.

The short version on law: robots.txt is not legally binding in most jurisdictions, but ignoring it while also breaching a site's terms of service builds a stronger case in litigation. The major AI companies carry robots.txt compliance language in their published docs partly for this reason.

On licensing, several publishers have signed deals directly with OpenAI and Google for training data access. Those deals typically override robots.txt restrictions. If you have one, confirm with your counterpart whether your robots.txt settings interfere, because they might be blocking crawlers your license actually permits.

The IAB Tech Lab has been building standards for AI training data permissions through 2024 and 2025, including possible extensions to robots.txt syntax. None are finalized or widely adopted yet. Worth watching at iabtechlab.com.

How often should I review and update my robots.txt for AI crawlers?

Quarterly, at minimum. The AI crawler landscape moves fast. New user-agent strings appear, companies split crawlers into more granular bots, and now and then a crawler you allowed changes its behavior in a way you care about.

Set a recurring reminder to check:

1. OpenAI's GPTBot documentation for new or renamed user-agents.
2. Anthropic's support pages for ClaudeBot updates.
3. Google Search Central docs for any new Extended or AI-specific user-agents.
4. Perplexity's developer docs.

When a major AI product launches, check right away whether it has a documented crawler. Several crawlers show up in logs months before the company formally documents them. If an unfamiliar user-agent that looks like an AI company appears in your logs (common patterns are [CompanyName]Bot or [CompanyName]Crawler), read the company's developer docs before you block or ignore it.

Fold robots.txt into any major site restructure review. Move content from /blog/ to /articles/ and your existing Disallow and Allow paths may now point at nothing. A structural change that orphans your robots.txt rules is a common and easily missed error.

For teams running an ongoing ai seo program, robots.txt belongs in the quarterly technical audit next to structured data validation, sitemap freshness, and page speed. It is not exciting. It is foundational.

Sources

1. IETF, Robots Exclusion Protocol Internet Draft (Google, 2022)
2. OpenAI, GPTBot documentation
3. Google Search Central, Google-Extended user-agent documentation
4. Originality.ai, Analysis of robots.txt blocking and AI traffic (2024)
5. Google Search Console, robots.txt report documentation
6. Common Crawl Foundation, dataset documentation
7. Anthropic, ClaudeBot crawler documentation
8. Spawning.ai, Have I Been Trained and noai/noimageai meta tag proposal (2023)
9. Search Engine Land, AI citation pattern analysis (2023)
10. European Parliament, EU AI Act (Regulation 2024/1689)
11. U.S. Court of Appeals, Ninth Circuit, hiQ Labs v. LinkedIn (2022)