How web scraping policies affect AI training inclusion

TL;DR: AI companies train on web content they crawl. Your robots.txt directives and Terms of Service language can block or allow those crawlers, but blocking training data doesn't block AI citations from search-time retrieval. The two pipelines are separate, and conflating them is the most expensive mistake marketers make when thinking about AI visibility.

What does robots.txt actually do to AI training crawlers?

Robots.txt is a plain-text file at the root of your domain that tells crawlers which paths they may or may not visit. It's a request, not a technical barrier. Any crawler can ignore it. For decades this mattered mostly for Googlebot. Since 2023 it's become a fight between publishers and AI labs.

The major AI companies have published named user-agents specifically for training crawls. OpenAI uses GPTBot [1]. Google uses Google-Extended for Gemini training, separate from its search crawler [2]. Common Crawl, the nonprofit archive that supplies training data to dozens of AI projects, uses CCBot [3]. Apple's training crawler is Applebot-Extended [4]. Anthropic uses ClaudeBot, though its policy documentation is less detailed than OpenAI's.

Disallowing any of these in robots.txt tells the crawler not to fetch those pages. Whether they comply is a matter of corporate policy, legal pressure, and reputation. Not code. OpenAI and Google have both publicly committed to honoring robots.txt for their training crawlers. Common Crawl has a historically inconsistent record, though it updated its policy documentation in 2023 to encourage compliance.

The practical reality: if you block GPTBot and Google-Extended today, that content is less likely to enter the next training run for those specific models. But existing training data already ingested before you added the block stays ingested. There's no delete mechanism.

Is blocking AI training crawlers the same as blocking AI citations?

No, and this distinction is the whole ballgame.

AI assistants like ChatGPT (in browse mode), Perplexity, and Google's AI Overviews retrieve content at query time using a separate crawl pipeline. Blocking GPTBot affects training. Blocking OAI-SearchBot (OpenAI's separate retrieval crawler) affects ChatGPT's ability to cite you in real-time answers [1]. These are different bots with different disallow rules.

Perplexity uses its own crawler, PerplexityBot. Google's AI Overviews pull from the regular Google Search index, which Googlebot (not Google-Extended) builds. So a publisher who aggressively blocks every AI user-agent may block training inclusion and destroy their AI citation presence at the same time. That's almost certainly not the outcome they wanted.

Here's the rough separation:

| Goal | Relevant crawler to block | Notes | |---|---|---| | Block GPT-4 training | GPTBot | OpenAI has committed to honoring this | | Block Gemini training | Google-Extended | Separate from Googlebot | | Block Common Crawl archive | CCBot | Compliance not guaranteed | | Block ChatGPT citations | OAI-SearchBot | Blocks real-time retrieval | | Block Perplexity citations | PerplexityBot | Blocks Perplexity answers | | Block Google AI Overviews | Googlebot | Same bot as regular search |

If your goal is AI visibility (getting cited in answers), blocking training crawlers is largely irrelevant. What matters is being crawlable and authoritative for the retrieval crawlers.

For a fuller picture of how AI search retrieval works, see our overview of ai search.

Do Terms of Service prohibitions actually stop AI training?

Terms of Service are a legal instrument, not a technical one. Whether they stop AI training is a live legal question, and the honest answer is: nobody knows yet.

The core legal argument publishers make is that scraping in violation of ToS constitutes either breach of contract or, in some readings, a violation of the Computer Fraud and Abuse Act (18 U.S.C. § 1030) [5]. Courts have gone both ways. The Ninth Circuit in hiQ Labs v. LinkedIn (2022) held that scraping publicly available data isn't automatically a CFAA violation, but that ruling didn't resolve the ToS breach question cleanly [6].

For AI training specifically, the pending cases as of mid-2025 include The New York Times Co. v. Microsoft Corp. and OpenAI (S.D.N.Y., filed December 2023) and a cluster of class-action suits from authors and image creators. These cases argue that even where robots.txt allowed crawling, the copyright in the underlying works wasn't licensed for training purposes [7]. The robots.txt/ToS layer and the copyright layer are legally distinct.

Practically speaking: adding "no AI training" language to your ToS may strengthen a future copyright or contract claim. It does nothing today to prevent a crawler from fetching your pages. Major AI labs are unlikely to change behavior based on ToS language alone unless a court orders it or a settlement requires it.

Which AI crawlers have documented robots.txt compliance commitments?

| | | |---|---| | GPTBot (OpenAI training) | 3 | | OAI-SearchBot (ChatGPT retrieval) | 3 | | Google-Extended (Gemini training) | 3 | | Applebot-Extended (Apple AI training) | 2 | | PerplexityBot (retrieval) | 2 | | ClaudeBot (Anthropic training) | 1 | | CCBot (Common Crawl) | 1 |

Source: OpenAI (GPTBot docs), Google Search Central, Common Crawl Foundation, Apple Support, 2024-2025

Which AI companies actually honor opt-out signals?

OpenAI is the most documented. Their GPTBot page states that site owners can disallow GPTBot in robots.txt and OpenAI will not use that content to train models [1]. They also accept a web form opt-out for content already in training data, though the effectiveness of retroactive removal is unverified.

Google has documented Google-Extended, which affects Bard/Gemini training and Vertex AI. Disallowing it does not affect your ranking in Google Search [2]. Google has a clear incentive to be transparent here because their search business depends on publisher trust.

Meta has not published a named training crawler user-agent with equivalent clarity. Meta's AI training reportedly drew heavily from Common Crawl and licensed datasets, not live crawls.

Apple documented Applebot-Extended in 2023 as the crawler for Apple Intelligence training, separate from Applebot (which handles Spotlight and Siri web results) [4].

Anthropic uses ClaudeBot. Their documentation is thinner than OpenAI's, but they've stated in public communications that they honor robots.txt disallow rules.

Common Crawl is the wildcard. It's a nonprofit that archives the web and provides the data free to researchers and AI companies. Their crawl is not commissioned by any single AI lab, so your opt-out of GPTBot doesn't opt you out of Common Crawl, and Common Crawl data ends up in hundreds of model training runs. Blocking CCBot is your lever there, but researchers have noted that CCBot compliance historically lagged behind other major crawlers [3].

How does blocking training data affect what AI models say about your brand?

This is where the stakes get concrete for marketers.

Large language models learn brand associations, factual claims, and tone from training data. If your content never entered training, the model's knowledge of your brand comes from whatever else mentioned you: press coverage, reviews, competitor comparisons, Wikipedia. You lose control of the narrative at the source.

A 2024 study from researchers at Northeastern University analyzed citation behavior in AI-generated responses and found that sources with higher crawl accessibility and consistent structured markup were cited significantly more often in retrieval-augmented generation outputs [8]. The study didn't isolate training versus retrieval, but the directional finding matches what practitioners observe: visibility correlates with accessibility.

Blocking training crawlers to protect content also means the model learns about you secondhand. For brands with mostly positive coverage, this may be fine. For brands in contested spaces (healthcare, finance, legal), where nuance matters and competitor content dominates, being absent from training data is a real risk.

That said, if your content is high-value proprietary research or a commercial product that AI labs are monetizing without a license, the calculus flips. Some publishers have chosen to block training entirely and negotiate licensing deals instead. The Associated Press, the Financial Times, and several academic publishers have taken this path [9].

For brands focused on growing AI citation presence rather than protecting proprietary data, the generative engine optimization playbook matters more than robots.txt tuning.

What scraping policies do the major AI labs actually publish?

Here's a factual summary as of mid-2025:

OpenAI publishes a GPTBot documentation page with the user-agent string (Mozilla/5.0 ... GPTBot/1.1), the IP ranges it crawls from, and explicit instructions for robots.txt disallow syntax [1]. They also maintain OAI-SearchBot for ChatGPT Browse and plugins.

Google's documentation for Google-Extended covers Bard and Gemini products. Their Search Central page notes that Google-Extended is a standalone product token separate from Googlebot [2]. Blocking Googlebot still affects your core search presence.

Common Crawl publishes crawl terms at their main site and lists CCBot as their crawler [3]. Their archive is publicly downloadable, which means any researcher or company can train on it without touching your site again after the initial crawl.

Apple's Applebot documentation page was updated in late 2023 to describe Applebot-Extended as the agent for "improving Apple's products and services" including generative AI features [4].

Perplexity has published documentation on PerplexityBot, noting it respects robots.txt for pages disallowed to all crawlers but has been criticized by publishers for aggressive crawling of paywalled content. Perplexity's CEO addressed this in 2024, committing to better compliance, but the documentation remains thinner than OpenAI's or Google's.

For brands wanting to track how policy changes ripple into actual citation rates, ai search visibility metrics kpis is a useful next read.

Should you block AI training crawlers or allow them?

It depends on what you're optimizing for, and most brands haven't made this decision consciously.

If you're a publisher with high-value proprietary content, blocking training crawlers and pursuing licensing agreements makes strategic sense. You're asserting that your content has monetizable value beyond what ad-supported traffic generates. The AP-OpenAI deal and the Financial Times deal with OpenAI announced in 2024 show this is a real market [9].

If you're a brand selling a product or service and your content is primarily marketing and educational material, blocking AI training crawlers is almost certainly a mistake. You want the model to know your brand, your claims, your differentiators. That happens through training data and through retrieval. Blocking the former shrinks your surface area.

If you're in a regulated industry where AI models might hallucinate about your products in dangerous ways, the calculus is genuinely uncertain. There's a reasonable argument for blocking training and putting effort into controlling the retrieval-time narrative with structured, authoritative content instead.

Here's a simple decision frame:

| Situation | Recommended stance | |---|---| | Brand content, marketing-focused | Allow all, optimize for retrieval | | High-value proprietary data/research | Block training, pursue licensing | | News/journalism | Consider selective blocking + licensing talks | | Regulated product (pharma, finance) | Allow training, heavily structure retrieval content | | Competitor mentions you negatively in their crawlable content | Allow your own crawl, improve content quality |

Running an ai visibility tool audit before making any robots.txt changes will show you your current citation footprint, so you know what you'd be giving up.

How does the C2PA standard and content provenance fit into this?

The Coalition for Content Provenance and Authenticity (C2PA) is a technical standard for embedding cryptographic metadata into digital content to declare its origin and consent preferences [10]. Think of it as a machine-readable label: "this content was created by X, and X does not consent to AI training use."

C2PA is still early. As of 2025, it's built into Adobe's Content Credentials, used by some news organizations, and supported in Microsoft's AI Builder stack. AI labs have not yet committed to honoring C2PA opt-outs in training pipelines the way they've committed to robots.txt.

But the direction is important. The EU AI Act, which entered application phases in 2024-2025, requires AI systems to disclose training data provenance for high-risk applications and puts obligations on AI developers to "put in place a policy to comply with Union law on copyright and related rights" [11]. C2PA is one technical mechanism that could satisfy provenance obligations under this kind of regulatory framework.

For now, C2PA signals are worth adding if you're already using Adobe's workflow or similar tooling. They don't have a verified effect on current AI training inclusion, but they create a legal and technical record.

How does copyright law interact with AI training opt-outs?

Robots.txt and copyright are legally independent. You can allow crawling and still hold copyright. You can block crawling and still not have copyright in something that isn't copyrightable.

The current U.S. legal debate centers on whether training a model on copyrighted text constitutes "fair use" under 17 U.S.C. § 107 [12]. The four-factor fair use test includes the effect on the market for the original work, which is where AI training cases tend to turn. If an AI model can substitute for purchasing access to the original content, that weighs against fair use.

The New York Times lawsuit argues that GPT-4 can reproduce Times articles near-verbatim and that this market substitution is direct and measurable [7]. OpenAI argues training is transformative. Courts will decide, probably over several years.

For a brand that isn't a news organization, the copyright concern is usually lower. Marketing content, blog posts, and product pages rarely have the kind of commercial value that makes the fair use analysis come out differently. But if your business model includes content subscriptions or data licensing, the copyright layer matters as much as the robots.txt layer.

The practical takeaway: robots.txt opt-outs and ToS restrictions are complementary to copyright, not a substitute. If you want legal protection, you need the copyright registration, more than the disallow rule.

What's the best robots.txt setup for AI visibility right now?

If your goal is maximum AI citation presence, the setup is straightforward: allow the crawlers, structure your content well, and keep your retrieval-time pages fast and semantically clear.

Here's a robots.txt configuration for a brand that wants to allow training and retrieval across major AI systems:

User-agent: GPTBot Allow: /

User-agent: Google-Extended Allow: /

User-agent: CCBot Allow: /

User-agent: OAI-SearchBot Allow: /

User-agent: PerplexityBot Allow: /

User-agent: Applebot-Extended Allow: /

For a brand that wants to block training but preserve retrieval citations:

User-agent: GPTBot Disallow: /

User-agent: Google-Extended Disallow: /

User-agent: CCBot Disallow: /

User-agent: OAI-SearchBot Allow: /

User-agent: PerplexityBot Allow: /

User-agent: Applebot-Extended Disallow: /

Beyond robots.txt, content quality and structure matter more for citation frequency. Research on AI-generated answer sources consistently shows that pages with clear entity markup, first-person factual claims, and high inbound link authority get cited more often [8]. Robots.txt gets you in the door. Content quality determines if you're cited.

For deeper coverage on how to optimize content for AI retrieval, the ai seo guide covers the full content strategy layer. If you want to benchmark your current AI citation rate before making policy changes, tools like those covered in ai seo tools can give you a baseline.

How should brands monitor whether their policy choices are working?

This is genuinely hard because AI labs don't publish crawl logs, training dataset composition is confidential, and there's no direct signal analogous to Google Search Console's crawl stats for training pipelines.

What you can monitor:

Retrieval-time citations: Query major AI assistants for your brand and track whether they cite your URLs. This is imperfect (small sample, varies by phrasing) but directionally useful. Tools that automate this at scale are covered in ai visibility tool.

Crawl logs on your server: Your server access logs will show GPTBot, OAI-SearchBot, Google-Extended, CCBot, and PerplexityBot hits if they're visiting. This confirms crawlers are finding you before you worry about what they do with the content.

AI answer monitoring: Services like Spawned track brand citation frequency across ChatGPT, Perplexity, Claude, and Gemini at scale, which lets you see directional changes after you make robots.txt updates. It typically takes 60 to 90 days after a training-data policy change to see any effect in model outputs, because model training cycles aren't continuous.

Common Crawl index lookup: The Common Crawl project publishes an index you can query at index.commoncrawl.org to see whether your domain is in recent crawls and how often [3].

There's no clean feedback loop here, and anyone selling a precise "training inclusion score" is probably oversimplifying. The honest approach is to make your content technically accessible, monitor retrieval citations quarterly, and revisit your robots.txt decisions annually as the legal and technical landscape shifts.

Sources

1. OpenAI, GPTBot documentation
2. Google Search Central, Google-Extended documentation
3. Common Crawl Foundation, About and Terms
4. Apple, Applebot support documentation
5. U.S. Congress, Computer Fraud and Abuse Act (18 U.S.C. § 1030)
6. Ninth Circuit Court of Appeals, hiQ Labs v. LinkedIn (2022)
7. U.S. District Court S.D.N.Y., The New York Times Co. v. Microsoft Corp. and OpenAI (filed December 2023)
8. Northeastern University, study on AI citation behavior in retrieval-augmented generation (2024)
9. Reuters, reporting on AP-OpenAI and FT-OpenAI licensing deals (2024)
10. Coalition for Content Provenance and Authenticity (C2PA), specification overview
11. European Parliament, EU Artificial Intelligence Act (Regulation 2024/1689)
12. U.S. Copyright Office, fair use overview (17 U.S.C. § 107)