How to Block AI Crawlers in robots.txt (and Which Ones You Should Not)

"Should I block AI crawlers?" is two questions wearing one coat, and the reason most robots.txt advice on the subject is bad is that it answers them together. One question is about licensing: do you want your text used to train the next model? The other is about distribution: do you want to appear when someone asks ChatGPT, Claude or Perplexity a question your page answers? The user agents that do those two jobs are different, they are documented separately by every major vendor, and a robots.txt that treats them as one thing either gives away the training rights you meant to keep or removes you from the answer engines you meant to be in.

This article lists every AI user agent that matters, sorted by which job it does, gives you three copy-pasteable robots.txt configurations for the three positions people actually hold, and covers the two things people get wrong after they have written the file: assuming robots.txt is enforcement, and assuming Google-Extended affects Google.

Two kinds of bot

Every large AI vendor now runs at least two distinct crawlers, and publishes them as separate tokens precisely so you can treat them differently.

Training crawlers fetch pages to build a corpus. What they collect goes into a model's weights months later. Blocking one is a licensing decision: your pages are not used for training, and nothing about your visibility today changes, because a model trained without your page does not know or care that your page exists.

Retrieval crawlers fetch pages at query time, or to build the live search index that answers are drawn from. When someone asks Perplexity a question and the answer cites three sources, those sources were retrieved by this kind of bot. Blocking one removes you from the answer. There is no later model release that gets you back in; you are simply not in the index the answer engine reads.

A third category, user-triggered fetchers, sits with retrieval. When a person pastes your URL into ChatGPT or asks Claude to read a page, the fetch is made by a token like ChatGPT-User or Claude-User. OpenAI's documentation states that ChatGPT-User is not used for training. Blocking these makes your page unreadable to a human who explicitly asked for it, which is rarely what anyone wants.

The user agents, by job

Current as of this writing. Vendors change these; each vendor's crawler documentation page is the source of truth, and the tokens below are the ones the audit engine on this site checks.

Token	Operator	Job	Blocking it means
GPTBot	OpenAI	Training	Not used to train OpenAI models. No effect on ChatGPT search answers.
OAI-SearchBot	OpenAI	Retrieval	Removed from ChatGPT search results and citations.
ChatGPT-User	OpenAI	User-triggered	A user who pastes your URL into ChatGPT gets nothing.
ClaudeBot	Anthropic	Training	Not used to train Claude models.
Claude-SearchBot	Anthropic	Retrieval	Removed from Claude's web search results.
Claude-User	Anthropic	User-triggered	A Claude user who asks for your page gets nothing.
PerplexityBot	Perplexity	Retrieval	Removed from Perplexity's index and answers.
Perplexity-User	Perplexity	User-triggered	A Perplexity user who cites your URL directly gets nothing.
Google-Extended	Google	Training opt-out	Not used to train Gemini. No effect on Google Search or AI Overviews.
Applebot-Extended	Apple	Training opt-out	Not used to train Apple's models. Applebot itself still crawls for Siri and Spotlight.
CCBot	Common Crawl	Training (indirect)	Not in the Common Crawl corpus, which many models train on.
Bytespider	ByteDance	Training	Not used to train ByteDance models. Historically poor at honouring robots.txt.
Amazonbot	Amazon	Mixed	Used for Alexa answers and, per Amazon, may be used for training.
Meta-ExternalAgent	Meta	Training	Not used to train Meta models.

Three details in that table do most of the work.

Google-Extended is not a crawler. It is a robots.txt token that Googlebot honours as a signal about whether the content Googlebot already fetched may be used for Gemini training and grounding. Blocking it changes nothing about crawling, indexing, ranking, or whether you appear in AI Overviews. AI Overviews are built on the ordinary Google Search index. Any guide that tells you Google-Extended keeps you out of AI Overviews is wrong, and the cost of believing it is usually a site owner who blocked Googlebot itself trying to achieve what they were told Google-Extended would do.

The retrieval tokens are new and many robots.txt files predate them. A file written in 2023 that blocks GPTBot and calls it a day is doing what it was written to do. But if the same file was later extended to User-agent: * with a Disallow: / under the impression that this covers "AI stuff", it is now blocking OAI-SearchBot and PerplexityBot too, and the site has quietly left every answer engine.

Vendors publish IP ranges for the retrieval bots. OpenAI, Anthropic and Perplexity all publish the address ranges their crawlers use. That matters for the last section of this article, because robots.txt is a request and IP ranges are how you enforce one.

Three configurations

Position 1: no training, stay in the answers

This is the position most publishers and most SaaS companies actually want once the two jobs are separated. Your content is not a free training corpus, and a person asking an AI assistant about your topic can still be sent to you.

# Training crawlers: no.
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: CCBot
User-agent: Bytespider
User-agent: Meta-ExternalAgent
Disallow: /

# Retrieval and user-triggered fetchers: yes.
User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
Allow: /

User-agent: *
Allow: /
Sitemap: https://example.com/sitemap.xml

The explicit Allow: / groups for the retrieval bots are not decoration. Under RFC 9309, the standard that Google, OpenAI and Anthropic all implement, a crawler uses the most specific group that names it and ignores * entirely. If your * group has any Disallow lines at all, a retrieval bot that is not named explicitly falls back to * and inherits those rules. Naming the retrieval bots and giving them their own group makes the outcome independent of whatever the * group says now or later.

Position 2: block everything AI

Some sites, particularly those whose content is the product, want out entirely.

User-agent: GPTBot
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: ClaudeBot
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: CCBot
User-agent: Bytespider
User-agent: Amazonbot
User-agent: Meta-ExternalAgent
Disallow: /

User-agent: *
Allow: /

Be clear-eyed about what this buys. You will not appear in ChatGPT search, Claude search or Perplexity answers, and users who paste your URL into those tools will be told the page could not be read. You will still appear in Google Search and in AI Overviews, because those run on Googlebot, and blocking Googlebot is a different and much larger decision. If your traffic analysis shows that AI referrals are a rounding error today, this position costs little today. Whether it costs little in two years is a bet.

Position 3: allow everything, but say so deliberately

If you want maximum distribution and do not mind training use, the right file is one that names the bots anyway:

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
Allow: /

User-agent: *
Allow: /

This looks redundant, and functionally it is. Its value is that the next person to edit the file sees the decision written down, rather than inferring it from an absence. Robots files accumulate Disallow lines over years; an explicit allow group for the bots you care about survives that.

What robots.txt does not do

It does not enforce anything. It is a published request that well-behaved crawlers honour. OpenAI, Anthropic, Google, Apple and Perplexity honour it. Several scrapers that feed smaller training sets do not, and at least one large vendor's crawler has a documented history of ignoring it. If blocking is a legal or contractual requirement rather than a preference, robots.txt is where you state it and not where you enforce it. Enforcement is at the edge: a WAF rule keyed on user agent for the honest bots, and on published IP ranges for the ones that publish them.

Cloudflare now blocks AI crawlers by default for new zones. Since mid-2025, Cloudflare's default for newly onboarded sites blocks known AI crawlers at the edge and offers a "pay per crawl" mechanism on top. This is enforcement, not a robots.txt change, and it is worth checking, because it does not distinguish training from retrieval in the way this article does. If your site is behind Cloudflare and you want Position 1, look at the AI crawler controls in the dashboard, not only at your robots.txt. A site that carefully allowed OAI-SearchBot in robots.txt and is serving it a 403 from the edge has done the work and got the opposite result. The audit engine here reports the HTTP status it received, which is the fastest way to notice.

It does not retroactively remove anything. Blocking GPTBot today does not pull your pages out of a model trained last year. It stops future collection. There is no mechanism for the other thing.

It does not stop a bot that already has the page from somewhere else. Common Crawl, syndication, cached copies and archive services all carry text that originated on your site. Blocking CCBot limits one of those channels.

Checking your own file

Three things to verify once the file is live.

1. Fetch it as the bot. curl -A "OAI-SearchBot" https://example.com/robots.txt and confirm you get the file, not a challenge page. Then fetch a real page the same way and confirm a 200. A WAF that blocks the bot returns 403 before robots.txt is ever consulted.
2. Parse it as the bot would. The rule that bites people is the one above: a named group replaces *, it does not extend it. If you have User-agent: PerplexityBot followed by a single Disallow: /private, PerplexityBot may crawl everything except /private regardless of what * says. If you meant it to also inherit the * rules, copy them into the group.
3. Check the ordering rule. Where two rules in the same group match a path, the longer one wins, and Allow wins a tie. Disallow: /docs with Allow: /docs/public lets /docs/public/anything through and blocks /docs/private. An empty Disallow: line means allow everything.

The site audit on this site runs all three: it fetches robots.txt, evaluates every token in the table above against the audited path using RFC 9309 rules, and reports retrieval blocks as failures and training blocks as informational, because one is a defect and the other is a decision. If you want the longer treatment of how robots.txt, meta robots and X-Robots-Tag divide the work, that article is here, and if your actual goal is to be quoted rather than merely fetched, how answer engines choose sources covers the part robots.txt cannot.

A short decision procedure

• Do you sell the content itself, or is the content how people find what you sell? If the latter, you want retrieval bots in. That is most businesses.
• Do you object to training use? If yes, block the training tokens. It costs you nothing in visibility.
• Are you behind Cloudflare or another edge that blocks AI crawlers by default? Then robots.txt is not the whole story. Check what the edge is doing to the retrieval bots you allowed.
• Is anything in your * group disallowed? Then name the retrieval bots explicitly with their own Allow: /, or they inherit the block.
• Google-Extended is a Gemini training opt-out. It is not an AI Overviews switch. Do not touch Googlebot to achieve that; there is no way to achieve that.

That is the whole thing. The file takes five minutes once the two questions are separated; nearly every bad outcome comes from answering them as one.