Allow or block AI crawlers in robots.txt

AI crawlers fall into two groups: bots that fetch pages for answers and bots that collect training data. Here is how to allow one and block the other without breaking search.

DeskSEO editorial team · Nov 8, 2026 · 5 min read

Every AI company that crawls the web publishes the name its crawler sends in the User-Agent header, and most of them say the crawler respects robots.txt. That gives a site owner a real lever: a few lines of plain text decide which assistants can read the site and which cannot. The catch is that "AI bots" are not one thing. Some fetch pages so an assistant can quote and link to them in an answer; others collect text to train future models. Blocking the first group removes the site from places where people are asking questions. Blocking the second group is a business decision with no effect on search traffic.

Two groups, two different decisions

The table below lists the crawlers the free AI bot checker reads for, grouped by what they are used for according to each vendor's own documentation.

User-agent token	Vendor	Used for	Group
`Googlebot`	Google	Google Search, including AI Overviews	Search / answers
`Bingbot`	Microsoft	Bing Search and Copilot answers	Search / answers
`OAI-SearchBot`	OpenAI	Search results shown inside ChatGPT	Search / answers
`ChatGPT-User`	OpenAI	Fetching a page when a ChatGPT user asks for it	Search / answers
`Claude-SearchBot`	Anthropic	Search inside Claude	Search / answers
`Claude-User`	Anthropic	Fetching a page when a Claude user asks for it	Search / answers
`PerplexityBot`	Perplexity	Perplexity's search index	Search / answers
`GPTBot`	OpenAI	Collecting training data	Training
`ClaudeBot`	Anthropic	Collecting training data	Training
`Google-Extended`	Google	Gemini training and grounding; does not affect Search	Training
`Applebot-Extended`	Apple	Apple model training	Training
`CCBot`	Common Crawl	Open web archive many models are trained on	Training
`Meta-ExternalAgent`	Meta	Meta model training	Training

Two tokens in that list are not real crawlers. Google-Extended and Applebot-Extended never appear in your access logs; the pages are still fetched by Googlebot and Applebot. The tokens exist only so robots.txt can express "you may index this, but do not use it for model training". That is why blocking Google-Extended leaves Google Search untouched.

Vendors rename and add crawlers from time to time. Before copying any list, including this one, check the vendor's current crawler documentation for the exact token.

How robots.txt groups actually work

Most mistakes come from one rule: a crawler obeys only the single group that matches it most specifically, and ignores every other group. Rules are not added together. If there is a group for GPTBot, then GPTBot reads that group and nothing from the User-agent: * group applies to it.

User-agent: *
Disallow: /admin/
Disallow: /cart/

User-agent: GPTBot
Disallow: /private-reports/

In this file GPTBot is allowed into /admin/ and /cart/, because its own group does not repeat those lines. Whenever a named group is added, copy the general restrictions into it as well.

Other details worth knowing:

Several User-agent lines can share one group: list them one under another, then the rules.
User-agent matching is case-insensitive; paths are case-sensitive. Disallow: /Blog/ does not block /blog/.
An empty Disallow: means "nothing is disallowed". Disallow: / means everything is.
When an Allow and a Disallow rule both match a URL, crawlers that follow the robots.txt standard (RFC 9309) use the longest, most specific match.
robots.txt controls crawling, not use of content that was already collected. Blocking a training crawler today does not remove pages from a dataset gathered last year.

Three common setups

Allow everything (the default)

If there is no group naming a bot and the * group does not block it, the bot is allowed. A missing robots.txt (a 404) also means everything is allowed. You do not need to list AI crawlers just to permit them.

Stay visible in AI answers, opt out of training

This is the most common choice for publishers and businesses that want citations and referral clicks but do not want to supply training data:

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: CCBot
User-agent: Meta-ExternalAgent
Disallow: /

User-agent: *
Disallow: /admin/
Disallow: /cart/

Sitemap: https://example.com/sitemap.xml

Search and answer crawlers have no group of their own, so they fall into * and keep access to the public site.

Block one assistant entirely

To keep a single vendor out of both search and training, name all of its tokens. Blocking GPTBot alone does not stop OAI-SearchBot or ChatGPT-User.

User-agent: GPTBot
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
Disallow: /

Be aware of the trade-off: the site then cannot appear as a cited source in that assistant's search results.

A note on user-triggered fetchers

ChatGPT-User and Claude-User fetch a page because a person asked the assistant to look at it, not to build an index. Vendors describe these user-triggered fetchers separately from their autonomous crawlers, and the wording on whether robots.txt applies to them has not been identical across vendors or over time. If keeping them out matters for your site, read each vendor's current crawler page and back robots.txt with a server-side rule rather than relying on robots.txt alone.

robots.txt is not the only gate

A perfectly written robots.txt does nothing if the request never reaches it. Firewalls and bot-protection services decide before robots.txt is read:

Cloudflare bot settings. "Bot Fight Mode", managed challenges and the AI-crawler blocking toggle can serve a challenge page to crawlers regardless of what robots.txt says. A crawler that cannot solve a JavaScript challenge simply sees a challenge page. The AI bot checker flags this when the response carries Cloudflare's challenge header.
Security plugins and WAF rules that block "unknown bots" or whole cloud IP ranges, which is where many crawlers run.
Rate limits that return 429 after a handful of requests.
Login walls and geo-blocking.

If your server logs show no visits from a crawler you allowed, check these layers first.

Common mistakes and how to spot them

A leftover Disallow: / from staging. The whole site is blocked for everyone. The SEO checker reports this as a critical issue.
A named group that forgot the general rules. Described above; compare every named group against *.
Blocking Googlebot to stop AI Overviews. That removes the site from Google Search. Google-Extended does not control AI Overviews either; according to Google, appearing in Search features is governed by Googlebot and the usual snippet controls such as nosnippet.
Typos in tokens. GPT-Bot or Claudebot-Search match nothing, so the bot falls back to *.
robots.txt served as HTML by a single-page app that answers every URL with the same page. Crawlers cannot parse it. Open /robots.txt in a browser and confirm it is plain text.

Checking the result

Run the free AI bot checker after every change. It reads robots.txt the way a standards-following parser does, shows each crawler as allowed or blocked on the homepage, marks bots that have their own group, and warns when a bot challenge is likely to stop crawlers before robots.txt matters. For the wider question of whether assistants actually mention your brand once they can read the site, see how to measure AI visibility.

Questions

Does blocking GPTBot remove my site from ChatGPT search?

No. GPTBot collects training data. ChatGPT search uses OAI-SearchBot, and pages a user asks about are fetched by ChatGPT-User. Each needs its own rule.

Does blocking Google-Extended affect my Google rankings?

No. Google states that Google-Extended only controls use of content for Gemini training and grounding. Crawling and ranking in Google Search are handled by Googlebot.

I allowed a crawler but it never visits. Why?

Usually a firewall, bot-protection challenge, rate limit or IP block stops the request before robots.txt is read. Check your CDN or security plugin settings and your server logs.