serkankaya.
Türkçe sürüm
Let’s work together ↗

AI Crawlers and robots.txt: GPTBot, ClaudeBot and Others

Managing GPTBot, ClaudeBot, Google-Extended and PerplexityBot with robots.txt: training, search and user bots, sample rules and how to test them.

The key to managing AI crawler bots with robots.txt is this: every AI company uses separate bots for training, search and user requests, and you manage each one with its own lines. A site that blocks GPTBot and allows OAI-SearchBot can ask that its content stay out of training while still appearing in ChatGPT search. On the Google side, Google-Extended does not affect Search or AI Overviews.

In this post I cover the documented bots from OpenAI, Anthropic, Google and Perplexity, robots.txt examples you can copy and adapt, and the interpretation mistakes I see most often. For the wider picture of AI visibility, start with my post on what GEO is; this post is its technical access side.

What jobs do AI crawlers do?

AI crawlers do three different jobs, and the robots.txt decision is made separately for each one. The first group collects content for model training. The second builds an index for AI search and decides which pages get cited. The third opens a page on the spot when a user asks a question.

This distinction has large practical consequences. Blocking a training bot means asking that your content not be used in future models. Blocking a search bot closes off your chance of being shown as a source in that platform’s answers. Blocking both on the same line is an unwanted outcome for most service sites.

Training bot Search bot User fetcher Cited in answer
To earn citations, the search bot and, when needed, the user fetcher must be able to reach the page; the training bot is not part of this chain.

robots.txt is a rules file that tells bots which URLs they may access. Its rules are defined by RFC 9309, the current specification of the protocol. Each rule group starts with a User-agent line, followed by Allow and Disallow lines.

Which AI bots exist and what do they do?

The table below groups the bots that four major companies define in their own documentation by function. Bot names can change over time, so I recommend checking the sources before copying the list into your own robots.txt.

CompanyTrainingSearchUser-triggered
OpenAIGPTBotOAI-SearchBotChatGPT-User
AnthropicClaudeBotClaude-SearchBotClaude-User
GoogleGoogle-Extended (token)Googlebot—
Perplexity—PerplexityBotPerplexity-User

OpenAI: GPTBot, OAI-SearchBot and ChatGPT-User

According to OpenAI, OAI-SearchBot is used to surface websites in ChatGPT’s search features.[1] Sites that opt out of this bot are not shown in ChatGPT search answers, though they can still appear as navigational links. GPTBot, on the other hand, collects content that may be used to train generative AI foundation models.

OpenAI states clearly that the two settings are independent.[1] On a site that allows both bots, the results of a single crawl may be used for both purposes to avoid duplicate crawling. ChatGPT-User opens a page when a user asks ChatGPT a question and does not crawl automatically. OpenAI notes that because these actions are initiated by a user, robots.txt rules may not apply.[1]

Anthropic: ClaudeBot, Claude-SearchBot and Claude-User

Anthropic uses the same three-part structure.[2] ClaudeBot collects content that could contribute to training, and when it is blocked, the site’s future content is excluded from training datasets. Claude-SearchBot indexes content to improve search result quality. Claude-User opens a page when a user asks Claude a question.

Two details in Anthropic’s documentation stand out. First, they support the non-standard Crawl-delay directive. Second, the blocking rule has to be written for every subdomain separately.[2] They also say that blocking IP addresses is not a reliable approach, because it also prevents the bot from reading robots.txt.

Google: Googlebot and Google-Extended

The situation is different at Google, because its AI features run inside Search. Google-Extended is a product token with no separate HTTP user agent; crawling is done with existing Google bots.[3] The token manages whether content may be used to train future Gemini models and for grounding in Gemini Apps and Vertex AI. Grounding is when a model draws on current content from a search index while generating an answer.

Google states that Google-Extended does not affect a site’s inclusion in Google Search and is not a ranking signal.[3] According to Google’s AI features documentation, visibility in AI Overviews and AI Mode is governed by Googlebot rules and controls such as nosnippet, max-snippet and noindex. In other words, blocking Google-Extended will not remove you from AI Overviews.

Perplexity: PerplexityBot and Perplexity-User

Perplexity states that PerplexityBot is designed to surface and link websites in search results and is not used to collect content for AI foundation models.[5] Perplexity-User opens a page while answering a user’s question. Perplexity writes plainly in its own docs that, since the user requested the fetch, this fetcher generally ignores robots.txt rules.[5]

How do you block GPTBot with robots.txt?

Short answer

How do you block GPTBot with robots.txt?

To block GPTBot, add a “User-agent: GPTBot” line to your robots.txt file with a “Disallow: /” rule below it. This rule only stops training crawls. To keep appearing in ChatGPT search, leave OAI-SearchBot open in a separate group of its own.

  1. Open the robots.txt file at your site root.
  2. Add a User-agent: GPTBot line.
  3. Write a Disallow: / rule directly below it.
  4. Define a separate OAI-SearchBot group with Allow: / for search.
  5. Publish the file and check your logs after at least 24 hours.

For most service and content sites, the starting policy I recommend is to keep search bots open and make a deliberate decision about training bots. A robots.txt file for a site that chooses to close training bots could look like this. The first group opens search and citation bots, the second closes training bots and tokens, and the third applies to all other bots.

User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
Allow: /
Disallow: /my-account/

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
Disallow: /

User-agent: *
Allow: /
Disallow: /my-account/

Sitemap: https://yourdomain.com/sitemap-index.xml

Writing several User-agent lines one after another combines them into a single group. Google’s documentation also shows that consecutive user agent lines are treated as one group and the rule applies to all of them.[4] That way you do not have to repeat the same rule for every bot.

If you also want to allow training, simply delete the training group. The Allow: / rule in the general * group then applies to those bots too. Conversely, if you want to close all AI bots, you move the search bots into the Disallow: / group as well; in that case you also close off your chance of appearing as a source in those platforms’ answers.

How are robots.txt rules interpreted for AI bots?

A bot follows only the single group in robots.txt that matches it best and ignores all other groups entirely. According to Google’s documentation, a crawler picks the group with the most specific user agent that matches it, and the order of groups in the file does not matter.[4] This rule is the source of the robots.txt mistake I see most often.

Wrong assumption

  • When you write a separate group for GPTBot, the rules in the * group still apply
  • The order of rules sets priority
  • The file on the main domain also covers subdomains
  • Blocking Google-Extended turns off AI Overviews

What actually happens

  • The bot reads only its own group; shared rules must be repeated in that group
  • Group order is irrelevant; the most specific user agent match wins
  • Every subdomain, protocol and port needs its own robots.txt file
  • AI Overviews are managed with Googlebot and snippet controls
The four most common assumptions about AI bots in robots.txt and what the documentation actually says.

That is why I repeated the Disallow: /my-account/ rule in the search bot group in the example above. The moment you define a separate group, that bot no longer sees the account page block in the * group.

There are a few more limits worth knowing from Google’s technical documentation:[4]

  • Size: Google ignores content after 500 KiB. Instead of growing the bot list endlessly, consolidate rules by directory.
  • Encoding: The file must be UTF-8 encoded plain text. In another encoding, paths with non-ASCII characters can become invalid.
  • Cache: Google generally caches the file for up to 24 hours. OpenAI also says it can take about 24 hours for a change to be reflected in search.[1]
  • Server errors: If robots.txt returns a 5xx error, Google stops crawling the site for the first 12 hours, then uses the last valid version for 30 days.
  • Scope: yourdomain.com/robots.txt applies only to the same host, protocol and port; en.yourdomain.com needs its own file.

Is robots.txt enough to stop AI bots?

robots.txt is not a security measure; it is a request that a bot chooses to honor. Documented automated crawlers say they follow the rules. For user-triggered fetchers and bots that hide their identity, however, robots.txt alone is not enough. Sensitive content needs authentication or server-side access control.

When you apply your decision, think about three layers together:

  • Verification: User agent strings are easy to spoof. OpenAI and Perplexity publish their bots’ IP ranges as JSON files.[1][5] You can use these ranges to verify whether a log request labeled GPTBot really came from OpenAI.
  • CDN and WAF: Bot protection rules in layers such as Cloudflare can stop a bot at the door even when robots.txt allows it. Perplexity notes that sites using a WAF may need to explicitly allow its bots.[5] A WAF (Web Application Firewall) is a firewall that filters requests to a web application according to rules.
  • Managed robots.txt: Cloudflare’s managed robots.txt feature can automatically add rules for training bots to your file. If you use it, the live file may differ from the one in your repository; always check the live URL.

Content Signals and AIPREF: new preference signals

robots.txt answers “may it access this URL?” but not “what may it use the content for?”. New signals are arriving to fill that gap. Cloudflare’s Content Signals format states separately whether content may be used for a search index (search), as input to AI answers (ai-input) and for model training (ai-train).

The IETF’s aipref working group is also working on a shared machine-readable signal for training preferences. AIPREF is the name of the draft signal this group is preparing, which expresses a preference about whether content may be used for AI training. Anthropic announced that, as of October 30, 2026, it has begun recognizing a “no AI training” signal based on this draft.[2] Google also states that it intends to support this shared signal in early 2027.[3]

According to Anthropic, a robots.txt rule that names ClaudeBot directly is treated as more specific than the general preference signal and takes priority.[2] Since the standards have not settled yet, the most reliable approach today is still to write your decision explicitly in robots.txt using bot names.

Which AI bots should you allow?

Which bots you allow depends on how your content makes you money. For a site that wants to bring visitors to a service or product page, search bots are a visibility channel. For a publisher that sells the content itself, training bots and even search bots call for a different calculation.

Site typeSearch botsTraining botsNote
Service site, clinic, agencyOpenDeliberate decisionCitations can be a channel for new patients and clients
E-commerceOpenDeliberate decisionClose cart, account and filter URLs to all bots
Paid content publisherCase by caseUsually closedA licensing and contract matter; get legal advice
Internal tool, test environmentClosedClosedUse authentication, not robots.txt

This decision does not end with robots.txt. Once search bots have access, your content needs to be worth citing; I build that on the technical and content foundation I describe on my SEO services page. An llms.txt file can be an extra step to help agents understand your site quickly; but llms.txt sets no access rules and does not replace robots.txt.

How do you test robots.txt for AI crawlers?

The most reliable way to test robots.txt is to check the live file and then look at your server or CDN logs to see how bots actually behave. Analytics tools do not show these requests, because bots usually do not run JavaScript. OpenAI says it may add a separate marker to the user agent on robots.txt requests, which makes it easier to tell file requests apart from page requests in logs.[1]

The robots.txt report in Google Search Console shows when Google last fetched your file and any parsing errors. According to Cloudflare, Search Console may report “Syntax not understood” for newer directives such as Content Signals. Cloudflare says it has observed no impact on crawl rates from this.

robots.txt checklist for AI crawlers0/8

Where should you start?

The first step in managing AI crawlers is to see who your current robots.txt file blocks today. Then make the search and training decision separately for each company, repeat shared rules in separate groups, and align your CDN settings with the same decision. Finally, check the result in your logs a day later.

This area is changing fast: new bots are being added, and preference signals such as Content Signals and AIPREF are maturing. That is why I recommend re-verifying your bot list against official sources every few months. You can find the content side of AI visibility in my post on what GEO is, and how I run this work on my about page.

Key terms

AI crawler
AI crawler is a bot that AI companies use to automatically download web pages for model training, AI search, or answering a user's question. Each company usually uses separate user agent names for these jobs.
GPTBot
GPTBot is the crawler OpenAI runs to collect content that may be used to train its generative AI foundation models. Blocking it in robots.txt signals that content should not be used for training; it does not affect visibility in ChatGPT search.
ClaudeBot
ClaudeBot is Anthropic's crawler that collects web content that could contribute to training its AI models. When it is blocked, the site's future content is excluded from training datasets; search uses a separate bot, Claude-SearchBot.
Google-Extended
Google-Extended is a robots.txt product token that lets publishers manage whether content Google crawls may be used to train Gemini models and for grounding in Gemini Apps. It is not a separate bot and does not affect Google Search.
Content Signals
Content Signals is a machine-readable preference format that Cloudflare adds to robots.txt, stating with search, ai-input and ai-train fields whether content may be used for a search index, as input to AI answers, or for model training.

Frequently asked questions

If I block GPTBot, will I disappear from ChatGPT search?

No. According to OpenAI's documentation, GPTBot and OAI-SearchBot are independent settings. A site that blocks GPTBot and allows OAI-SearchBot can ask that its content not be used for training while still appearing in ChatGPT search results.

Does Google-Extended block AI Overviews?

No. Google says Google-Extended does not affect a site's inclusion in Google Search and is not a ranking signal. AI Overviews are part of Search, so visibility there is governed by Googlebot rules and preview controls such as nosnippet and max-snippet.

Do AI bots follow robots.txt?

OpenAI, Anthropic and Perplexity document that their automated crawlers follow robots.txt. For user-triggered fetchers such as ChatGPT-User and Perplexity-User, their own docs state that robots.txt rules may not apply. robots.txt is a request file, not a security measure.

How do you block ClaudeBot?

Add a User-agent: ClaudeBot line to the robots.txt file at your site root, followed by a Disallow: / rule. Anthropic notes that this has to be done for every subdomain, and that IP blocking is not a reliable alternative because it also prevents the bot from reading robots.txt.

How long does a robots.txt change take to apply?

Google generally caches robots.txt content for up to 24 hours. OpenAI also says it can take about 24 hours for a robots.txt update to be reflected in its search systems. After a change, wait at least a day before checking the result.

Sources

  1. OpenAI: Overview of OpenAI Crawlers
  2. Anthropic: Does Anthropic crawl data from the web, and how can site owners block the crawler?
  3. Google Search Central: Google's common crawlers
  4. Google Search Central: How Google interprets the robots.txt specification
  5. Perplexity: Perplexity Crawlers