Reader guide
Turn this article into action
Plain-English point
Training bots and search bots do different jobs. See what blocking each of OpenAI, Anthropic and Perplexity's bots does, and how to write robots.txt rules.
Best use
Use this article when you are working on technical foundations and need a concrete GEO next step.
TL;DR: Do not block every AI bot with one rule. Training bots (GPTBot, ClaudeBot) and search bots (OAI-SearchBot, Claude-SearchBot, PerplexityBot) do different jobs, and each company lets you set them separately. Blocking a training bot keeps your content out of model training. Blocking a search bot can keep you out of AI answers. Decide on each bot, write the rules into robots.txt, then check that your firewall agrees.
Why is this question confusing?
We asked this exact question in Mettle: "Should I block GPTBot, ClaudeBot and PerplexityBot in robots.txt?" It is one of 19 questions in a tracked set we use to pick blog topics. In the last 30 days it ran 26 times. MettleGEO was mentioned in none of those runs.
We also read one answer from OpenAI, collected on Nov 10, 2026. It said blocking all three bots is "usually not necessary" and gave a three-bot block as an example. It treated the three as one group. It did not say that each company runs separate bots for training and for search. That is one answer, not a trend. But it shows why site owners end up with a rule that does more than they meant.
The three companies describe their own bots in public documentation. The sections below use those pages.
What is the difference between a training bot and a search bot?
A training bot collects pages that may be used to train AI models. A search bot collects pages so an assistant can show or link them in answers. A third kind fetches a page when a person asks for it.
| Company | Training | Search | User-triggered fetch |
|---|---|---|---|
| OpenAI | GPTBot | OAI-SearchBot | ChatGPT-User |
| Anthropic | ClaudeBot | Claude-SearchBot | Claude-User |
| Perplexity | none stated | PerplexityBot | Perplexity-User |
Perplexity says PerplexityBot is "designed to surface and link websites in search results on Perplexity" and is not used to crawl content for AI foundation models. So for Perplexity, the bot you were thinking of blocking is a search bot.
What to do: write down which of the three jobs you want to allow before you touch robots.txt.
What happens if I block each bot?
Each company says what a block does.
- GPTBot. OpenAI says disallowing GPTBot indicates that a site's content should not be used in training generative AI foundation models.
- OAI-SearchBot. OpenAI says sites that opt out will not be shown in ChatGPT search answers, though they can still appear as navigational links.
- ClaudeBot. Anthropic says blocking it signals that the site's future materials should be excluded from AI model training datasets.
- Claude-SearchBot. Anthropic says disabling it prevents indexing your content for search optimization, which may reduce your visibility and accuracy in user search results.
- PerplexityBot. It is the bot that surfaces your pages in Perplexity search results, per Perplexity's documentation.
The key point
Blocking a training bot is a policy choice about model training. Blocking a search bot is a visibility choice. A single rule that blocks both treats them as the same thing, and they are not.
What to do: if you want to appear in AI answers, keep the search bots allowed. Decide about the training bots on policy grounds.
What about bots that act when a person asks?
ChatGPT-User, Claude-User and Perplexity-User fetch a page because a person asked an assistant about it.
OpenAI says ChatGPT-User is not used for crawling the web in an automatic fashion, and because these actions are initiated by a user, robots.txt rules may not apply. Perplexity says Perplexity-User generally ignores robots.txt rules, since a user requested the fetch.
Anthropic is different. It says disabling Claude-User prevents retrieving your content in response to a user query, which may reduce your visibility for user-directed web search.
So robots.txt does not control all of these bots in the same way. Do not assume a Disallow line stops a user-triggered fetch.
What to do: read each company's page for the bot you care about. If you must stop a fetch, you need a firewall rule, not only robots.txt.
How do I allow search and block training?
Use one block per bot. OpenAI says each setting is independent of the others and gives the example of allowing OAI-SearchBot while disallowing GPTBot. Perplexity says the same about its settings.
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /
This file blocks the two training bots and allows the three search bots. Anthropic's page uses the same User-agent: ClaudeBot and Disallow: / pattern in its own example.
Allow only the paths you want crawled by changing the Allow lines. Block private areas such as admin pages and checkout for every bot.
What to do: make the change, wait a day, then re-check.
Does robots.txt alone decide whether a bot can reach me?
No. robots.txt is a request. A firewall, CDN or bot-protection rule can still refuse a bot that robots.txt allows. Anthropic also warns that IP blocking may not work correctly or persistently guarantee an opt-out.
This is the most common accident. A site allows OAI-SearchBot in robots.txt, but a security rule returns a challenge page to the bot. The assistant never sees the content. Perplexity's page includes guidance on allow-listing its bots in a WAF, and OpenAI and Perplexity publish IP lists for their bots.
AvoidCopy a block list from a forum post, then forget it. Never check what the server sends to each bot.
PreferWrite a rule for each bot, then request your home page as that bot and confirm you get a normal page.
What to do: after any robots.txt change, test with a real request. Our AI Ready Check does this for the major AI bots and flags blocks that look unintentional. The guide to what it measures explains each check.
What if I am not sure what I want?
Use a simple default. Most sites that want to be found in AI answers can follow this order.
- Allow the search bots from all three companies.
- Decide on the training bots as a policy. Either answer is valid.
- Block private paths for every bot.
- Check that your firewall serves a normal page to each allowed bot.
- Review the file whenever a company adds or renames a bot.
AvoidOne rule that blocks every AI bot
PreferSeparate rules for training bots and search bots.
AvoidAssuming Disallow stops a user-triggered fetch
PreferRead each company's page. OpenAI and Perplexity say robots.txt may not apply to those fetches.
AvoidEditing robots.txt and testing an hour later
PreferWait about 24 hours, then test.
AvoidAllowing a bot in robots.txt and blocking it in the firewall
PreferRequest your pages as that bot and confirm a normal response.
How do I know whether the change helped?
A robots.txt change only removes a barrier. It does not make assistants mention you. In the 30 days we looked at, MettleGEO's own mention rate was close to zero across our tracked questions, so we know first-hand that being reachable and being mentioned are different problems.
Track a fixed set of questions on a schedule and record which sources each answer cites. Our guide to ChatGPT brand citation tracking explains the method. To see whether assistants send you visits, use the steps in how to track ChatGPT traffic in GA4.
If you use Mettle, citation_list returns the domains cited in answers for a campaign. For our blog-topic set it returned 3,452 citations across all domains in the last 30 days. The top domain was semrush.com with 90 citations. That tells you which pages assistants already trust for these questions, which is a better target than guessing.
What is the short version?
- Training bots and search bots are different. Treat them separately.
- Blocking a search bot can keep you out of AI answers.
- robots.txt may not control user-triggered fetches.
- Firewalls can override robots.txt.
- Wait about a day, then test with a real request.
Questions this article answers
Does blocking GPTBot remove my site from ChatGPT search?
No. OpenAI says disallowing GPTBot means content should not be used in training. Search is handled by OAI-SearchBot, and OpenAI says each setting is independent of the others.
What happens if I block OAI-SearchBot?
OpenAI says sites that opt out will not be shown in ChatGPT search answers, though they can still appear as navigational links.
Is PerplexityBot a training crawler?
Perplexity says PerplexityBot is designed to surface and link websites in Perplexity search results and is not used to crawl content for AI foundation models.
Does robots.txt stop ChatGPT-User and Perplexity-User?
Not necessarily. OpenAI says robots.txt rules may not apply to ChatGPT-User because a person starts the action. Perplexity says Perplexity-User generally ignores robots.txt rules.
How long does a robots.txt change take to apply?
OpenAI says it can take about 24 hours for its search systems to adjust, and Perplexity says up to 24 hours. Test again after a day.
