Which AI crawlers should you allow in robots.txt?
Most businesses that want to be named in AI answers should allow the crawlers that feed live answers, such as OAI-SearchBot for ChatGPT search, PerplexityBot and Bingbot, along with Googlebot. Then decide separately on training crawlers such as GPTBot, ClaudeBot, Google-Extended and CCBot. A Shopify skincare brand usually allows all of them, because training data is how future models learn the brand exists. Publishers selling their content may choose differently.
Search crawlers and training crawlers do different jobs
Each AI company publishes the names its crawlers use, called user agents, and robots.txt rules apply to each name separately. Here are the ones most sites should know.
- OAI-SearchBot (OpenAI): finds pages to show and cite in ChatGPT search answers.
- ChatGPT-User (OpenAI): fetches a page when a person asks ChatGPT to open it.
- GPTBot (OpenAI): collects content that may be used to train OpenAI models.
- ClaudeBot (Anthropic): collects content for training Claude models. Anthropic documents separate agents for search and user-requested fetches.
- PerplexityBot (Perplexity): indexes pages for Perplexity's cited answers.
- Google-Extended (Google): a robots.txt token, not a separate crawler, that controls whether Google may use your content for Gemini training and grounding. It does not affect Google Search.
- CCBot (Common Crawl): builds an open web archive that many AI developers have used for training.
- Bingbot (Microsoft): indexes pages for Bing, which Copilot and other assistants draw on.
For most businesses, the default is to allow them all
If you sell products or services, being known by AI models is an advantage. Blocking a training crawler does not remove you from models that already exist. It only stops new information about you from reaching future ones, which can leave an assistant repeating old prices or missing a new location.
Blocking a search crawler is costlier: it can keep your pages out of that assistant's live answers entirely.
Blocking makes sense when your content is the product
News sites, paid research, course creators and others who earn money from the text itself may reasonably block training crawlers while still allowing search crawlers that send visitors and citations. Robots.txt is a request, not a lock. The major crawlers listed above say they honor it, but enforcement needs firewall rules.
Check your firewall as well as robots.txt
Some hosting and CDN providers, Cloudflare among them, offer settings that block AI crawlers, and these can be switched on without anyone on your team noticing. Check both places, then look at your server logs or CDN analytics to confirm the bots you allow are actually reaching your pages.
Do this next
Follow-up questions
Does blocking GPTBot keep me out of ChatGPT?
Not out of ChatGPT search answers, as long as OAI-SearchBot is allowed. Blocking GPTBot only stops your content from being used to train future OpenAI models.
Do all AI crawlers obey robots.txt?
The major named crawlers say they do, but robots.txt is voluntary. Unknown or badly behaved bots need to be stopped with firewall or CDN rules.