GEO guide · Chapter 7 of 9

How do AI crawlers, robots.txt and llms.txt work?

For owners and marketers who need to decide which AI bots can read their site, without a developer background. After reading, you will be able to check your settings and choose a sensible policy.

Updated November 6, 20265 min readBy Pop & Soda

In one paragraph

AI crawlers are bots that read your site either to train models or to fetch pages for live answers, and robots.txt is the file that tells each one whether it may. Blocking training bots like GPTBot or CCBot does not remove you from search, but blocking search bots like OAI-SearchBot or PerplexityBot can keep you out of their answers. llms.txt is a proposed extra file, not a standard. Say a clinic's web host blocks 6 AI bots by default: it can drop out of ChatGPT search and Perplexity answers.

AI crawlers come in three kinds: training, search and user-triggered

A crawler is an automated program that visits web pages and copies their content. AI companies run separate crawlers for separate jobs, and each identifies itself with a name called a user agent. Knowing which is which lets you allow some and block others. These are the main names as of late 2026; vendors add and rename bots, so check their documentation.

OpenAI: GPTBot collects content that may be used to train models; OAI-SearchBot indexes pages for ChatGPT search; ChatGPT-User fetches a page when a user's request needs it.
Anthropic: ClaudeBot collects content for training; Anthropic also documents separate agents for search indexing and user-requested fetches, named Claude-SearchBot and Claude-User.
Perplexity: PerplexityBot builds Perplexity's search index; Perplexity-User fetches pages for a user's question.
Google: Googlebot crawls for Google Search, including AI Overviews and AI Mode. Google-Extended is not a separate crawler but a control you set in robots.txt to decide whether your content may be used for Gemini model training and some Gemini features; it does not affect Google Search.
Common Crawl: CCBot builds a free public web archive that many AI models have been trained on.
Microsoft: Bingbot crawls for Bing, which feeds Copilot and other assistants.

robots.txt is a polite request, and it is easy to get wrong

robots.txt is a plain text file at the root of your site, such as yoursite.com/robots.txt, that lists which crawlers may visit which parts of the site. Each rule names a user agent and then says Allow or Disallow for a path. Major AI companies say their crawlers follow it, but it is a request, not a lock.

The most common problem is blocking bots by accident. Some hosting companies, website builders and content delivery networks now offer one-click AI bot blocking, and at least one major content delivery network turned it on by default for new sites in 2025. Firewalls and bot-protection tools can also block crawlers that robots.txt allows. Another quiet problem: many AI crawlers read the raw page and may not run JavaScript, so content that appears only after scripts load can be invisible to them.

Worked example

a B2B SaaS startup

Say a software startup notices that ChatGPT search never cites its pricing page, even though competitors' pricing pages are cited often. Its robots.txt looks fine. But the server logs show no visits from OAI-SearchBot or PerplexityBot in 90 days. The cause turns out to be a bot-protection setting on its content delivery network that blocks unfamiliar crawlers. On top of that, the pricing table is loaded by a script, so the raw page contains no prices at all.

The team allows the AI search crawlers in the network settings, renders the pricing table as plain HTML, and leaves training crawlers blocked as a deliberate choice. Within a few weeks, the logs show OAI-SearchBot and PerplexityBot visiting, and both assistants begin quoting the startup's current prices with a link.

Choose a policy: allow search, then decide on training

For most businesses that want to be recommended, the sensible default is to allow AI search crawlers and make a deliberate choice about training crawlers. Blocking search crawlers removes you from that assistant's live answers. Blocking training crawlers keeps your content out of future models, which may protect original work but also means future models know less about you.

Publishers with paid content often block training bots and allow search bots. Shops, local businesses and software companies usually benefit from allowing both.

01
Open yoursite.com/robots.txt in a browser and read every User-agent line.
02
Look for Disallow rules that name GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, Google-Extended, CCBot or a wildcard.
03
Check your host, CDN and security plugin settings for AI bot blocking switches.
04
Write your chosen rules, for example a User-agent: OAI-SearchBot group with Allow: / to permit ChatGPT search.
05
Check your server logs or analytics after two weeks to confirm the allowed bots are visiting.

llms.txt is optional, cheap and unproven

llms.txt is a proposed convention, first suggested in 2024, for a plain text file at yoursite.com/llms.txt that summarizes your site and links to its most useful pages in a format language models can read easily. It is not an official standard, and support among the major assistants is limited and not well documented.

If you have a developer handy, adding one takes under an hour and does no harm. Just do not treat it as a fix. Crawler access, clear pages and outside mentions matter far more.

Do this next

Read your robots.txt file and list every AI crawler it mentions.
Check hosting, CDN and security plugin settings for AI bot blocking.
Decide in writing whether you allow training crawlers, search crawlers or both.
Confirm in server logs that the crawlers you allow are actually visiting.
Make sure key facts, such as prices and hours, appear in the raw HTML without needing JavaScript.
Add an llms.txt file only after the steps above are done.

Quick answers

Which AI crawlers should you allow in robots.txt? What is llms.txt and do you need one?

Questions people ask

Will blocking GPTBot remove me from ChatGPT?

Not from ChatGPT search. OpenAI says GPTBot is for training and OAI-SearchBot is for search, so you can block one and allow the other.

Does blocking Google-Extended remove me from AI Overviews?

No. AI Overviews and AI Mode are part of Google Search and follow Googlebot rules. Google-Extended controls use of your content for Gemini training and some Gemini features.

Should I add an llms.txt file?

It is low cost and harmless, so add one if it is easy. Just treat it as an extra, since it is still a proposal and the major assistants have not clearly documented how they use it.

Start with a free scan.