Blocking AI crawlers: training vs. search

GPTBot trains, OAI-SearchBot searches, ChatGPT-User fetches on request

OpenAI describes its crawlers in its official documentation, and each one is set separately. GPTBot collects pages that may be used to train its models; disallowing it signals that your content should not be used that way. OAI-SearchBot finds the pages ChatGPT shows in its search results. A site that disallows it stops appearing in those answers, except as a plain navigational link, and OpenAI warns that a change to the file takes about 24 hours to take effect.

ChatGPT-User is different. It visits a page when a ChatGPT user asks a question, and OpenAI says that because a user starts the action, robots.txt rules may not apply. It isn't used to decide whether a page appears in search.

Other companies follow the same split, documented on their own sites. In its page on its crawlers, Anthropic separates ClaudeBot, which collects for training, Claude-SearchBot, for its search, and Claude-User, which fetches a page when a user asks. Perplexity says in its documentation that PerplexityBot serves its search results and isn't used to train models, and that Perplexity-User, triggered by a user, generally ignores robots.txt. CCBot is the crawler of Common Crawl, a foundation that makes its web archives available to anyone, model training included.

Two of these names aren't crawlers. Google-Extended has no user agent of its own: Google crawls with its usual crawlers, and the name in robots.txt only decides whether crawled content can train future Gemini models and ground their answers in Gemini apps. Google's list of common crawlers states that it has no effect on a site's inclusion in Google Search. Apple says the same about Applebot-Extended in its page on Applebot: the name crawls nothing, it decides how pages are used for its models, and a page that refuses it can still appear in its search results.

Google's AI Overviews and AI Mode are part of Google Search. To show up there as a link, a page has to be indexed and eligible to appear with a snippet. To limit what Google takes from it, its page on AI features points to Search's own controls, not to Google-Extended.

Opting out of training without leaving search

The three jobs call for different decisions, because they don't cost you the same thing:

• Disallowing training crawlers (GPTBot, ClaudeBot, CCBot, Google-Extended, Applebot-Extended) asks that your pages stay out of future training. Anthropic notes that the request covers content going forward. Your presence in search answers doesn't change, as long as the search crawlers stay allowed.
• Disallowing search crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot) cuts your presence in those assistants' search results. With OpenAI, a site that refuses OAI-SearchBot isn't shown there anymore.
• User-triggered fetchers (ChatGPT-User, Claude-User, Perplexity-User) don't all follow robots.txt: OpenAI and Perplexity say the file may not apply to these visits, since a person starts them, while Anthropic says its crawlers honor it. Blocking them stops an assistant from reading your page at the moment someone asks it to.

Either way, robots.txt is a public request. A crawler that ignores it isn't stopped by it; stopping it is a job for the server, the firewall or your host.

A publisher whose text is the product, a news site or a paid knowledge base, has different reasons than a business whose site exists to be found. The first may well want to refuse training. The second has little to gain from refusing search.

Naming a crawler takes it out of the "*" group

A robots.txt file groups its rules by crawler. When a crawler finds a group with its own name, it follows that group and nothing else: the rules in the "*" group, the one for all crawlers, aren't added on top. Google says so in its reading of the standard: only one group applies to a given crawler.

The rule cuts both ways. Naming GPTBot to allow it, or to give it a special rule, frees it from every restriction written for "*": a private folder, a staging copy, files you didn't want crawled. In the other direction, a file that closes the whole site to "*" still lets in any crawler that another group allows by name.

Nothing flags these mistakes: the file looks right and every line is well formed. The guide on what robots.txt actually blocks covers the other lines crawlers skip without saying so.

This site pays the price itself. Its robots.txt opens the whole site to every crawler, then names nine AI crawlers, each in its own group, to allow them explicitly: GPTBot, OAI-SearchBot and ChatGPT-User for OpenAI, ClaudeBot and Claude-User for Anthropic, PerplexityBot, Google-Extended, Applebot-Extended and CCBot.

The "*" group already allowed them. Naming them opens nothing more; it makes the choice readable to anyone who opens the file, the companies involved included. The reason is what we sell. The audit measures what an AI assistant can read on a website, and shutting these assistants out of ours would contradict what we check on everyone else's.

The file makes one exception: the sample reports published as PDFs, which are meant to be read from the page that presents them. Written only in the "*" group, that restriction would have applied only to crawlers without a group of their own, and the nine named crawlers could have read those files. So every group carries it, ten copies of the same line.

The file doesn't name every AI crawler that exists. The ones it leaves out, Claude-SearchBot or Amazonbot for instance, follow the "*" group, which opens the site to them in the same way.

Decide job by job, then have it checked

You can make these decisions without writing a line of the file, in this order:

1. Decide on training and on search separately. The question to ask yourself: is your text what you sell, or is it there to make your business known?
2. Ask whoever runs the site, in writing, what your robots.txt currently tells each AI crawler. An SEO plugin, a theme or the platform hosting the site may have written it without anyone deciding.
3. Also ask whether a firewall or CDN setting blocks AI crawlers, whatever robots.txt says. Several of these services offer that block as a single setting, and it may have been switched on without you.
4. If the file names crawlers, require that every restriction meant for all crawlers is also written into their groups.
5. To truly close off an area, don't rely on robots.txt: a private area is protected with a password.
6. Have the file reviewed after every redesign, plugin change or move to a new host, and go over the crawler list once a year: AI companies keep adding new ones.

An llms.txt file, which summarizes the site for language models, controls no access at all. The guide on the llms.txt file covers who reads it and what it changes.

What makes a page understandable and attributable to an assistant, once it has access, is covered in the guide on visibility in AI search.

When robots.txt says yes and the server answers 403

The audit reads your robots.txt at the start of the crawl, in a browser, and applies the group rule: for each known AI crawler, it reads the group with its name if there is one, the "*" group otherwise. About ten crawlers get checked this way, from GPTBot and ClaudeBot to Google-Extended, CCBot or Bytespider, and the report says which ones are disallowed from the whole site. This doesn't count toward the score: blocking and allowing are both defensible choices. The report only flags a block that contradicts a goal the site states, and reports a block on Bing's crawler separately, since Microsoft's Copilot assistant relies on that search engine.

The audit then checks what the server sends these crawlers. On a few pages of each type, it requests the page under the names GPTBot, ClaudeBot and PerplexityBot, and compares the result with what a browser and Googlebot receive. A page where the search engine gets the content and these crawlers get an empty shell, or a refusal, is reported, even when robots.txt allows them. That finding is published without affecting the score.

Our published report on a sporting goods marketplace shows why both readings matter. Its robots.txt disallows only one crawler, CCBot, Common Crawl's; GPTBot, ClaudeBot and PerplexityBot are allowed. Yet the server answers with a 118-byte 403 to every request that identifies itself as a bot, Googlebot and Bingbot included, on each of the 24 pages queried. At the same moment, it serves a browser normally: the filter works on the name the request announces.

The report spells out the limit of that measurement. It comes from an address that doesn't belong to the search engines, so it doesn't show whether genuine crawlers, coming from their own addresses, are served; the site's presence in Google results indicates that they are. It is still enough to put the question to the host: the file allows crawlers that the server turns away as soon as they give their name.

In the one on a B2B business website, this part comes down to two findings: no conversational crawler is blocked by robots.txt, and the page text is served by the server, readable without scripts.

What each company then does with your pages, and whether its crawler honors your request, shows up in your server logs, not from the outside. The audit tells you what your site declares and what it serves each crawler.