AI crawlers explained: which bots search, train and fetch, and what to allow

AI companies run three kinds of bot: search crawlers that build the index an assistant searches, training crawlers that collect content that may be used to train models, and user-triggered fetchers that open a page because someone asked. Google, OpenAI, Anthropic and Apple document separate controls for search and training, so you can stay in their AI search while opting out of training. Microsoft's Bing crawler page lists no training control. For most expert businesses, visibility is the goal.

The details below come from each operator's own documentation, read on 22 October 2026. Crawler names and behaviour change, so check the linked page before you edit a live robots.txt.

What are the three jobs an AI crawler can do?

A search crawler is a bot that fetches pages to build the index an AI product or search engine queries when someone asks a question. Allowing one is a precondition for being cited, not a promise: Google says indexing and serving aren't guaranteed. Blocking one stops that fetching, but each operator describes the consequence differently, as the table shows.

A training crawler is a bot that collects content that may be used to train generative models. The opt-out looks forward: Anthropic describes blocking ClaudeBot as a signal that a site's future materials should be excluded from training datasets.

A user-triggered fetcher is a bot that requests one page because a person asked an assistant about it. It does not roam the web on its own schedule.

A control token is a robots.txt name with no crawler behind it. Google-Extended and Applebot-Extended are the two this article covers. You write rules for them, but nothing visits under those names, so you will not find them in server logs. Google says Google-Extended "doesn't have a separate HTTP request user agent string", and Apple says "Applebot-Extended does not crawl webpages."

Which crawler does what, and how does each treat robots.txt?

Purpose and behaviour in this table are as each operator describes them, not as third-party bot trackers report them.

Does blocking training also block search?

For Google, OpenAI and Apple, no. Google states that Google-Extended "does not impact a site's inclusion in Google Search nor is it used as a ranking signal in Google Search." OpenAI says each of its robots.txt settings is independent, and Apple says pages that disallow Applebot-Extended can still appear in search results. Two caveats apply. Google-Extended also limits whether crawled content may be used for training Gemini models and for grounding in Gemini Apps and Vertex AI, so blocking it is a decision about those products too. And Microsoft's Bing crawler page lists no separate training control, so what Bingbot access covers beyond Bing search is not something that page settles.

Google keeps AI features and training apart. Its AI features page (opens in a new tab)^[7] says AI Overviews and AI Mode have no additional technical requirements beyond being indexed and eligible to show a snippet in Search. It points to Google-Extended for limiting "AI training and grounding in some of Google's other systems", and to nosnippet, data-nosnippet, max-snippet and noindex for limiting what Search shows.

A separate setting, the Search generative AI control (opens in a new tab)^[14] under Settings in Search Console, manages a site's inclusion in AI Overviews, AI Mode and the generative AI features in Discover. Including the site is the default, and Google says the control reached all websites worldwide as of 31 August 2026. A change generally takes a few days to apply. Google adds that the control is not a ranking or inclusion signal for other parts of Search and does not affect AI training, which is Google-Extended's job.

The costly mistake runs the other way: blocking a search crawler by accident. OpenAI says sites opted out of OAI-SearchBot "will not be shown in ChatGPT search answers", though they can still appear as navigational links, and Anthropic says blocking Claude-SearchBot may reduce your visibility and accuracy in user search results. A blanket rule, "User-agent: *" with "Disallow: /", applies to every bot that has no group of its own, so it can keep all of those crawlers out at once.

Does robots.txt force an AI bot to comply?

No, because nothing in the protocol enforces it. RFC 9309 (opens in a new tab)^[15], the Standards Track specification published in September 2022, says a crawler that downloads the file successfully "MUST follow the parseable rules", so a compliant crawler is expected to obey. The same document says "These rules are not a form of access authorization", and Google's introduction to robots.txt (opens in a new tab)^[8] says "it's up to the crawler to obey them." Operators describe their own behaviour: Google says its common crawlers always obey robots.txt when crawling automatically, and Anthropic says its bots honour it. What the standard asks and what a given bot does are separate questions. Five consequences follow.

• Private material needs access control. A password keeps a page private, a robots.txt line does not, and RFC 9309 adds that listing a path in the file exposes it publicly. Google adds that a blocked URL can still appear in results, without a description, if other sites link to it, and recommends noindex or a password to keep a page out.
• User-triggered fetchers sit outside the rules. OpenAI says robots.txt rules may not apply to ChatGPT-User, and Perplexity says Perplexity-User generally ignores them. Google says the same of its user-triggered fetchers (opens in a new tab)^[5]. Anthropic's article says its bots honour robots.txt and does not carve out Claude-User.
• Changes are not instant. OpenAI says OAI-SearchBot can take about 24 hours to adjust to a robots.txt update, Perplexity says up to 24 hours, and Google says it generally caches robots.txt for up to 24 hours.
• A failing robots.txt is handled in two layers. RFC 9309's default is that a 4xx response means the crawler may access any resource, while an unreachable file (a 5xx error) means it must assume complete disallow. Operators add their own rules. In Google's robots.txt specification (opens in a new tab)^[12], 4xx errors other than 429 are treated as if no file existed, while on a 5xx error Google stops crawling for the first 12 hours, then uses the last good copy for up to 30 days, and assumes no restrictions if it has no cached copy. This article found no equivalent documentation for the other operators.
• A named group beats the wildcard. A crawler follows the group that matches its name, falls back to the "*" group if none does, and applies no rules if neither exists. Apple adds that Applebot follows Googlebot's rules when robots.txt mentions Googlebot but not Applebot.

How can you verify that a bot is real?

User agent strings are easy to fake. Google says the string can be spoofed, and Bing says the same. Check the source IP instead, using the method each operator documents.

Do not use IP blocking as your opt-out. Anthropic warns that blocking its crawler IPs can stop the bot reading your robots.txt, so the opt-out may not hold. A firewall or bot-protection service can also block search bots you meant to allow, and Perplexity documents allow rules for Cloudflare and AWS WAF that pair the user agent with its published IP ranges.

What should an expert business allow?

Start from the goal. For an expert business the page is the shop window: a prospective client asks an assistant a question, and the answer cites what it could read. Search and retrieval access lets a page be considered for that answer, and no operator promises inclusion. Training is a separate choice about your content, and both belong to the wider work of generative engine optimisation.

The training question has no universal answer, and the operators' pages do not settle it. Three questions help. Is the content free to read, or something you sell? Would you mind it appearing in a model's training data? And will you recheck the decision when an operator changes its documentation? If the content is paid or private, a login is the control that works, because a robots.txt line is only a request. For free articles meant to be shared, allowing training is a legitimate choice, and Google, OpenAI and Apple each say the training control is separate from search.

Open crawler access still leaves a page unreadable if its facts are missing from the text. As a hypothetical check, take a service page that returns HTTP 200 with no crawler blocked, but whose fees and eligibility rules appear only inside an image or a script-rendered widget. Google's AI features page asks that important content be available in textual form, and it points to the URL Inspection tool for seeing the HTML Googlebot received. For other bots, fetch the raw HTML (for example with curl) and search it for the fee wording: if the words are missing there, a crawler that does not run scripts cannot read them. The fix is to put those facts in the page's own text under a question-style heading, as answer-first writing describes.

What does an example robots.txt look like?

This is an example, not a template to copy blindly. It allows the search and retrieval bots from the table and opts out of training through the four controls that govern it. Google-Extended also limits grounding in Gemini Apps and Vertex AI, while inclusion in Google Search is unaffected, so weigh that before disallowing it. Switch those four to Allow if you decide to permit training.

User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: Googlebot
User-agent: Bingbot
User-agent: Applebot
Allow: /

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
Disallow: /

Three details matter. First, the sample has no "" group, so under RFC 9309 a crawler that is not named has no rules and may fetch everything. That makes it a fragment, not a complete file. Second, if your file already has a "" group, keep it and copy its Disallow lines into each named group, because a crawler that matches a named group ignores the "" group. Without that copy, a path such as /app/ that "" protects becomes crawlable by the named bots. Do not list truly private paths in robots.txt at all, since RFC 9309 notes that listing a path exposes it publicly. Third, ChatGPT-User and Perplexity-User sit in the allow group to record intent, since their operators say robots.txt may not apply to them.

For comparison, plandora.com's own robots.txt on 22 October 2026 has a single group for all crawlers. It disallows /app/, /docs and some /auth paths, allows the rest and lists a sitemap. It names no AI crawler, so each one follows that single group.

How do you audit what your site allows today?

1. Open your domain's /robots.txt in a browser and confirm it loads as a normal text file, not an error page.
2. Read every group. Look for a blanket "Disallow: /" under "*" and for named groups that override it.
3. Check your firewall or CDN bot settings for rules that block the search bots in the table.
4. In Search Console, open Settings and check that the Search generative AI control matches what you intend.
5. Search your server logs for the user agent names above, then verify a sample of the IPs with the methods in the verification table.
6. Record your training policy and the date. After an edit, wait about a day before testing again, since OpenAI says about 24 hours, Perplexity says up to 24 hours and Google says it caches robots.txt for up to 24 hours.