AI crawlers and robots.txt: what to allow in 2026

Last updated: November 2026

If you want AI assistants to recommend your brand, allow the search and user crawlers: OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, Googlebot, Bingbot and Applebot. Training tokens (GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot) are a separate choice. Blocking them does not remove you from AI search.

Key takeaways🔗

• Every major AI company now splits its crawlers by job: one for model training, one for search indexing, and one that fetches pages when a user asks a question.

• Blocking a training crawler (GPTBot, ClaudeBot, Google-Extended, Applebot-Extended) is a licensing choice. The vendors' own docs say it does not remove you from their search features.

• Blocking a search crawler (OAI-SearchBot, Claude-SearchBot, PerplexityBot, Bingbot, Googlebot) can remove you from the answers buyers see.

• Google AI Overviews are controlled by Googlebot, not Google-Extended.

• robots.txt is only half the check. A CDN or firewall setting, such as Cloudflare's AI bot controls, can block crawlers your robots.txt allows.

Which AI crawlers should you know about in 2026?🔗

The AI crawlers that matter in 2026 fall into three groups: training crawlers, search crawlers and user-triggered fetchers. OpenAI, Anthropic and Perplexity each publish a separate user agent for each job, and Google and Apple use a "control token" that you set in robots.txt without a separate bot.

Here is what each one does, according to the vendor's own documentation:

Two more rows round out the picture. Googlebot is the crawler behind Google Search, and Google says robots.txt rules for Googlebot are how you control crawling for Search, including its AI features (Google). Applebot powers search in Spotlight, Siri and Safari (Apple).

What is the difference between training, search and user crawlers?🔗

Training crawlers gather content that may help train future models, search crawlers build an index the assistant searches when it answers, and user fetchers grab one page in real time because someone asked about it. The difference matters because only the last two decide whether your brand shows up in an answer today.

Think of it this way:

• Training affects what a model "knows" months from now, with no link back to you.

• Search affects whether your page can be found and cited with a link when a buyer asks a question.

• User fetch affects whether the assistant can open your page when a buyer pastes your URL or asks it to check your site.

OpenAI states this split directly: each setting is independent, so a site "can allow OAI-SearchBot in order to appear in search results while disallowing GPTBot" (OpenAI). For the broader picture of how assistants choose sources, see what sources ChatGPT cites.

Does blocking GPTBot or ClaudeBot hurt your AI visibility?🔗

Blocking GPTBot or ClaudeBot does not remove you from ChatGPT search or Claude search, according to OpenAI's and Anthropic's docs, because those products use OAI-SearchBot and Claude-SearchBot instead. What you give up is a possible say in how future models describe your category.

There is a real trade-off here, and it is a business decision, not a technical one. Publishers who sell their content often block training. A software company or local business that wants to be recommended may prefer to allow it, since the model's built-in knowledge can shape answers when no web search runs. Nobody has published solid evidence on how much training access changes recommendations, so be wary of anyone who gives you a number.

Does Google-Extended control AI Overviews?🔗

No. Google-Extended controls whether your content is used to train Gemini models and to ground answers in Gemini Apps and Vertex AI, and Google says it "does not impact a site's inclusion in Google Search nor is it used as a ranking signal" (Google).

AI Overviews and AI Mode are part of Google Search. Google's guidance says "there are no additional requirements to appear in AI Overviews or AI Mode," and points site owners to the normal Search controls: Googlebot rules in robots.txt, plus nosnippet, data-nosnippet, max-snippet and noindex (Google). If you block Googlebot, you lose Google Search and its AI features together. Our Google AI Overviews guide covers what to do once access is in place.

What should your robots.txt look like?🔗

Your robots.txt should allow every search and user crawler on the pages you want recommended, and then make a deliberate choice about training tokens. Here is a starting point for a brand that wants to be found in AI answers but would rather not feed training:

# Search and user crawlers: allow
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
Allow: /

# Training: opt out (optional, your choice)
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: CCBot
Disallow: /

# Everyone else, including Googlebot, Bingbot and Applebot
User-agent: *
Disallow: /admin/

Sitemap: https://example.com/sitemap.xml

If you want to allow training too, delete the middle block. A few rules to keep in mind:

• A crawler follows only its most specific group. If you add a named group for a bot, your * rules stop applying to it, so copy over any Disallow lines it still needs.

• Changes are not instant. OpenAI says it can take about 24 hours for its systems to adjust after a robots.txt update (OpenAI), and Perplexity says changes may take up to 24 hours (Perplexity).

• Crawl-delay works for some bots. Anthropic supports the non-standard Crawl-delay line (Anthropic). Use it before blocking outright if load is the problem.

• Don't block your CSS and JavaScript. Apple warns that if those resources are blocked, Applebot "may not be able to render the content properly" (Apple).

For the Bing side of this, including the Bingbot group rule, see Bing SEO for ChatGPT.

Can your CDN or firewall block AI crawlers even if robots.txt allows them?🔗

Yes. robots.txt is a request that well-behaved bots follow, but a firewall or bot-management setting can stop a crawler before it ever reads that file, and it will not show up anywhere in robots.txt.

Cloudflare is the most common case. On July 1, 2025, it announced that every new domain signing up "will now be asked if they want to allow AI crawlers" (Cloudflare). Its current docs sort AI bots into Search, Agent and Training, with a choice per group to block everywhere, block only on pages with ads, or allow. The docs also say mixed-purpose crawlers that do both search and training are treated as Training, and they set September 15, 2026 as the start date for new defaults on new domains (Cloudflare).

What to do:

1. Log in to your CDN or host and find its AI bot or bot-fight settings. In Cloudflare, look under Security Settings.

2. Make sure the Search and Agent groups are allowed if you want to be recommended.

3. If you run your own firewall rules, allow the published IP ranges. OpenAI (OpenAI), Anthropic (Anthropic) and Perplexity (Perplexity) all publish JSON lists of their crawler IPs.

4. Check your server logs for the user agents in the table above. If a bot you allow never appears, something upstream is stopping it.

Also watch for fakes. Common Crawl warns that it is "aware of crawlers falsely identifying themselves as CCBot" (Common Crawl). The IP lists let you tell real bots from impostors.

How do you check that AI crawlers can actually reach your site?🔗

You check AI crawler access in three places: your robots.txt file, your CDN or firewall settings, and your server logs. Each can block a bot that the others allow.

That last row is a common trap. A December 2024 Vercel and MERJ study found that OpenAI's and Anthropic's crawlers fetch raw HTML and do not run JavaScript (Vercel), so a page that loads its text in the browser can look empty to them. Our GEO audit checklist puts access next to the other checks that matter.

Is crawler access enough to get recommended?🔗

Crawler access is necessary but not enough. It only means AI assistants can read your pages. Whether they recommend you depends on what those pages say, what other sites say about you, and how you compare with competitors for the questions buyers ask.

The only way to know is to ask the assistants the same buyer questions over time and record who gets named. TrendlyAI does this with weekly scans across ChatGPT, Perplexity, Claude and Google AI Overviews. I track TrendlyAI's own domain this way, and our score is still low, which is a useful reminder that an open robots.txt is the start, not the finish.

For a quick baseline, run the free TrendlyAI check. It tests 5 prompts on ChatGPT and Perplexity in about 30 seconds, with no signup. For the bigger picture, start at our GEO hub.

Frequently asked questions🔗

Should I block GPTBot?🔗

It depends on your business. OpenAI's docs say GPTBot relates to model training and OAI-SearchBot controls ChatGPT search, and the two settings are independent. You can block GPTBot and still appear in ChatGPT search answers as long as OAI-SearchBot is allowed.

Does blocking Google-Extended remove me from AI Overviews?🔗

No. Google says Google-Extended does not affect inclusion in Google Search, and AI Overviews are part of Search. Google points site owners to Googlebot rules and snippet controls such as nosnippet for AI features in Search.

Do ChatGPT-User and Perplexity-User follow robots.txt?🔗

Not reliably. OpenAI says robots.txt rules "may not apply" to ChatGPT-User because a user started the action, and Perplexity says Perplexity-User "generally ignores robots.txt rules." A firewall rule is the only firm way to block them.

What is the difference between ClaudeBot and Claude-SearchBot?🔗

ClaudeBot collects content that may be used to train Anthropic's models. Claude-SearchBot indexes content to improve Claude's search results. Blocking ClaudeBot keeps your future content out of training; blocking Claude-SearchBot may reduce your visibility in Claude's answers.

How long do robots.txt changes take to work?🔗

OpenAI says about 24 hours for its crawlers, and Perplexity says up to 24 hours. Other vendors do not publish a timeline. Check your server logs a few days after a change to confirm.

Related reading🔗

• Bing SEO for ChatGPT visibility: a step-by-step guide

• How to get your brand recommended by Claude

• How to get cited by Perplexity