Which AI crawlers should you allow?

The search and user crawlers by engine

• ChatGPT uses OAI-SearchBot to surface sites in ChatGPT search, and ChatGPT-User to fetch a page when a user asks. OpenAI's page, checked Oct. 28, 2026, says robots.txt rules may not apply to ChatGPT-User because a user starts the request.
• Perplexity uses PerplexityBot to surface and link sites in its results, and Perplexity-User to visit a page during a user's question. Perplexity's page, checked Oct. 28, 2026, says Perplexity-User generally ignores robots.txt.
• Gemini can ground answers in Google's Search index. Google's crawler page, updated Oct. 15, 2026, says the Google-Extended token controls that grounding in Gemini Apps and Vertex AI, as well as Gemini training.
• Claude uses Claude-SearchBot to index content for search and Claude-User to fetch pages for a user's question. Anthropic's help article, dated April 7, 2026, says disabling either one may reduce your visibility in Claude's search results.
• Google AI Overviews and AI Mode are part of Google Search. Google's AI features guide, updated Dec. 10, 2025, says robots.txt rules for Googlebot are the control, and that pages need to be indexed and eligible for a snippet.

Training crawlers are a separate decision

GPTBot collects content that may train OpenAI's models. ClaudeBot does the same for Anthropic. Blocking either one tells the vendor to leave your content out of training, per each vendor's documentation checked Oct. 28, 2026.

Google-Extended works differently. Google's crawler page, updated Oct. 15, 2026, says it does not affect inclusion or ranking in Google Search. It does cover grounding in Gemini Apps, so blocking it can remove your pages from some Gemini answers.

How to set it up

1. Open your robots.txt and list every AI user agent it names today.
2. Allow OAI-SearchBot, PerplexityBot, Claude-SearchBot and Googlebot on the pages you want cited.
3. Decide on GPTBot, ClaudeBot and Google-Extended as a training and grounding question for your legal and content teams.
4. Check your CDN or firewall too. Anthropic's article warns that blocking by IP can stop a bot from reading robots.txt at all.
5. Allow time for changes. OpenAI says a robots.txt change can take about 24 hours to reach ChatGPT search, and Perplexity says up to 24 hours. Opted out sites can still appear as plain navigational links.

Sources

1. Overview of OpenAI Crawlers · OpenAI, Oct. 28, 2026Supports: Checked Oct. 28, 2026, for OAI-SearchBot, GPTBot and ChatGPT-User roles, opt out effect on ChatGPT search and the 24 hour update time.
2. Perplexity Crawlers · Perplexity, Oct. 28, 2026Supports: Checked Oct. 28, 2026, for PerplexityBot and Perplexity-User roles, robots.txt behavior and the 24 hour update time.
3. Does Anthropic crawl data from the web, and how can site owners block the crawler? · Anthropic, April 7, 2026Supports: ClaudeBot, Claude-User and Claude-SearchBot roles, what blocking each does and the IP blocking warning.
4. Google's common crawlers · Google Search Central, Oct. 15, 2026Supports: Google-Extended covers Gemini training and grounding and does not affect Google Search inclusion or ranking.
5. AI features and your website · Google Search Central, Dec. 10, 2025Supports: Googlebot robots.txt rules control access for AI Overviews and AI Mode, which need indexed, snippet eligible pages.

Related questions

• How do you get cited by ChatGPT?
• What is LLM SEO?

Intrica gets crypto, fintech and AI companies named in AI answers. AI search optimization →