Are you blocking the bots behind AI answers?
We test 14 bots against your robots.txt and your server. Firewalls, Cloudflare included, can block bots that robots.txt allows.
What it checks
The full rubric. No black box.
Which bots your robots.txt lets in, for the page you entered.
We request the page with each bot’s name and compare with a normal browser.
OAI-SearchBot, ChatGPT-User, PerplexityBot, Perplexity-User, Claude-SearchBot, Claude-User.
Googlebot and Bingbot. AI Overviews and AI Mode depend on Googlebot.
GPTBot, ClaudeBot, Google-Extended and others. Blocking them is a licensing choice.
How it works
Any page works.
robots.txt first, then a live request as each bot.
Answer and search bots first. Training bots are your call.
Questions about the AI Crawler Check
Should I block GPTBot?
That’s a licensing choice. GPTBot is for training. Blocking it doesn’t remove you from ChatGPT search, which uses OAI-SearchBot.
Why does it say “possible firewall block”?
We send from Cloudflare’s network with each bot’s name. Firewalls that check bot IP addresses block fakes like ours too, so confirm in your logs.
Does Google-Extended affect AI Overviews?
No. Google says it covers Gemini training and grounding in the Gemini app, not Search or AI Overviews. AI Overviews follow Googlebot and your snippet settings.
Want the gaps fixed for you?
Send your website. Get a one-page report on your next growth in Google and AI. No call required.