Free AI crawler checker: see which AI bots can read your site.
Updated
This AI crawler checker tests your robots.txt against 19 bots from OpenAI, Anthropic, Perplexity, Google, Microsoft, Apple, Meta, Amazon, DuckDuckGo and Common Crawl. It tells the bots that answer people apart from the ones that train models, then visits your site as six AI search bots to see if your server lets them in.
Run it after a migration, a new firewall rule or any edit to robots.txt. Free, with no signup and no email.
- Free
- No signup
- Results in about 20 seconds
- PerplexityBotDisallow: /blog/
- Perplexity-UserDisallow: /blog/
- DuckAssistBotDisallow: /blog/
What the AI crawler checker tests
Robots.txt read the way Google reads it, then a real visit to your server.
19 bots in your robots.txt
The search and training bots of OpenAI, Anthropic, Perplexity, Google, Microsoft, Apple, Meta, Amazon, DuckDuckGo and Common Crawl, listed in full below.
Answering or training
Each bot is marked by what it does. A blocked search bot keeps you out of that assistant's answers; a blocked training bot only keeps you out of training.
Your page and your homepage
Both are tested, so a rule that blocks only your blog shows up.
The line that decided
For every verdict, the robots.txt rule that caused it and the group the bot obeyed.
Apple's fallback
Applebot follows Googlebot's rules when your file doesn't name it, as Apple documents, so we test it that way.
A live visit as six AI search bots
We load your homepage with each bot's user agent and compare the answer with a normal visit.
Firewalls and Cloudflare
A refusal (401, 403, 406, 429 or 503) is flagged, and so is a site served by Cloudflare.
From your address to every bot's verdict in five steps.
- 01
Enter a page
Your homepage or any page you care about.
- 02
We read robots.txt
From your site's root, as every bot does.
- 03
We test each bot
With the rules Google follows: the bot's own group first, the longest match wins.
- 04
We visit as AI bots
Six AI search bots and one normal visit to your homepage.
- 05
You get the verdicts
With the deciding rule and the lines to copy.
Why AI crawler access decides whether you're quoted
When someone asks ChatGPT, Claude or Perplexity a question, the assistant sends a search bot to fetch the pages it will quote. If your robots.txt or your server turns that bot away, your page can't be in the answer.
Each company runs more than one bot. OpenAI's GPTBot collects pages for training, while OAI-SearchBot and ChatGPT-User fetch pages for answers. A site that blocks "all AI" with one rule often blocks the bots it wanted to keep.
Google is different: AI Overviews and AI Mode use Googlebot, the same crawler as search. Google-Extended only controls use in Gemini and Vertex AI, and Google says it doesn't affect Search. So Googlebot is your SEO and your AEO at once, and the AI search bots are your GEO.
Search bots vs training bots: reading your verdicts
The headline counts only the bots that answer people. That's the number that decides whether AI assistants can quote you. Training bots are listed below it, and blocking them is a fair choice we never mark against you.
The table lists every bot we test:
The verdicts cover what your robots.txt says and what your server answered us. They don't prove a bot visited, or that an assistant will cite you.
| BOT | COMPANY | WHAT IT DOES | IF YOU BLOCK IT |
|---|---|---|---|
| OAI-SearchBot | OpenAI | Finds pages for ChatGPT search | Left out of ChatGPT search answers |
| ChatGPT-User | OpenAI | Fetches a page when a user asks | ChatGPT can't open your page for a user |
| GPTBot | OpenAI | Collects pages for training | Left out of training; answers unaffected |
| Claude-SearchBot | Anthropic | Finds pages for Claude's search | Claude can't cite your pages |
| Claude-User | Anthropic | Fetches a page when a user asks | Claude can't open your page for a user |
| ClaudeBot | Anthropic | Collects pages for training | Left out of training; answers unaffected |
| PerplexityBot | Perplexity | Finds pages for Perplexity's answers | Perplexity can't cite your pages |
| Perplexity-User | Perplexity | Fetches a page when a user asks | Perplexity can't open your page for a user |
| Googlebot | Crawls for Search, AI Overviews and AI Mode | Out of Google, AI answers included | |
| Google-Extended | A token that controls use in Gemini and Vertex AI | Left out of Gemini's use of your pages; Search unaffected | |
| bingbot | Microsoft | Crawls for Bing | Out of Bing's results |
| Applebot | Apple | Crawls for Siri, Spotlight and Safari | Out of Siri, Spotlight and Safari suggestions |
| Applebot-Extended | Apple | A token that controls Apple's model training | Left out of training; Siri unaffected |
| CCBot | Common Crawl | Builds an open crawl many models train on | Left out of that crawl |
| meta-webindexer | Meta | Finds pages for Meta AI's search answers | Meta AI can't cite your pages |
| meta-externalagent | Meta | Collects pages for training and products | Left out of training; Meta AI answers unaffected |
| Amzn-SearchBot | Amazon | Finds pages for search in Amazon's products, such as Alexa | Left out of Alexa's answers |
| Amazonbot | Amazon | Collects pages that may train Amazon's models | Left out of training; Alexa unaffected |
| DuckAssistBot | DuckDuckGo | Fetches pages for DuckDuckGo's AI answers | Left out of those answers; search rankings unaffected |
How to let AI search bots back in.
A "User-agent: *" group blocks everything:
give AI search bots a group of their own. A bot follows the group that names it and ignores the rest.
A folder is blocked:
check rules like "Disallow: /blog/" in the group each bot follows. Allow the folder, or move the rule into a training bot's group.
robots.txt allows it, but the server says 403:
look at your security plugin, host firewall or Cloudflare's AI crawler setting.
A staging file went live:
a launch that shipped "Disallow: /" for everyone needs the real file back first.
Stay in AI answers while keeping training bots out:
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
Allow: /
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
User-agent: Google-Extended
User-agent: Applebot-Extended
Disallow: /When to check your AI crawlers.
- After a launch or migration: a staging robots.txt that blocks every bot is an easy mistake to ship.
- After you blocked GPTBot: make sure ChatGPT's search bot, a different bot, still gets in.
- Publishers who want answers, not training: confirm the split works as you meant it.
- After a new firewall or Cloudflare rule: see whether AI bots are refused at the server.
- Before a pitch: agencies show a prospect which AI assistants can't read their site.
Allowed is step one. Zivooo shows who really visits.
Zivooo runs these checks on every page each month and reads your own site's logs to show which AI bots visit and what they read. $10 a domain a month after 7 days free, no card.
- AI crawler visits from your own site's logs
- Bot access on every page, in every audit
- Whether ChatGPT, Gemini and Google name you
- Branded reports for each client
Free AI bot checker vs Search Console and server logs
Google Search Console's robots.txt report covers Googlebot, and Bing Webmaster Tools tests bingbot; neither tests the bots behind ChatGPT, Claude or Perplexity. Your server logs are the final word on which bots came, but they take work to read. This free AI bot checker covers 19 bots in one run. If you were looking for a tool that spots AI written text, this isn't that: it checks which bots can read your site.
AI crawlers, answered.
Can ChatGPT crawl my site?
ChatGPT uses OAI-SearchBot to find pages for its search answers and ChatGPT-User to open a page a user asks for. If your robots.txt and server let both in, ChatGPT can read your site. GPTBot is separate and only collects pages for training.
Does blocking GPTBot stop ChatGPT from citing me?
No. OpenAI sets each bot separately, so blocking GPTBot keeps you out of training while OAI-SearchBot can still find you for answers. Block OAI-SearchBot and you leave ChatGPT search.
Should I block AI training bots like GPTBot, ClaudeBot and CCBot?
It's your call. Blocking them keeps your pages out of future training and doesn't remove you from AI answers, as long as the search bots stay allowed.
Does blocking Google-Extended hurt my Google rankings or AI Overviews?
Google says Google-Extended doesn't affect Search. AI Overviews and AI Mode use Googlebot, so as long as Googlebot is allowed, blocking Google-Extended only opts you out of use in Gemini and Vertex AI.
Do AI crawlers respect robots.txt?
The major companies say their crawlers do. OpenAI and Perplexity also say a fetch a user asks for, by ChatGPT-User or Perplexity-User, may not follow it, since a person asked for that page.
My robots.txt allows AI bots. Why are they still blocked?
Something in front of your site may be refusing them: a security plugin, a host firewall or a CDN setting. The live visit in this check shows the status your server gave each AI search bot.
What if my site has no robots.txt?
Then every bot may read every page, which is how robots.txt works. You only need one to keep something out.
How do I allow AI search bots but block training bots?
Give each kind its own group, as in the code above: the search bots with "Allow: /" and the training bots with "Disallow: /". Then run this check again to confirm.
Keep checking.
- AI READINESS & TECHNICALSEOGEORobots.txt CheckerValidate your robots.txt line by line and test any bot on any path.Check my robots.txt
- AI READINESS & TECHNICALSEOGEORobots.txt GeneratorBuild a robots.txt from presets and 19 named bots, then copy or download it.Generate my robots.txt
- AI READINESS & TECHNICALGEOAEOllms.txt CheckerTest your llms.txt against the format, and see if its links work.Check my llms.txt
- AI READINESS & TECHNICALAEOGEOSEOAI Readiness CheckerSee if AI search bots can reach, read and quote a page.Check my AI readiness
Let the right bots in, then get named.
Your keys, your costs, $10 a domain. 7 days free, no card.