Free robots.txt checker: find the lines bots skip or misread.
Updated
This robots.txt checker reads your whole file the way Google does: every group, every rule, every Sitemap line. It flags the lines bots skip or misread, shows which of 19 search and AI bots can open your page, and lets you test any bot on any path.
Use it as a robots.txt tester after you change a rule, or as a validator after a migration. Free, with no signup and no email.
- Free
- No signup
- Results in about 15 seconds
- LINES
- 18
- GROUPS
- 3
- RULES
- 7
- SITEMAPS
- 1
- Line 9No field and colon, so every bot skips this line. Write it as Field: value.
- Line 5Google stopped reading Noindex in robots.txt in 2019. Put a noindex meta tag on the page instead.
- Line 13The path "private" doesn't start with / or *, so it may not match what you meant.
What the robots.txt checker validates
Every line of the file, read with the rules in Google's robots.txt guide and RFC 9309.
Every line, numbered
Your file as served, line by line, with each finding marked on the line it came from.
Syntax errors
Lines with no colon, rules above the first User-agent, empty user agents, misspelled fields and files past Google's 500 KiB limit.
Fields Google ignores
Crawl-delay, Host, Noindex and anything else Google doesn't read, with who does read them.
Groups and rules
Each group with the bots it names and the rules under it, as a crawler sees them.
Sitemap lines
Each Sitemap address checked for a full URL, and the first three loaded to see that they answer.
19 bots on your page
Googlebot, bingbot, the AI search bots and the training bots, each with the rule that decided.
Any bot, any path
A tester in the result that runs the same matcher on your file, for bots and paths we don't list.
From your address to a line by line verdict.
- 01
Enter your site
Any page works; robots.txt always lives at the root.
- 02
We fetch the file
From /robots.txt on that host, as every crawler does.
- 03
We read each line
Groups, rules, sitemaps and the lines bots skip.
- 04
We test 19 bots
On the page you entered, longest match first.
- 05
You test the rest
Any bot and any path, right in the result.
Why one bad robots.txt line can hide your site
robots.txt is the first file a crawler asks for. Googlebot reads it before it crawls a page, and so do the bots behind ChatGPT, Claude and Perplexity. A page they're told to skip can't be crawled for search or fetched for an AI answer.
Small mistakes carry far. A rule with no colon is skipped, so a folder you meant to close stays open. A stray "Disallow: /" under User-agent: * closes the site to every bot without its own group. And since Google retired its robots.txt Tester in 2023, Search Console shows what Google fetched but no longer tests a path for you.
For SEO, robots.txt decides what Googlebot crawls, and so what can rank. For AEO, AI Overviews and AI Mode draw on that same Googlebot crawl. For GEO, each AI assistant sends its own bot, and each one obeys only the group that names it.
Errors, warnings and notices: what each one means
Each finding sits on the line it came from. Errors are lines bots skip, or that do the opposite of what they say. Warnings work, but probably not the way you meant. Notices are worth knowing and often fine.
The bot table tests the page you entered with Google's rules: a bot follows the group that names it, else the * group, and the longest matching rule wins, with Allow winning a tie. Applebot follows Googlebot's group when your file doesn't name it, as Apple documents.
A clean file only means bots read it as you wrote it. It doesn't show whether a firewall turns bots away; our AI Crawler Checker visits as the bots to test that.
| LEVEL | WHAT IT MEANS | EXAMPLES |
|---|---|---|
| Error | Bots skip the line, or the whole file | A line with no colon, a rule above the first User-agent, a web page served as robots.txt |
| Warning | It works, but likely not as you meant | Noindex in robots.txt, a path without a leading slash, Googlebot shut out |
| Notice | Worth knowing, often fine | Crawl-delay, no Sitemap line, one bot named in two groups |
How to fix the common robots.txt errors
A line bots skip:
write every rule as Field: value, with the colon, one rule per line. Comments start with #.
A rule above the first User-agent:
move it under the group it belongs to, or start the file with User-agent: *.
Noindex or Crawl-delay meant for Google:
Google ignores both here. Use a noindex meta tag or an X-Robots-Tag header to keep a page out of results; Google sets its own crawl rate.
Disallow: / under User-agent: *:
if it came from a staging site, put the live file back. If you meant it, give the bots you want a group of their own with Allow: /.
A clean starting file that keeps two folders closed and names your sitemap:
User-agent: *
Disallow: /admin/
Disallow: /cart/
Sitemap: https://example.com/sitemap.xmlWhen to run a robots.txt test.
- After you publish a change: confirm the new rule closes what it should and nothing else.
- After a migration or relaunch: a staging file with "Disallow: /" is an easy launch mistake to miss.
- When pages drop out of Google: check whether a rule now covers them before you look anywhere else.
- When you block AI training bots: make sure the group you wrote doesn't catch the search bots too.
- When a plugin or host writes the file: SEO plugins and hosting panels can add rules you never typed. See what is really served.
One clean check today, a full audit every month.
Zivooo's Site Health can audit your whole site every month: bot access in robots.txt, HTTP to HTTPS, redirect chains, a missing sitemap and on-page issues, with the fix for each. $10 a domain a month after 7 days free, no card.
- Robots.txt and AI bot access in every audit
- AI crawler visits from your own logs
- Fixes written out for each issue
- Whether ChatGPT, Gemini and Google name you
Robots.txt validator vs Search Console and crawlers
Search Console's robots.txt report shows the files Google fetched and any errors, but it no longer tests a path, and it speaks only for Google. Desktop crawlers such as Screaming Frog apply your robots.txt across a whole crawl. This robots.txt validator reads the live file, flags each line, tests 19 search and AI bots and lets you try any other. It reads the file as served, not a draft.
Robots.txt, answered.
How can I check if robots.txt is working?
Open yourdomain.com/robots.txt to see that it loads, then run it through this checker: it shows how each bot reads the file and which lines they skip.
How do I test if a URL is blocked by robots.txt?
Run the check, then use the tester under the results: pick a bot, type the path, and you get Allowed or Blocked with the rule that decided.
What happened to Google's robots.txt Tester?
Google retired it in late 2023 and added a robots.txt report to Search Console. The report shows what Google fetched and any errors; to test a rule, Google's help points to other testers.
Does Google support Crawl-delay or Noindex in robots.txt?
No. Google reads User-agent, Allow, Disallow and Sitemap. It ignores Crawl-delay, and it stopped reading Noindex in 2019, so use a noindex meta tag or header instead.
Does robots.txt stop a page from being indexed?
Not on its own. A blocked page can still be indexed from links pointing to it, just without its content. To keep a page out of results, let Google crawl it and add noindex.
How do I fix a blocked by robots.txt error?
Run the check, then test the page's path for Googlebot in the tester under the results; it names the rule that blocks it. Remove or narrow that Disallow line, or add an Allow line for the path, then ask Search Console to validate the fix.
How big can a robots.txt file be?
Google reads the first 500 KiB and ignores the rest. Most files are a few lines long; if yours is near the limit, merge rules with wildcards.
Where does robots.txt have to be?
At the root of each host, as example.com/robots.txt. A file in a folder is ignored, and a subdomain like blog.example.com needs its own.
More crawl checks.
- AI READINESS & TECHNICALSEOGEORobots.txt GeneratorBuild a robots.txt from presets and 19 named bots, then copy or download it.Generate my robots.txt
- AI READINESS & TECHNICALGEOSEOAI Crawler CheckerSee which AI bots can read your site, and which you turned away.Check my AI crawlers
- AI READINESS & TECHNICALSEOSitemap CheckerValidate your XML sitemap and open a sample of the pages it lists.Check my sitemap
- AI READINESS & TECHNICALSEOAEONoindex CheckerSee if a page tells Google, Bing or AI search to stay away.Check for noindex
Write the rules once, then get read.
Your keys, your costs, $10 a domain. 7 days free, no card.