zivooo
FREE ROBOTS.TXT CHECKER

Free robots.txt checker: find the lines bots skip or misread.

Updated

This robots.txt checker reads your whole file the way Google does: every group, every rule, every Sitemap line. It flags the lines bots skip or misread, shows which of 19 search and AI bots can open your page, and lets you test any bot on any path.

Use it as a robots.txt tester after you change a rule, or as a validator after a migration. Free, with no signup and no email.

  • Free
  • No signup
  • Results in about 15 seconds
WHAT THIS TOOL CHECKS

What the robots.txt checker validates

Every line of the file, read with the rules in Google's robots.txt guide and RFC 9309.

  • Every line, numbered

    Your file as served, line by line, with each finding marked on the line it came from.

  • Syntax errors

    Lines with no colon, rules above the first User-agent, empty user agents, misspelled fields and files past Google's 500 KiB limit.

  • Fields Google ignores

    Crawl-delay, Host, Noindex and anything else Google doesn't read, with who does read them.

  • Groups and rules

    Each group with the bots it names and the rules under it, as a crawler sees them.

  • Sitemap lines

    Each Sitemap address checked for a full URL, and the first three loaded to see that they answer.

  • 19 bots on your page

    Googlebot, bingbot, the AI search bots and the training bots, each with the rule that decided.

  • Any bot, any path

    A tester in the result that runs the same matcher on your file, for bots and paths we don't list.

HOW IT WORKS

From your address to a line by line verdict.

  1. 01

    Enter your site

    Any page works; robots.txt always lives at the root.

  2. 02

    We fetch the file

    From /robots.txt on that host, as every crawler does.

  3. 03

    We read each line

    Groups, rules, sitemaps and the lines bots skip.

  4. 04

    We test 19 bots

    On the page you entered, longest match first.

  5. 05

    You test the rest

    Any bot and any path, right in the result.

WHY IT MATTERS FOR SEARCH AND AI ANSWERS

Why one bad robots.txt line can hide your site

robots.txt is the first file a crawler asks for. Googlebot reads it before it crawls a page, and so do the bots behind ChatGPT, Claude and Perplexity. A page they're told to skip can't be crawled for search or fetched for an AI answer.

Small mistakes carry far. A rule with no colon is skipped, so a folder you meant to close stays open. A stray "Disallow: /" under User-agent: * closes the site to every bot without its own group. And since Google retired its robots.txt Tester in 2023, Search Console shows what Google fetched but no longer tests a path for you.

For SEO, robots.txt decides what Googlebot crawls, and so what can rank. For AEO, AI Overviews and AI Mode draw on that same Googlebot crawl. For GEO, each AI assistant sends its own bot, and each one obeys only the group that names it.

HOW TO READ YOUR RESULT

Errors, warnings and notices: what each one means

Each finding sits on the line it came from. Errors are lines bots skip, or that do the opposite of what they say. Warnings work, but probably not the way you meant. Notices are worth knowing and often fine.

The bot table tests the page you entered with Google's rules: a bot follows the group that names it, else the * group, and the longest matching rule wins, with Allow winning a tie. Applebot follows Googlebot's group when your file doesn't name it, as Apple documents.

A clean file only means bots read it as you wrote it. It doesn't show whether a firewall turns bots away; our AI Crawler Checker visits as the bots to test that.

LEVELWHAT IT MEANSEXAMPLES
ErrorBots skip the line, or the whole fileA line with no colon, a rule above the first User-agent, a web page served as robots.txt
WarningIt works, but likely not as you meantNoindex in robots.txt, a path without a leading slash, Googlebot shut out
NoticeWorth knowing, often fineCrawl-delay, no Sitemap line, one bot named in two groups
HOW TO FIX WHAT IT FINDS

How to fix the common robots.txt errors

  • A line bots skip:

    write every rule as Field: value, with the colon, one rule per line. Comments start with #.

  • A rule above the first User-agent:

    move it under the group it belongs to, or start the file with User-agent: *.

  • Noindex or Crawl-delay meant for Google:

    Google ignores both here. Use a noindex meta tag or an X-Robots-Tag header to keep a page out of results; Google sets its own crawl rate.

  • Disallow: / under User-agent: *:

    if it came from a staging site, put the live file back. If you meant it, give the bots you want a group of their own with Allow: /.

A clean starting file that keeps two folders closed and names your sitemap:

ROBOTS.TXT
User-agent: *
Disallow: /admin/
Disallow: /cart/

Sitemap: https://example.com/sitemap.xml
USE CASES

When to run a robots.txt test.

  • After you publish a change: confirm the new rule closes what it should and nothing else.
  • After a migration or relaunch: a staging file with "Disallow: /" is an easy launch mistake to miss.
  • When pages drop out of Google: check whether a rule now covers them before you look anywhere else.
  • When you block AI training bots: make sure the group you wrote doesn't catch the search bots too.
  • When a plugin or host writes the file: SEO plugins and hosting panels can add rules you never typed. See what is really served.
GO FURTHER WITH ZIVOOO

One clean check today, a full audit every month.

Zivooo's Site Health can audit your whole site every month: bot access in robots.txt, HTTP to HTTPS, redirect chains, a missing sitemap and on-page issues, with the fix for each. $10 a domain a month after 7 days free, no card.

  • Robots.txt and AI bot access in every audit
  • AI crawler visits from your own logs
  • Fixes written out for each issue
  • Whether ChatGPT, Gemini and Google name you
FREE ROBOTS.TXT VALIDATOR VS OTHER OPTIONS

Robots.txt validator vs Search Console and crawlers

Search Console's robots.txt report shows the files Google fetched and any errors, but it no longer tests a path, and it speaks only for Google. Desktop crawlers such as Screaming Frog apply your robots.txt across a whole crawl. This robots.txt validator reads the live file, flags each line, tests 19 search and AI bots and lets you try any other. It reads the file as served, not a draft.

FAQ

Robots.txt, answered.

How can I check if robots.txt is working?

Open yourdomain.com/robots.txt to see that it loads, then run it through this checker: it shows how each bot reads the file and which lines they skip.

How do I test if a URL is blocked by robots.txt?

Run the check, then use the tester under the results: pick a bot, type the path, and you get Allowed or Blocked with the rule that decided.

What happened to Google's robots.txt Tester?

Google retired it in late 2023 and added a robots.txt report to Search Console. The report shows what Google fetched and any errors; to test a rule, Google's help points to other testers.

Does Google support Crawl-delay or Noindex in robots.txt?

No. Google reads User-agent, Allow, Disallow and Sitemap. It ignores Crawl-delay, and it stopped reading Noindex in 2019, so use a noindex meta tag or header instead.

Does robots.txt stop a page from being indexed?

Not on its own. A blocked page can still be indexed from links pointing to it, just without its content. To keep a page out of results, let Google crawl it and add noindex.

How do I fix a blocked by robots.txt error?

Run the check, then test the page's path for Googlebot in the tester under the results; it names the rule that blocks it. Remove or narrow that Disallow line, or add an Allow line for the path, then ask Search Console to validate the fix.

How big can a robots.txt file be?

Google reads the first 500 KiB and ignores the rest. Most files are a few lines long; if yours is near the limit, merge rules with wildcards.

Where does robots.txt have to be?

At the root of each host, as example.com/robots.txt. A file in a folder is ignored, and a subdomain like blog.example.com needs its own.

Write the rules once, then get read.

Your keys, your costs, $10 a domain. 7 days free, no card.

Start free, no card