zivooo
FREE ROBOTS.TXT GENERATOR

Free robots.txt generator: choose which bots get in.

Updated

This robots.txt generator writes the file for you from a preset and a choice per bot. It knows 19 search and AI bots, keeps the ones that answer people apart from the ones that train models, and checks the file it writes before you copy it.

Pick a preset to block AI crawlers in robots.txt while staying in AI answers, or set each bot yourself. It all runs in your browser: free, no signup, nothing sent to us.

  • Free
  • No signup
  • Runs in your browser
Start from a preset
Choose bot by bot

Bots that answer people

  • OAI-SearchBotOpenAI
    OAI-SearchBot
  • ChatGPT-UserOpenAI
    ChatGPT-User
  • Claude-SearchBotAnthropic
    Claude-SearchBot
  • Claude-UserAnthropic
    Claude-User
  • PerplexityBotPerplexity
    PerplexityBot
  • Perplexity-UserPerplexity
    Perplexity-User
  • GooglebotGoogle
    Googlebot
  • bingbotMicrosoft
    bingbot
  • ApplebotApple
    Applebot
  • meta-webindexerMeta
    meta-webindexer
  • Amzn-SearchBotAmazon
    Amzn-SearchBot
  • DuckAssistBotDuckDuckGo
    DuckAssistBot

Bots that train models

  • GPTBotOpenAI
    GPTBot
  • ClaudeBotAnthropic
    ClaudeBot
  • Google-ExtendedGoogle
    Google-Extended
  • Applebot-ExtendedApple
    Applebot-Extended
  • CCBotCommon Crawl
    CCBot
  • meta-externalagentMeta
    meta-externalagent
  • AmazonbotAmazon
    Amazonbot
Every other botBots not listed here follow the * group.
Every other bot
One per line, like /admin/. Bots you block entirely stay out anyway.
ROBOTS.TXT · NORTHWIND-CRM.COMEXAMPLE
# Kept out of the whole site
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: CCBot
User-agent: meta-externalagent
User-agent: Amazonbot
Disallow: /

# Every other bot
User-agent: *
Disallow: /admin/

Sitemap: https://northwind-crm.com/sitemap.xml
WHAT THIS TOOL WRITES

What the robots.txt generator builds

A file you can upload as it is, checked before you copy it.

  • 19 bots, by name

    OpenAI, Anthropic, Perplexity, Google, Microsoft, Apple, Meta, Amazon, DuckDuckGo and Common Crawl, each bot listed with its company.

  • Answering or training

    Bots that fetch pages for answers are listed apart from those that collect pages for training, so you can keep one and drop the other.

  • Four presets

    Stay in AI answers but skip training, allow everything, block AI bots, or block everything for staging.

  • A group per choice

    Blocked bots share one group with Disallow: /, and every other bot follows the * group.

  • Private folders

    Each folder you list becomes a Disallow line for every bot you let in.

  • Your sitemap

    A Sitemap line with the full address, which the major crawlers read.

  • Checked before you copy

    The file runs through the same parser as our Robots.txt Checker, and each bot's verdict is confirmed against your choice.

  • Apple's fallback handled

    Applebot follows Googlebot's rules when unnamed, so when the two differ we name Applebot outright.

HOW IT WORKS

From preset to uploaded file in five steps.

  1. 01

    Pick a preset

    The closest match to what you want.

  2. 02

    Adjust each bot

    Allow or block any of the 19.

  3. 03

    Add folders

    Paths no bot should crawl, one per line.

  4. 04

    Add your sitemap

    The full address of your sitemap.

  5. 05

    Copy or download

    Upload it to your site's root as robots.txt.

WHY IT MATTERS FOR AI ANSWERS AND SEARCH

Why blocking AI crawlers takes care

Most AI companies run two kinds of bot. One fetches pages to answer a person's question right now; the other collects pages to train future models. OpenAI's OAI-SearchBot and GPTBot, Anthropic's Claude-SearchBot and ClaudeBot, and Meta's meta-webindexer and meta-externalagent all split this way.

A file that blocks "all AI" with one group often blocks both, and the site quietly drops out of ChatGPT, Claude and Perplexity answers. A group per choice keeps it exact: training bots out, answer bots in.

Google is the exception. AI Overviews and AI Mode come from the same Googlebot crawl as search, so blocking Googlebot costs you SEO and AEO at once. Google-Extended is a separate token for use in Gemini and Vertex AI, and Google says it has no effect on Search.

HOW TO READ YOUR FILE

What each preset writes, and what it costs you

Every file has the same shape: a group for the bots you let in when the rest are closed, a group for the bots you block, the * group for everyone else, then your Sitemap line. A comment marks each group so you can find it later.

Under the file we show how many of the 19 bots it allows and blocks, read back with the checker's own rules. If that count isn't what you picked, the file isn't what you meant.

robots.txt is a request, not a lock. The major bots say they follow it, but it doesn't hide a page from people, or from bots that ignore it.

PRESETSEARCH AND AI ANSWERSMODEL TRAININGUSE IT WHEN
Stay in AI answersAllowedBlockedYou want citations but not training
Allow every botAllowedAllowedYou have nothing to keep out
Block AI botsGoogle, Bing and Apple onlyBlockedYou accept leaving AI assistants' answers
Block everythingBlockedBlockedA staging or private site
HOW TO PUT IT LIVE

How to upload and test your robots.txt

  • Where it goes:

    the root of each host, as example.com/robots.txt. A subdomain like blog.example.com needs its own file.

  • On WordPress:

    SEO plugins such as Yoast and Rank Math have a robots.txt editor, and a real file in the site root takes over from the one WordPress makes.

  • On Shopify, Webflow or Wix:

    Shopify edits it in the theme's robots.txt.liquid template; Webflow and Wix have a robots.txt field in their SEO settings.

  • After you upload:

    run the Robots.txt Checker on your site to confirm each bot gets the verdict you chose.

To block AI training bots while keeping answer bots in, the core is two groups:

ROBOTS.TXT
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
User-agent: meta-externalagent
User-agent: Amazonbot
User-agent: Google-Extended
User-agent: Applebot-Extended
Disallow: /

User-agent: *
Allow: /
USE CASES

When to generate a new robots.txt.

  • Launching a new site: start from Allow every bot with your sitemap, and keep admin folders closed.
  • Opting out of AI training: keep training bots out without leaving ChatGPT, Claude or Perplexity answers.
  • Cleaning up an old file: years of plugin rules pile up; a fresh file with only what you need is easier to read.
  • Setting up staging: Block everything keeps a test site out of crawls, and makes it obvious which file must never go live.
  • Agencies onboarding a client: write the same clear bot policy for every site you manage.
GO FURTHER WITH ZIVOOO

Write it today. Zivooo keeps checking it.

Zivooo's Site Health can check bot access in robots.txt every month, alongside the rest of your technical audit, and AI crawler tracking shows which bots actually come. $10 a domain a month after 7 days free, no card.

  • Bot access checked in every audit
  • AI crawler visits from your own logs
  • Whether AI assistants name you
  • Articles for the questions buyers ask AI
FREE ROBOTS.TXT GENERATOR VS OTHER OPTIONS

This robots.txt generator vs plugins and hand editing

Writing the file by hand works if you know the rules, and one missing colon undoes a line. SEO plugins write a file for you but rarely list AI bots by name. Many free generators offer one "block AI" switch that also blocks the bots that answer people. This one lists 19 bots by what they do, writes a group per choice and checks the result with the same parser as our checker. It uploads nothing: you copy the file and put it on your site.

FAQ

Generating robots.txt, answered.

How do I create a robots.txt file?

Pick a preset here, adjust any bots, add folders and your sitemap, then download the file. Upload it to your site's root so it opens at yourdomain.com/robots.txt.

How do I block AI crawlers with robots.txt?

Give the AI bots a group with Disallow: /. To stay in AI answers, block only the training bots (GPTBot, ClaudeBot, CCBot, meta-externalagent, Amazonbot, Google-Extended, Applebot-Extended) and leave the search bots allowed.

Will blocking GPTBot remove my site from ChatGPT?

No. GPTBot only collects pages for training. ChatGPT's answers use OAI-SearchBot and ChatGPT-User, which this generator lets you keep.

Should I block Google-Extended?

Only if you don't want your pages used in Gemini and Vertex AI. Google says it doesn't affect Search or rankings, which come from Googlebot, as do AI Overviews.

What should a basic robots.txt include?

A User-agent: * group with any folders to keep out, and a Sitemap line. If you have nothing to hide, User-agent: * with Allow: / and your sitemap is enough.

Does robots.txt keep a page private?

No. It asks bots not to crawl, but the file is public, people can still open the page, and a blocked page can still be indexed from links. Use a login or noindex for that.

Do I need a robots.txt at all?

No. Without one, every bot may read every page. You need one to keep something out, or to point crawlers to your sitemap.

Let the right bots in, keep the rest out.

Your keys, your costs, $10 a domain. 7 days free, no card.

Start free, no card