Free robots.txt generator

Default crawl setting

Sitemap URL

Absolute URL. Helps search engines discover every page.

Bot-specific rules

Override the default for a specific crawler.

Google

Standard web crawler

Googlebot-Image

Google Images

Googlebot-Mobile

Google mobile crawler

Bingbot

Bing & Microsoft Copilot

Yahoo

Yahoo Search

Baidu

Baidu Search

ChatSEOOurs

On-demand · chatseo.app/bot

Restricted directories

One path per line. Each will be added as a Disallow rule.

What is a robots.txt file?

A robots.txt is a plain-text file at the root of your domain that tells search engines and other crawlers which parts of your site they can request. It's the first thing Googlebot fetches when it visits, and getting it right protects your crawl budget so search engines spend their time on the pages that actually matter. The free robots.txt generator above helps you create a robots.txt file in seconds — no syntax to memorize, no editor to fight with.

How to create a robots.txt file

  1. 1

    Fill in your settings

    Choose Allow all or Block all, set a crawl-delay if you need one, paste your sitemap URL, and add any directories you want crawlers to skip.

  2. 2

    Generate live

    The robots.txt file builds in real time on the right — every change updates the output instantly. This robots.txt maker validates as you type and flags common mistakes.

  3. 3

    Copy & paste to your root domain

    Hit Copy or Download .txt to create your robots.txt file, then upload it to the root of your site so it's reachable at the path below.

Your robots.txt file must be accessible at yourdomain.com/robots.txt — not in a subfolder, not on a CDN.

Robots.txt directives explained

User-agent
Identifies which bot the rule block applies to. Use * to target every crawler, or a specific name like Googlebot or Bingbot to target one.
Disallow
Blocks the listed URL path. Disallow: /admin/ stops crawlers from requesting anything under /admin/. Disallow: / blocks the entire site.
Allow
Explicitly allows a path inside an otherwise-blocked directory. Used to override a Disallow — for example, allowing one file inside a blocked folder.
Crawl-delay
Asks bots to pause this many seconds between requests. Bingbot, Yandex, and Baidu honor it; Googlebot ignores it (set crawl rate in Search Console instead).
Sitemap
Declares the absolute URL of your XML sitemap. Helps every search engine and AI crawler discover all your URLs faster than link-only crawling.

Robots.txt examples

Copy-paste ready snippets for the most common scenarios.

Allow all bots (default for most sites)

The standard setup: every crawler welcome, sitemap declared.

User-agent: *
Disallow:

Sitemap: https://yourdomain.com/sitemap.xml

Block all bots (maintenance mode)

Use during a private staging period or major migration. Don't forget to remove this before launch.

User-agent: *
Disallow: /

Block specific AI bots

Block named AI training crawlers while keeping search engines allowed. Add as many User-agent blocks as you need.

User-agent: GPTBot
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: *
Disallow:

Block specific directories

Hide admin, staging, or temporary folders from every crawler at once.

User-agent: *
Disallow: /admin/
Disallow: /staging/
Disallow: /tmp/

Sitemap: https://yourdomain.com/sitemap.xml

Robots.txt vs sitemap — what's the difference?

They look related but do opposite jobs. You almost always want both.

robots.txt — what NOT to crawl

A short text file that tells crawlers which URLs they should skip. It's a polite traffic sign, not a lock — it controls crawling, not indexing.

sitemap.xml — what TO crawl

An XML list of every URL you want search engines to find. It speeds up discovery and signals priority. Declare it inside robots.txt with the Sitemap: directive.

Hot take

Should you block AI crawlers in 2026?

The reflex over the last two years has been to slam the door on every AI bot. We think that's the wrong call for almost every site. Here's why.

AI assistants are the new front door

ChatGPT, Perplexity, Claude, Gemini, and Copilot now collectively send measurable referral traffic — and that share is growing every quarter. If your content isn't in their training data and live retrieval index, you don't show up in answers. You're not protecting yourself; you're un-listing your business from the channel that's eating Google's lunch.

Citations are the new backlinks

When ChatGPT or Perplexity cites your page as a source, you get brand exposure plus a click — without any link-building work. Sites that block GPTBot and PerplexityBot disqualify themselves from this entirely. Sites that allow them get free distribution to a billion+ AI users.

Blocking AI bots doesn't actually protect your content

Common Crawl (CCBot), academic datasets, and a long tail of scrapers will keep ingesting your content regardless. Real protection lives at the WAF or paywall level — Cloudflare bot rules, Vercel firewall, server-side rate limits — not in a text file every bot decides whether to obey.

When blocking does make sense

Block AI crawlers when you sell the content itself: paywalled investigative journalism, original research datasets, premium courses, proprietary databases. Also reasonable if a single bot is hammering your origin and inflating costs — block that one bot, not all of them. For everyone else (SaaS, ecom, agencies, personal sites, blogs), allowing AI crawlers is upside with no real downside.

Our take: keep them allowed by default

Until AI assistants stop driving traffic and citations, blocking them is a strategic mistake dressed up as a safety measure. If you do want to block one (the example block above shows the syntax), block specific bots with a clear reason — not all of them by default.

Frequently asked questions

Robots.txt is one file. ChatSEO audits 50+ others.

  • Connects to your Google Search Console for real ranking data
  • Audits sitemap, schema, internal links, and on-page SEO automatically
  • Free to start, no credit card required
Try ChatSEO free