Robots.txt Generator

Generate robots.txt files for your website. Control which bots crawl what, set crawl delay, and link your sitemap. Free, private.

⚙️ Settings

Default Rule for All Bots: Crawl Delay (seconds): Sitemap URL: Disallowed Paths (one per line): Block Specific Bots:

GPTBot CCBot Claude Google AI ByteSpider Ahrefs Semrush

📄 Output

📝 What goes in robots.txt?

User-agent: Which crawler the rule applies to (* = all)
Allow/Disallow: Paths the crawler can or cannot access
Crawl-delay: Seconds to wait between requests
Sitemap: URL of your XML sitemap

All processing in-browser · No data sent to any server

About Robots.txt Generator

Generate robots.txt files for your website. Control which pages search engines can crawl. Set crawl delay, sitemap URLs, and per-bot rules. All in your browser — no data sent anywhere.

Visual Builder

Add allow/disallow rules without writing syntax. See the robots.txt build up as you add directives.

Per-Bot Rules

Set different rules for Googlebot, Bingbot, and generic crawlers. Block AI scrapers selectively.

Presets Included

Start from WordPress, ecommerce, or custom presets. Tweak as needed for your specific site.

Validation Built-in

Checks syntax before you download. No broken robots.txt files — validated output every time.