How do I create a robots.txt file?

Pick a preset (allow all, block all, WordPress, block AI bots), add custom rules if needed, attach your sitemap URL, then copy or download and upload to your site root.

Should I add a sitemap to robots.txt?

Yes — the Sitemap: line helps every search engine find your pages and is the single most valuable line in the file.

🫛 toolspea

Home › SEO Tools › Robots.txt Generator

Robots.txt Generator

Build a correct robots.txt without memorizing the syntax — presets for common setups, custom rules when you need them.

Start from a preset

Sitemap URL (recommended)

Crawl-delay (sec)

User-agentRulePath

🔒 Generated in your browser — nothing about your site is sent anywhere

Ad Space · AdSense code goes here later

Small file, big consequences

robots.txt is the first thing search engines read on your site — and a one-character mistake can deindex everything. This generator keeps you safe: start from a preset (allow all, block all while staging, WordPress-recommended, or block AI training crawlers like GPTBot and CCBot), add custom Allow/Disallow rules per bot, and attach your sitemap URL — the single best SEO line in the file. Upload the result to your site root as /robots.txt and you're done.

FAQ

What does "Disallow:" with an empty path mean?

Empty Disallow means "block nothing" — i.e. allow the whole site. It's the standard, safest default and exactly what the allow-everything preset produces.

Does blocking a page in robots.txt hide it from Google?

It stops crawling, not necessarily indexing — a blocked URL can still appear in results if other sites link to it. To truly hide a page, use a noindex meta tag and let it be crawled.

What does the "Block AI bots" preset do?

It disallows known AI-training crawlers (GPTBot, CCBot, Google-Extended) while leaving normal search engines untouched — your rankings are unaffected.

Related Tools

🏷️ Meta Tag Generator 🔗 URL Slug Generator 📝 Word Counter 📷 QR Code Generator 🗺️ Sitemap Generator Soon 🧪 Robots.txt Tester Soon