🤖 Robots.txt Generator
Create a perfect robots.txt file to guide search engine crawlers. Allow or block specific bots with ease.
Configuration
Leave empty if you don't have a sitemap.
Use cautiously. Search engines might ignore this or crawl less.
robots.txt Preview
How Robots.txt Works (and How It Breaks Sites)
robots.txt is the first file a crawler requests on your site. It's a gatekeeper, not a lock: well-behaved bots (Googlebot, Bingbot, GPTBot) obey it, bad bots ignore it. The syntax is tiny — User-agent picks who a rule group applies to, Disallow/Allow set the paths — but the stakes are not: a single stray "Disallow: /" has de-indexed entire businesses. Rule of thumb: block crawl-waste (search results pages, faceted filters, cart/checkout, admin), never block pages you want ranked, and never use robots.txt to "hide" sensitive URLs — the file is public and listing a secret path advertises it.
The AI crawler decision every site now faces
Since ChatGPT, Claude, Perplexity and Google's AI results became mainstream answers engines, robots.txt gained a second job: deciding whether AI systems may read your content. Blocking GPTBot, ClaudeBot, PerplexityBot and Google-Extended keeps your text out of AI training and AI answers — a fair choice for paywalled publishers. But for most sites the calculus is reversed: AI assistants now send real referral traffic and citations, and being invisible to them means competitors get quoted instead. This generator includes ready-made allow/block groups for every major AI crawler so the choice is explicit, not accidental.
A sane default for most websites
Allow everything by default; Disallow only /admin/, internal search, and duplicate-generating parameters; explicitly allow the AI crawlers you want citing you; end with a Sitemap: line pointing at your XML sitemap. Test the result in Google Search Console's robots.txt report before deploying — and remember changes can take up to 24 hours to be re-fetched.
Frequently Asked Questions
What is a robots.txt file?
robots.txt is a plain text file at the root of your website (yoursite.com/robots.txt) that tells search engine crawlers which parts of the site they may and may not visit. It's the first file most crawlers request before crawling anything else.
Where do I put the robots.txt file?
It must live at the root of your domain — https://www.example.com/robots.txt. Robots files in subdirectories are ignored. Each subdomain (blog.example.com) needs its own robots.txt.
Does robots.txt block a page from appearing in Google?
Not reliably. Disallow stops crawling, not indexing — a blocked URL can still appear in results (without a description) if other sites link to it. To keep a page out of Google, use a noindex meta tag or X-Robots-Tag header and let it be crawled.
What does 'User-agent: *' mean?
The asterisk is a wildcard matching every crawler that doesn't have its own specific rule group. Rules under 'User-agent: *' apply to all bots; a bot with its own named group (e.g. Googlebot) follows only its specific group.
Should I block AI crawlers like GPTBot and ClaudeBot?
It's a trade-off. Blocking GPTBot, ClaudeBot, PerplexityBot etc. keeps your content out of AI training and answers — but also removes you from AI search results, a fast-growing traffic source. Sites that want AI visibility (GEO) explicitly allow these bots.
What is 'Crawl-delay' and does Google support it?
Crawl-delay asks a bot to wait N seconds between requests. Bing and Yandex honor it; Google ignores it entirely — Googlebot's rate is managed automatically or via Search Console settings.
Should the sitemap be listed in robots.txt?
Yes. Adding a 'Sitemap: https://www.example.com/sitemap.xml' line lets any crawler discover your sitemap without you submitting it to each search engine separately. It can appear anywhere in the file.
Can a wrong robots.txt destroy my SEO?
Yes — a single 'Disallow: /' under 'User-agent: *' blocks your entire site from all crawlers and rankings will collapse within days. Always test changes; this generator prevents the most common syntax mistakes.