robots.txt Generator
Build your robots.txt file visually. Add user-agent groups, allow/disallow rules, sitemaps, and crawl delays with one-click presets.
Group 1
Sitemaps
robots.txt Output
User-agent: * Disallow:
What is robots.txt?
robots.txt is a text file placed at the root of your website that tells search engine crawlers which pages they can or cannot access. It's part of the Robots Exclusion Protocol.
Common Use Cases
- Block admin or private directories from indexing
- Prevent AI crawlers (GPTBot, CCBot) from scraping your content
- Set crawl delays to reduce server load
- Point crawlers to your sitemap
Frequently Asked Questions
What is robots.txt?
robots.txt is a text file at the root of your website that tells search engine crawlers which pages they can or cannot access. It follows the Robots Exclusion Protocol.
Can robots.txt block AI crawlers?
Yes. You can add rules for user agents like GPTBot, CCBot, and Google-Extended to prevent AI companies from scraping your content for training data.
Does robots.txt prevent indexing?
No. robots.txt only controls crawling, not indexing. To prevent indexing, use a noindex meta tag or X-Robots-Tag header instead.