robots.txt Generator
Build your robots.txt file visually. Add user-agent groups, allow/disallow rules, sitemaps, and crawl delays with one-click presets.
Group 1
Sitemaps
robots.txt Output
User-agent: * Disallow:
What is robots.txt?
robots.txt is a text file placed at the root of your website that tells search engine crawlers which pages they can or cannot access. It's part of the Robots Exclusion Protocol.
Common Use Cases
- Block admin or private directories from indexing
- Prevent AI crawlers (GPTBot, CCBot) from scraping your content
- Set crawl delays to reduce server load
- Point crawlers to your sitemap
Frequently Asked Questions
What is robots.txt?
robots.txt is a text file at the root of your website that tells search engine crawlers which pages they can or cannot access. It follows the Robots Exclusion Protocol.
Can robots.txt block AI crawlers?
Yes. You can add rules for user agents like GPTBot, CCBot, and Google-Extended to prevent AI companies from scraping your content for training data.
Does robots.txt prevent indexing?
No. robots.txt only controls crawling, not indexing. To prevent indexing, use a noindex meta tag or X-Robots-Tag header instead.
Related Tools
All Toolsllms.txt Generator & Validator
Draft and check llms.txt structure. Website discovery and URL checks use server requests.
Meta Tag Generator
Build title, description, Open Graph and Twitter Card tags with illustrative previews.
Schema Markup Generator
Build JSON-LD for articles, products, businesses and more; validate eligibility before publishing.
Privacy Policy Template Generator
Draft an editable website policy template. Review it against your real data practices and legal requirements.