Robots.txt Generator
Build a robots.txt file with a per-rule editor: pick user-agents, Allow/Disallow patterns, crawl-delay, sitemaps, and the optional host directive. Three starter presets and live validation.
Interactive tool
Builder
What is the Robots.txt Generator?
A visual editor for the robots.txt file that lives at the root of every public website. Pick a starter preset (allow everything, block AI scrapers, block staging, WordPress defaults) or build groups from scratch: User-agent, Allow, Disallow, Crawl-delay, Sitemap, and the optional Host directive. Output is RFC 9309-conformant.
How to Use the Robots.txt Generator
- 1Start with a preset or click "Add group" to build from scratch
- 2For each group, list the user-agents it applies to and the Allow / Disallow patterns
- 3Add the canonical sitemap URL(s) at the file level (these apply to every group)
- 4Optionally set Crawl-delay per group or Host at the file level (Yandex extension)
- 5Watch the validation panel for path-format errors and conflicting rules
- 6Copy the robots.txt body and serve it at https://yoursite.com/robots.txt
- 7Use the "Test in Robots Tester" deep-link to verify allow/disallow decisions against your real URLs
Key features
Three starter presets
"Block AI scrapers" disallows GPTBot, ChatGPT-User, Google-Extended, CCBot, anthropic-ai, Claude-Web, PerplexityBot, and cohere-ai. "Block staging" disallows everything. "WordPress" disallows wp-admin (but allows admin-ajax.php) and other standard paths.
Per-group editor
Add user-agent groups, mix Allow and Disallow patterns, set per-group Crawl-delay. Drag rows to reorder.
RFC 9309 round-trip
Output passes through the same robots-parser the Robots.txt Tester uses: verified in unit tests so what you build matches what crawlers see.
Live validation
Path-must-start-with-/, Allow+Disallow conflicts, malformed sitemap URLs, unrealistic crawl-delays: all surfaced before you deploy.
Sitemap + host
Declare one or more Sitemap URLs (absolute), plus the optional Host directive (non-standard Yandex extension).
Deep-link to the Tester
One-click hand-off to the Robots.txt Tester with the generated robots.txt pre-filled, so you can confirm allow/disallow decisions against real URLs.
Why use a robots.txt?
robots.txt is the contract between your site and well-behaved crawlers. It lets you keep search engines out of admin panels, staging environments, and faceted URL space, and (since 2024) lets you block AI training scrapers without affecting normal search indexing.
Common use cases
- Blocking AI training crawlers (GPTBot, Google-Extended, CCBot, anthropic-ai, ...) without affecting Googlebot
- Keeping a staging environment off Google
- Excluding faceted URL parameters that would otherwise dilute indexing
- Declaring multiple sitemaps for a large multilingual site
- Setting a Crawl-delay for crawlers that hammer your origin too hard
robots.txt vs noindex
- robots.txt disallow prevents the page from being crawled. It does not prevent indexing. Google still sees the URL through links and may list it as a "URL only" result.
- For full removal, allow the crawl and serve a meta robots="noindex" header instead. Use the Robots Meta Generator for that.
AI scrapers: moving target
The "Block AI scrapers" preset covers the major announced user-agents as of 2026. New scrapers appear constantly: re-check the list every few months and update.
Tips & best practices
Specific groups win
Crawlers pick the most specific User-agent block that matches them. A "User-agent: Googlebot" block applies only to Googlebot, ignoring any "User-agent: *" block.
Allow wins over Disallow
When a path matches both Allow and Disallow patterns, the longer (more specific) match wins, and Allow wins ties (RFC 9309).
Sitemap URLs must be absolute
Always use https://yoursite.com/sitemap.xml. Relative paths are technically valid but every major crawler treats them as absolute against the robots.txt origin.
Validate before deploy
Hop straight into the Robots.txt Tester to confirm your most-important URLs allow/disallow correctly across Googlebot, Bingbot, and GPTBot.
Privacy & security
This tool runs entirely in your browser. The robots.txt body never leaves your device.
Related tools
- CSP Evaluator
Audit a Content-Security-Policy for unsafe directives, wildcards, missing object-src/base-uri, weak nonces, and other XSS escape hatches. Paste a policy or fetch a URL: get severity-grouped findings with remediation.
SEO Tools
- HTTP Header Checker
Inspect every response header from any URL: security (HSTS, CSP, X-Content-Type-Options, X-Frame-Options, Referrer-Policy, Permissions-Policy, COOP/COEP/CORP, cookie flags), performance (Cache-Control, ETag, compression), and the full redirect chain.
SEO Tools
- Schema Markup Generator
Generate valid JSON-LD schema markup for Article, Product, Organization, LocalBusiness, BreadcrumbList, FAQPage, Event, Recipe, VideoObject, WebSite, and WebPage: Rich Results-ready.
SEO Tools
- HTML Link Extractor
Extract every link from any HTML page: anchors, images, iframes, scripts, stylesheets, and meta canonical/og:url. Paste HTML or fetch a URL, get a categorised table with internal vs external counts and CSV export.
SEO Tools
- Open Graph Preview
Preview how a URL appears on Facebook, LinkedIn, Twitter, Slack, and Discord, with a checklist for missing OG and Twitter card tags
SEO Tools
- Redirect Rules Generator
Generate Apache .htaccess and Nginx redirect snippets from a single rule list. Supports exact, prefix, and regex matches with 301/302/307/308 status codes, query-string passthrough, case-insensitive matching, and CSV import.
SEO Tools
Frequently Asked Questions
Where should I put my robots.txt?
At the root of every host you want to control: e.g. https://example.com/robots.txt. Crawlers fetch it once per host and cache the result; subdomains need their own.
Does Disallow remove a URL from Google?
No. Disallow only blocks crawling. The URL can still be indexed via inbound links: listed as a "URL only" result. For full removal, allow the crawl and serve <meta name="robots" content="noindex">. The Robots Meta Generator handles that.
How do I block AI scrapers but keep Google indexing?
Use the "Block AI scrapers" preset. It disallows GPTBot, ChatGPT-User, Google-Extended (training-only), CCBot, anthropic-ai, Claude-Web, PerplexityBot, and cohere-ai: leaving Googlebot, Bingbot and friends unaffected.
What does Crawl-delay do?
Crawl-delay is a non-standard directive that asks the crawler to wait N seconds between requests. Bingbot and Yandex honour it; Googlebot ignores it (configure Googlebot crawl rate in Search Console instead).
Is the generator client-side?
Yes. The robots.txt body never leaves your browser. The optional "Test in Robots Tester" link opens our companion tester tool with the content pre-filled.