Robots.txt Tester
Test whether a list of URLs are allowed or blocked by a robots.txt file, across multiple user-agents. RFC 9309-compliant matching with rule and line attribution. Paste content or fetch from any domain.
Interactive tool
robots.txt
Results
LiveAdd at least one URL and one user-agent: results appear here as you edit.
What is the Robots.txt Tester?
A free Robots.txt rule tester. Paste a robots.txt file (or fetch one from any domain) and test multiple URLs across multiple user-agents at once. Results show allow/disallow per cell, the line that matched, the active crawl delay, and any sitemap or host directives the file declared. Matching follows RFC 9309 (the official Google specification).
How to Use the Robots.txt Tester
- 1Paste your robots.txt content or fetch it from a live URL
- 2Confirm the base URL (the origin the robots.txt was served from)
- 3Add one URL per line in the test field: relative paths are resolved against the base URL
- 4Toggle the user-agents you want to test against (or add a custom one)
- 5Read the live results: each cell shows Allow/Disallow plus the matching line number
- 6Iterate on the rules until every URL has the expected verdict for every UA
Key features
RFC 9309-compliant matcher
Uses the well-tested robots-parser library, which implements the official Google specification
Multi-URL × multi-UA grid
Test dozens of URLs against ten or more user-agents in one shot
Rule attribution
Every result reports which line of the robots.txt matched, so you know exactly which rule fired
Crawl-delay reporting
Per-UA crawl-delay values are surfaced inline alongside allow/disallow
Sitemap and host extraction
Lists every Sitemap: declaration and the preferred Host: directive parsed from the file
Fetch from URL
Pull /robots.txt directly from any origin via our SSRF-safe backend
Common UAs preloaded
Googlebot, Googlebot-Image, Bingbot, DuckDuckBot, GPTBot, AhrefsBot, and more: one click each
Why Test Robots.txt?
A typo in robots.txt can quietly de-index your entire site. A "Disallow: /" intended for a staging environment that ships to production is the SEO equivalent of rm -rf /. Worse, the failure mode is silent. Google just stops crawling and rankings decay over weeks. Testing your robots.txt against the actual URLs you want to allow or block before you deploy is the single cheapest piece of SEO insurance you can buy.
Common use cases
- Pre-deploy QA on robots.txt changes before they go live
- Auditing a competitor's robots.txt to understand what they block
- Verifying staging environments are properly disallowed from production crawlers
- Confirming AI scrapers (GPTBot, ClaudeBot) are blocked if that is your policy
- Diagnosing why specific URLs are not being indexed in Search Console
- Testing custom user-agents your own bots use for first-party crawls
How matching works
- 1. Find the most specific User-agent block that matches the bot (Googlebot beats *).
- 2. Within that block, find the most specific path rule that matches the URL.
- 3. The longer of the matching Allow vs Disallow rule wins (RFC 9309 §2.2).
- 4. If no rule matches, the URL is allowed by default.
- This tool reports the exact line of the matching rule so you can see Step 3 visually.
Wildcards and end-anchoring
* matches any sequence of characters within a path. $ at the end anchors the rule to the end of the URL. So "Disallow: /private/*.json$" blocks any JSON file under /private/ but allows the directory listing itself.
Common pitfalls this tool catches
- Trailing slash mismatches: "Disallow, /admin" blocks /admin and /admin/x but not /administer; "Disallow: /admin/" only blocks paths with the trailing slash.
- Order-independence: robots.txt rules are matched by specificity, not order, so reorganizing for readability does not change the verdict.
- Case sensitivity: paths are case-sensitive. "Disallow: /Admin/" does not block "/admin/".
- Group separation: separate UA blocks must not share Disallow lines; each UA needs its own group.
- Commenting: # starts a comment, but only on its own line. Inline trailing comments are unreliable.
AI scraper blocks
GPTBot (OpenAI), ClaudeBot (Anthropic), CCBot (Common Crawl), and Google-Extended (Google Bard / SGE) are the most-blocked AI bots. This tool includes GPTBot in the preset list; add the others as custom UAs.
Tips & best practices
Block noindex pages from being crawled, not just indexed
noindex via meta tag still costs crawl budget. For non-public pages, prefer Disallow in robots.txt.
Always allow CSS and JS
Google needs to render your pages with CSS and JS to evaluate them. Blocking either tanks Search Console "Mobile-Friendly" signals.
Use the User-agent: * fallback wisely
The wildcard block applies to any UA without a specific block. Most well-behaved bots respect it; aggressive scrapers ignore it.
Sitemap directives are advisory
Sitemaps in robots.txt are a hint, not authoritative. Submit your sitemap to Google Search Console and Bing Webmaster Tools as well.
Validate after every deploy
CDN rewrites can change which file is served at /robots.txt. Hit the live URL with this tool every time you deploy infra changes.
Privacy & security
Paste mode runs entirely in your browser. Fetch mode sends only the site URL to our SSRF-safe backend, which fetches /robots.txt with non-public-address blocking and response size and timeout caps. The URL is not logged.
Related tools
- Robots.txt Generator
Build a robots.txt file with a per-rule editor: pick user-agents, Allow/Disallow patterns, crawl-delay, sitemaps, and the optional host directive. Three starter presets and live validation.
SEO Tools
- Robots Meta Generator
Build a <meta name="robots"> tag and the matching X-Robots-Tag HTTP header. Combine noindex, nofollow, noarchive, nosnippet, max-snippet, max-image-preview, max-video-preview, and unavailable_after with per-engine support hints.
SEO Tools
- HTML Meta and Heading Extractor
Extract every meta, title, link, canonical, and hreflang tag from an HTML page, or analyze the H1-H6 heading outline with accessibility flags. Paste HTML or fetch any URL.
SEO Tools
- URL Normalizer
Bulk-normalise URLs against RFC 3986: lowercase scheme/host, strip default ports, remove dot segments, canonicalise percent-encoding, dedupe and sort query parameters, and optionally strip tracking parameters.
SEO Tools
- CSP Nonce & Hash Generator
Generate cryptographically-random CSP nonces and SHA-256/384/512 hashes for inline <script> and <style> sources. Bulk-hash several inline snippets at once.
SEO Tools
- XML Sitemap Validator
Validate XML sitemaps against the sitemaps.org schema. Checks URL count, file size, lastmod ISO 8601, changefreq enum, and priority range. Recursively validates sitemap-index files.
SEO Tools
Frequently Asked Questions
What standard does this tester follow?
RFC 9309 (the IETF Robots Exclusion Protocol). Matching uses the well-tested robots-parser library, which implements the same specificity rules Google publishes for Googlebot.
How does specificity work?
When both an Allow and a Disallow match the same URL, the rule with the longer path wins. If the lengths are equal, Allow wins. The User-agent block is also matched by specificity: Googlebot beats * for Googlebot, even if * is listed first.
Can I test against my own custom user-agent?
Yes. Type the UA name into the "Custom UA" field and press Enter (or Add). It is added to the test grid alongside the presets.
Why does an URL show ", " instead of Allow/Disallow?
Either the URL was invalid, or it points to a different origin than the robots.txt file. Robots.txt only governs paths under the same origin (host + scheme + port).
Is fetching robots.txt safe?
Yes. Fetch mode routes through our SSRF-safe backend. Non-public addresses are blocked, response size and timeout caps are enforced, and the URL is never logged.
Does this tool follow Crawl-delay?
It reports the Crawl-delay value per UA so you can see what the file declares, but it does not throttle this tester or any test fetch.