robots.txt tester
Avoid accidentally blocking search engines or important pages.
How it measures
- Parsing follows RFC 9309: the most specific user-agent group applies, the longest matching rule wins and allow wins ties.
- A 5xx response for robots.txt is reported because crawlers may treat the whole site as disallowed in that case.
Limits
- robots.txt is a crawling hint, not access control, and does not remove already indexed pages.
- Only the first 512 KiB of the file are parsed.
Troubleshooting
- Disallowed unexpectedly
- Check for a broad Disallow: / in the matching group or a rule with a wildcard that is longer than your Allow.