Robots.txt tester & validator
Check whether Googlebot, Bingbot or an AI crawler may fetch a URL, and see exactly which line of the file decides it.
No file handy? .
How the verdict is decided
- Pick the group. The crawler looks for the group whose User-agent token matches its name most specifically. Googlebot-Image uses a
Googlebot-Imagegroup if there is one, otherwise aGooglebotgroup, otherwise*. Groups with the same name are merged. - Match every rule. Each Allow and Disallow path is compared against the URL path plus query string, from the start.
*matches anything;$means “ends here”. - Longest match wins. Specificity is measured in characters of the rule’s path. If an Allow and a Disallow match with equal length, Google applies the Allow.
- No match means allowed. And
/robots.txtitself is always allowed.
These are the rules of RFC 9309 as described in Google’s robots.txt documentation. Other crawlers follow the same standard, although some older bots use first-match instead of longest-match, which is one more reason to keep rules unambiguous.
Remember that blocked is not the same as de-indexed: a disallowed URL can still appear in results, without a snippet, if other pages link to it. Build or fix the file with the robots.txt generator.
Questions
How do I test if a URL is blocked by robots.txt?
Load the robots.txt (fetch it or paste it), enter the URL or path and choose a crawler. The tester picks the group that crawler obeys, lists every rule that matches, and shows which one wins under the longest-match rule of RFC 9309.
Google retired its robots.txt Tester. What replaced it?
Search Console now has a robots.txt report showing the files Google fetched and any parse errors, and the URL Inspection tool tells you if a specific URL is blocked. This tester follows the same public specification Google documents, so you can check rules before you deploy them.
Which rule wins when Allow and Disallow both match?
The rule with the longest matching path. If both are the same length, Google uses the least restrictive rule, which is Allow.
Is robots.txt case-sensitive?
Paths are case-sensitive. Field names like User-agent and Disallow are not, and user-agent values are matched case-insensitively.
Does the tester validate syntax too?
Yes. It flags lines that are not field: value pairs, rules before any User-agent, relative Sitemap URLs, and directives Google does not support such as Crawl-delay and Noindex.