robots.txt Tester
Only the most specific matching user-agent group applies — rules from other groups, including the * group, are ignored entirely for that crawler.
Matched group: googlebot
| Path | Result | Deciding rule |
|---|---|---|
| / | Allowed | Allow: / (line 10) |
| /admin/users | Allowed | Allow: / (line 10) |
| /search?q=test | Allowed | Allow: / (line 10) |
| /search/help/faq | Allowed | Allow: / (line 10) |
| /data/report.json | Allowed | Allow: / (line 10) |
| /private/notes | Blocked | Disallow: /private/ (line 9) |
| /blog/post-1 | Allowed | Allow: / (line 10) |
robots.txt Tester — Free Online Tool
Tests whether a URL path is allowed for a given crawler by applying the robots.txt matching rules Google uses — the most specific user-agent group wins, the longest matching path pattern wins, Allow beats Disallow on an equal-length tie, and * and $ wildcards are honored — showing the deciding rule and its line number for every test path.
How to use the robots.txt Tester
- Open the tool above — it runs entirely in your browser, so your input never leaves this page.
- Paste or type your input into the field, then press the relevant button (e.g. Encode / Decode, Generate, or Convert).
- Copy the result from the output area with the copy button orCtrl/Cmd + C.
- No signup, no upload, and no tracking — repeat as often as you need.
Frequently asked questions
Why did my Googlebot rules stop the * rules applying?
Because only one group applies. A crawler picks the single most specific matching User-agent group and ignores every other group, including *. Anything you still want blocked has to be repeated inside the specific group.
Which rule wins when Allow and Disallow both match?
The longest matching pattern, counting characters other than the wildcards. On an exact tie, Allow wins. That is why Disallow: /search plus Allow: /search/help leaves the help pages crawlable.
Does blocking a URL in robots.txt remove it from search results?
No — it stops crawling, not indexing. A blocked URL can still appear in results from external links, without a snippet. Use a noindex meta tag or header on a page that must not be indexed, and let it stay crawlable so the directive is seen.
Free & private — why use this robots.txt Tester
The robots.txt Tester runs 100% client-side in your browser. Your data is never uploaded to a server, no account is required, and the tool is completely free. It works offline once the page has loaded and is part of a growing collection of privacy-first developer utilities.