robots.txt Tester
Fetch a site's robots.txt, see its parsed rules, and check whether a specific path is allowed for Googlebot. Free, no account needed.
3 free checks per day. No signup needed.
How the path check works
If you enter a path to test, this applies Googlebot's own precedence rule: the longest matching Allow or Disallow pattern wins, and an Allow beats a Disallow of the exact same length. Other crawlers can interpret ties differently, so this reflects Googlebot's behavior specifically, not a universal answer.
Frequently asked
What is robots.txt for?
A plain-text file at a site's root that tells well-behaved crawlers which paths they should and shouldn't request. It's a request, not an enforced restriction.
Whose interpretation of the rules does the path check use?
Googlebot's: the longest matching pattern wins, and Allow beats Disallow on an exact-length tie. Other crawlers can interpret ties and wildcards differently.
What if a site has no robots.txt at all?
No file means no restrictions, crawlers are free to request anything on the site. It's the same as a robots.txt with no rules in it.
Related tools: Sitemap Link Extractor ·DNS Lookup