robots.txt Validator

Paste your robots.txt content to check its syntax. Parses User-agent, Disallow, Allow, and Sitemap directives and visualizes crawler access rules.

Last updated: 2026-09-13

[[ labels.url_label ]]

[[ fetchError ]]

[[ labels.or_paste ]]

[[ labels.input_label ]]

[[ labels.sample_label ]]:

[[ labels.no_input ]]

[[ labels.result_label ]]

[[ labels.errors_label ]]

[[ labels.line_label ]] [[ err.line ]]: [[ errorMessage(err) ]]

✓ [[ labels.no_errors ]]

[[ labels.blocks_label ]]

User-agent: [[ ua === '*' ? labels.all_crawlers + ' (*)' : ua ]]

Disallow
Allow	[[ path ]]
[[ labels.no_rules ]]
Crawl-delay	[[ block.crawlDelay ]]

[[ labels.sitemaps_label ]]

[[ url ]]

What Is the robots.txt Validator?

The robots.txt validator lets you check a robots.txt file for syntax errors and subtle mistakes simply by pasting its content or fetching it directly from a live URL. It parses every directive — User-agent, Disallow, Allow, Sitemap, and Crawl-delay — and lays out, in a single readable view, exactly which paths each crawler is allowed or blocked from accessing.

A hand-written robots.txt file is deceptively easy to get wrong: directives listed in the wrong order, an Allow rule with no matching Disallow, or a block placed under the wrong User-agent group can all pass a casual read-through while quietly telling search engines to stay away from your entire site. Running the file through this checker before you publish it catches those mistakes while they still cost nothing to fix.

How to Use the robots.txt Validator

Prepare the robots.txt you want to check Have the file's contents copied and ready, or simply know the URL of the live site whose robots.txt you want to inspect.
Fetch it from a URL, or paste it directly Enter the site's address in the "Fetch from URL" field to retrieve it automatically, or paste the raw text straight into the textarea below.
Review the errors and warnings in the results Unknown directives, directives that appear before any User-agent line, and other mistakes are listed with the exact line number where they occur.
Check the allowed and blocked paths per crawler The results organize every rule by crawler, so you can quickly compare what is actually blocked against what you intended.
Confirm the Sitemap directive is present and correct Verify that your sitemap's URL is spelled correctly and included, since search engines rely on this line to discover it.

Tips for getting more out of it

User-agent: * applies to all crawlers. Blocks with a specific bot name (e.g. Googlebot) take precedence over the wildcard block.
Disallow: with an empty value means "allow all paths". Disallow: / (slash only) blocks everything. Do not confuse the two.
The Allow: directive takes precedence over Disallow: when both match. For example, Disallow: /private/ followed by Allow: /private/public.html allows that single file.
Crawl-delay: is a non-standard directive specifying the delay in seconds between requests. Bing and Yandex respect it, but Google does not — use Google Search Console's crawl rate settings instead.
The Sitemap: directive tells search engines the URL of your sitemap. Both Google and Bing recognize it. Combine it with manual submission via Google Search Console or Bing Webmaster Tools for best results.

Use Cases

Final check before a site relaunch

Paste your production robots.txt before going live to make sure a leftover staging rule that blocks the entire site was not accidentally carried over.

Auditing the default file from a new CMS or framework

Many platforms auto-generate a robots.txt on install. Check it here to confirm it is not blocking directories you actually want indexed.

A quick technical check during an SEO audit

When a site's rankings are underperforming, this tool lets you confirm within seconds whether crawl access is part of the problem.

Inspecting a competitor's robots.txt

Fetch a competitor's site by URL to see which directories or file types they deliberately keep out of search engine crawls.

Glossary

User-agent: The directive that names which crawler a rule group applies to. * matches every crawler that does not have a more specific rule of its own.
Disallow: The directive that blocks crawling of a given path. An empty value means "allow everything," which is easy to confuse with a single slash meaning "block everything."
Allow: The directive used to carve out an exception within a path that a Disallow rule would otherwise block, permitting just that specific file or subpath.
Sitemap directive: Tells search engines where to find your sitemap XML file. It is read independently of any Disallow rules and is always processed.
Crawl-delay: A non-standard directive specifying the minimum interval, in seconds, between requests. Bing and Yandex honor it, but Google ignores it entirely.
Robots Exclusion Protocol (REP): The underlying convention that robots.txt implements, devised in 1994. It remained an informal standard for nearly three decades until it was formally published as RFC 9309 in 2022.

FAQ

Yes. Blocking a page with Disallow prevents crawlers from reading its content, so it will not be indexed. However, if the page is linked from other sites, search engines may still list it as "known but not crawled." For full exclusion, combine robots.txt with a noindex meta tag.

No. robots.txt is scoped per origin. example.com/robots.txt does not apply to blog.example.com. Each subdomain needs its own robots.txt file.

Googlebot supports * (any sequence of characters) and $ (end of URL). For example, Disallow: /private/*.pdf$ blocks all PDF files under /private/. Not all crawlers support wildcards, so check each bot's documentation.

Side Note — The Birth of robots.txt: The Day the Internet Got Manners

In 1994, Dutch engineer Martijn Koster invented the Robots Exclusion Protocol (REP). At the time, web crawlers were roaming sites uncontrolled, placing excessive load on servers. Koster's idea was simple: "If you place a file at the site root with rules, crawlers will read it." That file became robots.txt.

Crucially, robots.txt is a voluntary gentlemen's agreement — it has no enforcement power. Well-behaved crawlers like Googlebot and Bingbot respect it, but malicious bots ignore it. This means robots.txt works well for telling search engines what not to index, but it should never be relied on to protect sensitive content from unauthorized access. Confidential pages must be protected with authentication.

The Robots Exclusion Protocol existed as an informal convention for nearly three decades before being formally registered as RFC 9309 by the IETF in September 2022 — almost 28 years after its creation. The RFC clarified that directive names are case-insensitive and resolved various long-standing syntactic ambiguities.

Browse all trivia