Robots Meta Tag / X-Robots-Tag Validator

Paste the value of a tag or an X-Robots-Tag header to validate its directives in one pass. Detects conflicting instructions like combining noindex with index, and malformed values such as an invalid max-snippet, with a full directive breakdown.

Tips

Avoid combining noindex with a robots.txt disallow for the same page. If disallow blocks crawling entirely, the crawler never gets to read the noindex tag on the page, so removal from the index may not proceed as intended.
Google does not error out on directive names it does not recognize — it simply ignores them. That means an unknown directive is not fatal, but it also means a typo can go unnoticed unless you check for it.
max-snippet:-1 and max-video-preview:-1 are special values meaning "unlimited." A value of 0 means something different — "show no snippet at all" — so do not confuse the two.
The X-Robots-Tag header is the only way to apply directives like noindex to non-HTML files such as PDFs and images; a tag can only be embedded in an HTML document.
When combining multiple directives, list them comma-separated in a single content attribute or header value. Writing several separate tags does not make the last one win — each is evaluated independently, which can produce surprising results.

FAQ

robots.txt controls, at the site or URL-pattern level, whether a crawler is allowed to fetch a given URL at all. The robots meta tag (including the X-Robots-Tag header) instead controls, per page, whether an already-crawled page may be indexed or whether its links may be followed. In short, one governs crawling, the other governs indexing and link-following — two separate stages.

noindex tells Google not to include that page in search results. nofollow tells it not to follow the links found on that page (i.e. not to use them as signals for crawling their targets). Combined, they form a strict setting: the page itself stays out of search results, and its outbound links carry no crawling signal either.

A tag can't be used since it only works inside an HTML document, but X-Robots-Tag is sent as an HTTP response header, so it can be applied to any file type — PDFs, images, videos, and more. A common approach is to configure your web server to attach it in bulk based on file extension.

It lets you cap the number of characters Google may show in the search-result snippet for a page. That's useful when you want to show only the opening lines of a news article, or prevent the full body of paywalled content from appearing, without going as far as nosnippet, which turns snippet display off entirely.

You can set both, but they won't work as most people expect. Once robots.txt disallows a page, the crawler can never fetch it, so it never reads the noindex tag written on it — meaning the page may not actually get removed from the index. If your goal is to keep a page out of search results, the correct approach is to allow crawling and rely on noindex alone.

Side Note — robots.txt vs. the robots meta tag: a division of labor between crawling and indexing

robots.txt and the robots meta tag are both instructions for search engines, but they operate at different stages. robots.txt is a static text file placed at the site root that controls whether a URL pattern may be crawled at all. The robots meta tag (and the X-Robots-Tag header) instead controls, on a per-page basis, whether an already-crawled page may be indexed or whether its links may be followed — the two operate as independent layers.

Because of this division of labor, adding a noindex tag to a page that robots.txt already disallows has no effect. The crawler can never fetch that page in the first place, so it never gets a chance to read the noindex instruction embedded in it. If you want a page removed from search results, you need to keep crawling allowed and rely on noindex alone.

The X-Robots-Tag header was introduced by Google around 2007, extending the same kind of control to non-HTML resources like PDFs, images, and videos where a tag cannot be embedded. Because it can be set in bulk per file extension through web server configuration (e.g. Apache), it is well suited to tasks like noindexing an entire folder of PDF files at once.

Value-bearing directives such as max-snippet, max-image-preview, and max-video-preview are relatively recent additions, introduced in 2019 when Google clarified its policy around search-result snippet display. Before that, publishers who wanted to limit snippet display had only the blunt option of nosnippet, which turned it off entirely — the newer directives gave them much finer-grained control.

Browse all trivia

Robots Meta Tag / X-Robots-Tag Validator

Tips

FAQ

What's the difference between robots.txt and the robots meta tag?

What does "noindex, nofollow" actually do?

Can I use X-Robots-Tag for images or PDFs?

What's the benefit of setting max-snippet?

Is it a problem to set both noindex and a robots.txt disallow?

Side Note — robots.txt vs. the robots meta tag: a division of labor between crawling and indexing