Robots Meta Tag / X-Robots-Tag Validator
Paste the value of a tag or an X-Robots-Tag header to validate its directives in one pass. Detects conflicting instructions like combining noindex with index, and malformed values such as an invalid max-snippet, with a full directive breakdown.
Inspecting your crawling and indexing instructions
`` and `X-Robots-Tag` are how you tell a search engine not to index a page or not to follow its links. Paste the value in and this tool breaks the directives apart, surfaces contradictory settings and faulty values, and lays out the result.
**The first thing to grasp is that where instructions contradict one another, the more restrictive one prevails.** Write `index, noindex` and `noindex` wins. **The commonest accident by far, though, is placing `noindex` on a page while also disallowing it in robots.txt.** The crawler can then no longer fetch the page, so **it never reads the `noindex` instruction at all and the page stays in the index.** A page you want removed must be left crawlable so that the `noindex` can be read. Note too that `X-Robots-Tag`, being an HTTP header, lets you give the same instructions for PDFs and images, which have no HTML to carry a meta tag.
How to use it
- Paste in the value A `` tag, a line beginning `X-Robots-Tag:` or the bare directives are all accepted.
- Try the samples first A good example and a bad one are provided.
- Read the contradictions and errors **Writing `noindex` alongside `index`, and faulty values in `max-snippet` and the like, are both called out.**
- Review the breakdown You can see the directives taken apart and establish which ones are in force.
Tips for getting more out of it
- Avoid combining noindex with a robots.txt disallow for the same page. If disallow blocks crawling entirely, the crawler never gets to read the noindex tag on the page, so removal from the index may not proceed as intended.
- Google does not error out on directive names it does not recognize — it simply ignores them. That means an unknown directive is not fatal, but it also means a typo can go unnoticed unless you check for it.
- max-snippet:-1 and max-video-preview:-1 are special values meaning "unlimited." A value of 0 means something different — "show no snippet at all" — so do not confuse the two.
- The X-Robots-Tag header is the only way to apply directives like noindex to non-HTML files such as PDFs and images; a tag can only be embedded in an HTML document.
- When combining multiple directives, list them comma-separated in a single content attribute or header value. Writing several separate tags does not make the last one win — each is evaluated independently, which can produce surprising results.
Where it comes in useful
Preventing a staging site from going public
**You can confirm before launch that `noindex` is genuinely in effect.**
Investigating why a page will not leave the index
**It gives you the prompt to check whether a robots.txt disallow is being used alongside `noindex`.**
Adjusting how a result appears
You can confirm that `max-snippet` and `max-image-preview` are correctly specified.
Deciding how to treat PDFs and the like
Files with no HTML require `X-Robots-Tag` instead of a meta tag.
Terms in robots instructions
- Noindex
- The instruction asking that a page be kept out of search results. **Disallowing it in robots.txt means it is never read, so the two must not be combined.**
- Nofollow
- The instruction asking that the links on the page not be followed.
- Noarchive
- The instruction asking that no cached copy be offered in search results.
- Max-snippet
- Sets the maximum length of the description shown in results. `0` suppresses it and `-1` lifts the limit.
- Max-image-preview
- Sets the size of the image preview, choosing among `none`, `standard` and `large`.
- X-Robots-Tag
- The same instructions **returned as an HTTP header.** It serves for resources such as PDFs and images that carry no HTML.
FAQ
Side Note — robots.txt vs. the robots meta tag: a division of labor between crawling and indexing
robots.txt and the robots meta tag are both instructions for search engines, but they operate at different stages. robots.txt is a static text file placed at the site root that controls whether a URL pattern may be crawled at all. The robots meta tag (and the X-Robots-Tag header) instead controls, on a per-page basis, whether an already-crawled page may be indexed or whether its links may be followed — the two operate as independent layers.
Because of this division of labor, adding a noindex tag to a page that robots.txt already disallows has no effect. The crawler can never fetch that page in the first place, so it never gets a chance to read the noindex instruction embedded in it. If you want a page removed from search results, you need to keep crawling allowed and rely on noindex alone.
The X-Robots-Tag header was introduced by Google around 2007, extending the same kind of control to non-HTML resources like PDFs, images, and videos where a tag cannot be embedded. Because it can be set in bulk per file extension through web server configuration (e.g. Apache), it is well suited to tasks like noindexing an entire folder of PDF files at once.
Value-bearing directives such as max-snippet, max-image-preview, and max-video-preview are relatively recent additions, introduced in 2019 when Google clarified its policy around search-result snippet display. Before that, publishers who wanted to limit snippet display had only the blunt option of nosnippet, which turned it off entirely — the newer directives gave them much finer-grained control.