What Is the X-Robots-Tag? | Customer Impact

The X-Robots-Tag is an indexing instruction that you do not put in the HTML of a page, but pass along through the HTTP header of the server response. It lets you tell search engines not to index a file, even when that file is not a web page. In this article you will read what the X-Robots-Tag is exactly, how it differs from the regular meta robots tag and when you need it.

What is the X-Robots-Tag exactly?

Every time your server delivers a file, it sends a small set of invisible data along with it: the HTTP header. The X-Robots-Tag is a line you add to that header, for example X-Robots-Tag: noindex. The visitor sees nothing of it, but a crawler does read the instruction.

The power sits in that word “header”. A regular meta robots tag lives inside the HTML of a page. But what if your file has no HTML, such as a PDF, an image or a video file? Then there is no <head> to place a tag in. The X-Robots-Tag solves that: it sits at header level and therefore works for every type of file your server delivers.

Google supports exactly the same values for the X-Robots-Tag as for the meta robots tag. Google confirms this in its documentation on robots meta tags and the X-Robots-Tag.

Which values does the X-Robots-Tag support?

Because the values match those of the meta robots tag, you will recognise them right away:

noindex. Do not include this file in the search results.
nofollow. Do not follow the links in this file.
noarchive, nosnippet. Show no cached copy or text snippet.
max-snippet, max-image-preview, max-video-preview. Limit the size of the preview shown.

You can combine several values in a comma-separated list. With conflicting rules, according to Google the most restrictive one wins. If max-snippet:50 and nosnippet appear together, for example, then nosnippet applies.

When do you use the X-Robots-Tag?

For regular web pages the meta robots tag is perfectly sufficient. The X-Robots-Tag comes into play when a tag in the HTML is not an option:

PDF files you do not want in the search results, such as quotes, whitepapers behind a form or internal documents.
Images and video files you want to keep out of the image results.
Large groups of files at once, because you set the rule centrally in your server configuration instead of file by file.

You set the X-Robots-Tag through the configuration of your web server, for example in an .htaccess file with Apache or in the server blocks with Nginx. That makes it powerful at scale, but also more technical than a single meta tag.

The pitfall: do not block the file in robots.txt

Just as with the meta robots tag, a crucial rule applies here. The X-Robots-Tag is only read when the crawler fetches the file. If you block that same file in your robots.txt, Google will never visit it, will never see the noindex, and the file can still end up in the index.

If you really want a file out of the search results, keep it crawlable and put an X-Robots-Tag with noindex on it. Robots.txt, noindex and canonical look alike but each solves a different problem. Which one to use when, and why the intuitive combinations go wrong, we explain in noindex, canonical or robots.txt.

What does this mean for B2B?

For a B2B company the X-Robots-Tag is rarely your biggest worry, but it is handy in a few concrete situations. Think of a whitepaper or price list you only want to offer behind a form: you do not want the bare PDF to simply show up in Google and bypass your lead form. That is where the X-Robots-Tag is the right instrument.

Our honest view: do not overdo it. For most sites you arrange findability with strong pages and clean indexing, and you only need the X-Robots-Tag for those few non-HTML files you deliberately want to shield. If you want to get your entire indexing strategy in order as part of your SEO approach, we will look at which files and pages strengthen your authority and which are only noise.

Frequently asked questions

What is the difference between the X-Robots-Tag and the meta robots tag? The meta robots tag sits in the HTML and only works for web pages. The X-Robots-Tag sits in the HTTP header and also works for files without HTML, such as PDFs and images. The supported values are identical.

Can I keep a PDF out of Google with the X-Robots-Tag? Yes. That is one of its best-known applications. Put X-Robots-Tag: noindex on the PDF file through your server configuration and make sure the file is not blocked in robots.txt.

Where do I set the X-Robots-Tag? In the configuration of your web server, for example an .htaccess file with Apache or the server blocks with Nginx. So it is a more technical intervention than a meta tag in your HTML.

Does the X-Robots-Tag work if the file is blocked in robots.txt? No. Google has to be able to crawl the file in order to read the header. If it is blocked, the instruction is ignored and the file can still get indexed.

Unsure about your indexing?

Are files showing up in Google that do not belong there, or do you want to be certain that your shielded documents really are shielded? Tell us about your situation.

We are a small team that moves fast and tells you honestly where your biggest opportunities lie, in Google and in AI answers. Book your free intake.