What Is the Meta Robots Tag?

The meta robots tag is a piece of HTML that tells search engines how they may handle your page: may they include it in the search results, and may they follow the links on it? It is your fine-tuning control for visibility, page by page. In this article you will learn what the meta robots tag is exactly, which values exist and which common mistake stops your instruction from working at all.

What is the meta robots tag exactly?

The meta robots tag sits in the <head> of a page and looks like this: <meta name="robots" content="noindex, nofollow">. It does not address your visitor, but the crawlers of search engines such as Googlebot. The tag gives instructions at page level: it says nothing about your entire site, only about the single page it appears on.

If you want to give an instruction to all search engines, you use name="robots". If you only want to address Google, name="googlebot" also exists. The supported values are the same. Google describes them in full in its robots meta tag specifications.

You will also find a short definition of this term in our wiki entry on the meta robots tag.

The most important values

The tag has several values, which you can combine with commas. These are the most important ones:

index / noindex. May the page appear in the search results? noindex keeps it out. index is the default and does not need to be set explicitly.
follow / nofollow. May the search engine follow the links on this page and pass on link value? nofollow switches that off.
noarchive. Google may not show a saved copy (cache) of the page.
nosnippet. Show no text fragment or preview in the results.
max-snippet, max-image-preview, max-video-preview. These let you limit how large the displayed fragment may be.

By far the most used are noindex and nofollow. The rest is fine-tuning for specific cases. If you set nothing, the search engine assumes everything is allowed: indexing and following links.

The mistake that makes your instruction useless

This is the biggest pitfall, and we see it often. A meta robots tag with noindex only works if the search engine is allowed to crawl the page. After all, the instruction sits inside the page itself. If you also block that same page in your robots.txt file, Google never opens the page, never reads the noindex, and the URL can still end up in the index.

In other words: stacking robots.txt and noindex on the same page is counterproductive. If you really want a page out of the search results, keep it crawlable and put a noindex on it. Only once it has disappeared from the index may you optionally block it in robots.txt.

Robots.txt, noindex and the canonical tag all three look alike but do something completely different. We explain which to use when, with a decision matrix, in noindex, canonical or robots.txt. That article goes one step deeper than this definition.

When do you use noindex?

Not every page belongs in Google. Good candidates for noindex:

Thank-you and confirmation pages after a form.
Login and account screens.
Internal search results and thin filter combinations.
Staging or test environments that are accidentally public.

The flip side: never use noindex lightly on pages that can generate leads. Accidentally setting a commercial page to noindex is a classic way to make yourself unfindable. So check in Google Search Console whether your instructions really do what you intend. To learn how pages end up in the index, read website indexing.

What does this mean for B2B?

For a B2B company, findability revolves around a handful of strong pages that convince your decision-makers. The meta robots tag is not a growth lever then, but it is a hygiene instrument: you use it to keep noise out and to concentrate your authority on the pages that really matter.

Our honest view: do not start with tags, but with the question of which pages have value for your pipeline. Noindexing a page that could generate leads is just as damaging as letting a worthless page be indexed. Making that trade-off is exactly where a well-considered SEO approach adds value: not blindly applying the rule, but recognising the exception. That way you keep your on-page SEO clean and focused.

Frequently asked questions

What is the difference between noindex and nofollow? noindex keeps the page out of the search results. nofollow says the search engine should not follow the links on the page. You can use them separately or together.

Why is my page still in Google despite a noindex? Usually because the page is also blocked in robots.txt. Google then never reads the noindex. Remove the block so the page becomes crawlable, and the noindex can do its work.

Do I need the meta robots tag if I do not want to block anything? No. Without a tag, the search engine assumes by default that the page may be indexed and the links followed. You only add the tag if you want to deviate from that behaviour.

What is the difference with the X-Robots-Tag? The meta robots tag sits in the HTML and therefore only works for web pages. The X-Robots-Tag gives the same instructions via the HTTP header and also works for files without HTML, such as PDFs and images.

Unsure about your indexation?

Do you suspect that wrong instructions are getting in the way of your findability, or are there pages in Google that do not belong there? Tell us about your situation.

We are a small team that moves fast and tells you honestly where your biggest opportunities lie, in Google and in AI answers. Book your free intake.

What Is the Meta Robots Tag? Noindex and Nofollow Explained