Noindex, canonical or robots.txt: which directive when?

Noindex, canonical and robots.txt are constantly mixed up. All three sound like ways to forbid Google something, so they get deployed more or less at random. The result: pages you wanted to keep out sit happily in the index, and pages you actually wanted to show disappear. Each of the three directives solves a different problem. In this article you get a decision matrix that picks which one to use in every situation, and you will see why the combinations you make intuitively often do the exact opposite of what you expect. If you want to sharpen the basics first, read our pillar on what SEO is.

Three directives, three different questions

The mistake almost always starts at the same point: people assume all three do the same thing, namely “take a page out of Google”. That is not true. Each one works at a different moment in the process Google runs through: first crawling, then indexing, then deciding which version counts.

• Robots.txt says: is Google allowed to crawl this page (or folder), yes or no? It governs access, not visibility.
• Noindex says: Google may crawl this page, but may not include it in the search index. It governs indexation.
• Canonical says: this page is a variant of another one; count the other as the real one. It governs consolidation, not exclusion.

Once you are clear on the question behind each directive, most of the borderline cases resolve themselves. The skill is not knowing the syntax, but knowing which question you are actually asking.

The decision matrix

Work through these questions in order. The first one that returns a “yes” gives you your directive.

1. Do you want Google not to crawl this page, because it is worthless for crawl budget or because it is sensitive? Use robots.txt with a Disallow. Think of internal search results, filter combinations that generate URLs endlessly, or admin paths. Watch out: this does not necessarily keep the page out of the index. If others link to it, Google can still show the URL, just without knowing the content.

2. Do you want the page to be crawled but not to appear in the search results? Use a noindex (<meta name="robots" content="noindex"> or the HTTP header). This is the right choice for thank-you pages, login screens, thin tag archives or a staging environment. Google visits the page, sees the instruction and leaves it out of the index. Crucially: the page must then not be blocked in robots.txt, otherwise Google will never see the noindex.

3. Does the same or nearly the same content exist on multiple URLs, and do you want one version to receive all the signals? Use a canonical (<link rel="canonical" href="...">). Classic example: a product page reachable through several category paths, or URLs with tracking parameters. The canonical bundles link equity and prevents variants from cannibalising each other, while all versions stay reachable for the user.

4. Do you want content that is genuinely identical to disappear entirely, rather than be consolidated? Then a canonical is too soft. Choose a 301 redirect if the old URL can go, or a noindex if the page has to keep existing but stay invisible.

Why the intuitive combinations go wrong

The biggest pitfall is stacking robots.txt and noindex on the same page. It feels logical: twice as certain that the page is gone. In reality they cancel each other out. A noindex is an instruction that lives inside the page. If robots.txt forbids crawling, Google never opens the page and therefore never reads the noindex. Result: the URL can end up in the index anyway, because you made the one instruction that would have kept it out unreachable. If you really want a page out of the index, keep it crawlable and put a noindex on it. Only once it has disappeared from the index may you optionally block it in robots.txt.

A second misconception: using a canonical to hide unwanted pages. A canonical is a hint, not a command. Google may ignore it when the signals contradict it, for example when the two pages differ too much in substance. For true duplicates it works fine, but do not count on it to quietly retire thin or unwanted pages. For that you need a noindex or a redirect.

A third pitfall sits with pagination and faceted navigation in webshops. Many teams slap a canonical from page 2, 3 and onwards onto page 1. That hides your deeper products from the index. With pagination it is usually better to let every page be itself, and to consolidate only for genuine parameter duplicates. The trade-off depends on your structure, and that is exactly where an SEO specialist adds value: not applying the rule, but recognising the exception.

How this fits into your broader indexation strategy

Directives are not isolated tricks. They are the fine-tuning on top of a healthy foundation. Before you start blocking or consolidating, you want to know which pages have any value at all. Noindexing a page that could have produced leads is just as damaging as letting a worthless page get indexed. That is why every directive choice starts with a judgement about substance: what does this page do for your pipeline?

A practical order to run through per page:

1. Does the page have search value? If not, consider a noindex or even removal. If so, keep it fully indexable.
2. Is there a variant fragmenting link equity? Consolidate with a canonical or a redirect.
3. Is crawling wasting budget on infinite URL spaces? Close those patterns off in robots.txt, but never on pages where a noindex still has to do its work.
4. Check in Search Console whether Google is actually following your instructions. The index reports show which pages are excluded and why.

That last step is the one most often skipped. Setting a directive is no guarantee that Google reads or respects it; you verify it in practice. How a page ends up in the index in the first place is covered in our article on website indexing. If you want to steer the crawl process itself more effectively, our explanation on setting up an XML sitemap will take you further.

Our view: directives are pipeline choices, not technical tricks

At Customer Impact we do not treat crawl and indexation directives as a separate technical to-do, but as part of one growth engine with one yardstick: qualified pipeline. Every page you let index or deliberately keep out is a choice about where your authority and attention go. The same principle applies to AI search engines. ChatGPT, Google AI and Perplexity cite sources based on clarity and structure. If your index is a mess of duplicates and thin pages, you weaken not only your Google ranking but also your chance of being cited. Clean directives make you findable and quotable, and those are two layers of the same movement.

The mistake we see most often is that teams deploy directives reactively, page by page, without a picture of which content actually matters. As a result they sometimes block their best pages and leave their weakest ones standing. The matrix above only really works when it sits on top of a clear prioritisation of substance.

Get started

Not sure which directive belongs on which page, or do you suspect that a wrong combination is holding back your visibility? Tell us your situation and your goal. Schedule your free intake and within 24 hours you will hear where your biggest opportunities are, in both Google and AI answers.