Robots.txt
A text file in the root of your site that tells crawlers which parts they may or may not visit.
By Tanguy De Keyzer · Founder & digital strategist
Robots.txt is a simple text file in the root of your domain that instructs search engine crawlers which parts of your site they may or may not visit.
What does robots.txt do?
When a crawler visits your site, it first reads yoursite.be/robots.txt to see which rules apply. With Disallow you keep bots away from folders or pages, for example internal search results, admin environments or duplicate filter pages. This protects your crawl budget and prevents Google from wasting time on unimportant URLs. You can also mention your sitemap in it, so crawlers find your important pages faster.
An important misunderstanding
Robots.txt blocks crawling, not indexing. A page blocked via robots.txt can still appear in Google if links point to it, only without Google seeing the content. If you really want to keep a page out of the search results, use a meta robots tag with noindex on a page that is crawlable. Mixing the two up is one of the most common mistakes and can easily sabotage your indexing.
Robots.txt according to Customer Impact
A wrong little rule in robots.txt can make an entire site disappear from Google, so we treat this file with care. For B2B clients we use it in a targeted way to guide crawlers towards the pages that generate leads and revenue and away from noise that contributes nothing. No vanity, just a clean, considered foundation on which the rest of your SEO can build.
See also
From theory to growth.
We turn Robots.txt into measurable results for your business.