Customer Impact

Website & Development

XML sitemap and indexing for a new website: help Google understand your site

Copy for AI

An XML sitemap is a file that lists the pages you want Google to find.

You create one, place it on your domain and submit it in Google Search Console. For a new website, that step matters more than most people think.

The verdict: a clean XML sitemap is the fastest way to get a new site discovered. It’s a hint to Google, not an order, so the contents decide whether it helps.

A fresh domain has no external links pointing to it yet. Without those, your sitemap is often the first route Google takes to your pages.

New to how search engines work? Start with what technical SEO covers.

What is an XML sitemap and why do you need one?

An XML sitemap is a structured file that lists the most important URLs on your site for search engines.

It tells Google: these are the pages that matter, feel free to come and fetch them.

A sitemap is a hint, not an order

Google’s sitemap documentation calls submitting one “merely a hint”.

Google uses it to discover your pages faster. It still decides for itself whether to crawl and include them.

So a sitemap never forces indexing. You can read how that process works in our article on how to get a website indexed.

Why new sites benefit most

For a new site, that discovery advantage is very real.

An established site with plenty of inbound links gets found on its own. A fresh domain doesn’t have those signals yet.

That’s why a sitemap you actively submit is one of your strongest tools for becoming findable. Our sitemap glossary entry covers the other sitemap types, such as image and video sitemaps.

Your CMS usually builds it for you

Most platforms generate the sitemap automatically:

  • Webflow creates one by default.
  • WordPress does it through plugins such as Yoast, Rank Math or All in One SEO.
  • Custom builds generate it from the page list at build time.

So you rarely write the file by hand. Comparing platforms? See WordPress vs Webflow.

The craft is in the contents: which pages do you leave in, and which do you leave out?

What does a good XML sitemap file look like?

It’s a short, UTF-8 encoded list of absolute URLs, one <url> entry per page.

Here’s a minimal example with two pages:

<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <url>
    <loc>https://www.example.com/</loc>
    <lastmod>2026-11-04</lastmod>
  </url>
  <url>
    <loc>https://www.example.com/services/consulting/</loc>
    <lastmod>2026-10-20</lastmod>
  </url>
</urlset>

The four rules Google states

Google’s build-a-sitemap guide sets the basics:

  1. Encoding: the file must be UTF-8.
  2. URLs: use fully qualified, absolute URLs, not /mypage.html.
  3. Location: the site root is where Google recommends you post it.
  4. Size: at most 50,000 URLs or 50 MB uncompressed per file.

Which tags actually count

Only two tags do real work: <loc> and <lastmod>.

Google says it “ignores <priority> and <changefreq> values”. Setting every page to priority 1.0 changes nothing.

Watch out: <lastmod> only helps when it’s consistently accurate. Google uses it if it’s “consistently and verifiably” right.

So don’t let your CMS bump every date on each deploy. Update it when the page content really changed.

Which pages belong in your sitemap and which do not?

Only indexable, valuable end pages that you genuinely want to see in the search results.

Here’s that filter as one picture:

SITEMAP FILTER Does this page go in your sitemap? Indexable page that should rank? services, landing pages, articles, cases Include it with an honest lastmod Needed, but shouldn’t rank? thank-you, login, thin tag pages Noindex, leave out keep it crawlable Redirect, error or duplicate? filters, internal search, staging copies Leave it out list the final URL only A sitemap lists the pages you want found, not everything your CMS produces.

Do include

These pages earn a spot:

  • Your homepage, service pages and important landing pages.
  • Knowledge articles, blog posts and cases that bring in visitors and leads.
  • Contact and about pages that help build trust.

Do not include

These stay out:

  • Thank-you and confirmation pages after a form submission.
  • Login, account and checkout-style pages.
  • Internal search results and filtered overviews that generate endless variations.
  • Test, staging or duplicate pages.
  • Pages with a noindex tag, or pages that return a redirect or an error.

Why a bloated sitemap hurts

A common mistake is a sitemap that dumps everything the CMS produces.

That dilutes the signal. You point Google to pages that deliver nothing, instead of clearly to the ones that count. On large sites that also costs crawl budget.

For example, say a B2B site has 40 service and article pages. A filter plugin adds 600 parameter URLs to the sitemap. Now 9 in 10 listed URLs are noise.

Keep the list clean and focused on pages that can turn visitors into leads.

Duplicates: list only the version you set as canonical. Our guide to the canonical tag explains how to pick it.

Let your structure do the work

Which pages belong ties in closely with your website architecture for SEO.

A logical structure with clear internal links makes your sitemap tidy almost by itself.

When you need more than one file

Each sitemap file is capped at 50,000 URLs or 50 MB uncompressed, according to Google.

For a B2B site you rarely hit those limits. If you do, split the file into several sitemaps and reference them from a sitemap index file.

How do you use robots.txt and noindex without mistakes?

Use robots.txt to control crawling and noindex to control the index, and never swap them.

They look alike, but they do something fundamentally different. That’s exactly where things tend to go wrong.

robots.txt disallownoindex tag
Controlswhether a bot may crawl a pagewhether a page may appear in the index
Google reads the page?NoYes, it must
Keeps a page out of results?Not reliablyYes
Use it forcrawl waste, private areas behind a loginthank-you pages, thin tag overviews

More on the file itself in our robots.txt glossary entry.

The trap: blocking a noindex page

If you block a page in robots.txt, Google can’t visit it. So it can’t read the noindex tag on it either.

Google’s noindex documentation says it plainly: if robots.txt blocks the URL, “the crawler will never see the noindex rule”.

The consequence: the page can still surface in the results, and you can’t get it out.

Rule of thumb: to keep a page out of Google, put a noindex tag on it and deliberately leave it crawlable.

Four guidelines for a new site

Follow these during the build:

  1. Shield staging. Put the entire staging environment behind a password or HTTP authentication.
  2. Remove blocks at launch. Check that no site-wide noindex or blocking robots.txt survives go-live.
  3. Use noindex deliberately for pages you need but that don’t belong in Google.
  4. Point robots.txt at your sitemap with a Sitemap: line.

On the first point: don’t rely on robots.txt alone to keep a site hidden. Google’s robots.txt introduction says it “is not a mechanism for keeping a web page out of Google”.

On the second: this is the classic disaster. Your new site looks great, but shows up nowhere. It’s also a risk during a move, so read up on how to migrate a website without losing rankings.

What the sitemap line looks like

Google says you can insert the Sitemap: line “anywhere in your robots.txt file”, with the full URL of your sitemap.

Here’s our own robots.txt, with one rule for all crawlers and the sitemap line at the bottom:

The robots.txt file of customerimpact.be with one rule that allows every crawler plus the Sitemap line pointing to the XML sitemap Our own robots.txt for customerimpact.be, October 2026: everything crawlable, plus a Sitemap line so search engines find the file.

Short and open is the safe default for a business site.

Keep sitemap and noindex consistent

A page you set to noindex but leave in your sitemap sends contradictory signals.

So run one check: no URL in your sitemap may carry a noindex tag.

How do you submit your sitemap in Google Search Console?

Verify your site in Google Search Console, then add your sitemap URL in the Sitemaps report.

It’s a free step, and for a new site one of the first things you do. Our Google Search Console guide covers the setup in full.

The four steps

Broadly speaking:

  1. Create a free Google Search Console account and add your domain as a property.
  2. Verify that you’re the owner, usually through a DNS record or a verification file. Many CMSes have a built-in option for this.
  3. Go to the Sitemaps report and enter the path to your sitemap, usually something like sitemap.xml.
  4. Check back after a while to see whether Google could read the sitemap and how many URLs it discovered.

Here’s the same flow as a diagram:

SUBMITTING YOUR SITEMAP From property to inclusion in Search Console STEP 1 Add your property Create the domain in GSC STEP 2 Verify ownership DNS record or file STEP 3 Submit your sitemap Path to sitemap.xml STEP 4 Track inclusion Which URLs got picked up Submitting speeds up discovery, but does not force indexing
The four steps to correctly submit your sitemap to Google Search Console.

What the Sitemaps report shows

After submitting, the report shows a status per sitemap and the number of discovered URLs:

“Success” only means Google could read the file. Indexing is a separate verdict, per URL.

Track what Google crawls next

You can keep tracking in Search Console which pages get included, which don’t, and why.

The Crawl Stats report shows whether Googlebot is actually fetching your pages after launch:

Google Search Console Crawl Stats report for customerimpact.be with total crawl requests, download size, average response time and the breakdown by response Crawl Stats in our own Search Console for customerimpact.be, October 2026. Settings > Crawl stats shows requests per response type.

A rising line of crawl requests after you submit is a good first sign.

Request indexing for one page

Want a specific new page picked up faster? Request indexing through the URL inspection tool.

Do bear in mind that Google makes the final call. Submitting speeds up discovery, but forces nothing.

When do you sort this out during the build of a new site?

During the build, not as aftercare once you’ve launched.

Your sitemap, robots.txt and indexing choices aren’t a technical side issue. They decide whether your investment in a new site becomes findable at all.

During the build

Three things to do before launch:

  • Keep the staging environment shielded.
  • Plan which pages may be indexed.
  • Put every temporary block on a checklist to be removed at go-live.

On launch day

Four checks, in this order:

  1. The site-wide noindex is gone.
  2. Robots.txt allows crawling.
  3. Your sitemap contains the right pages, and only those.
  4. You’ve submitted it in Search Console.

Those few checks prevent the most painful mistake in web development: a beautiful new site that stays invisible for weeks.

Tip: crawl your live sitemap once with a crawler and compare it with your page list. Our SEO audit guide shows that routine.

Part of the build, not an extra

At Customer Impact we build conversion-focused B2B sites. We see this configuration as part of solid craftsmanship, not as a separate extra.

A site that has to deliver leads must first be found correctly.

You can read how we approach that technical foundation on our website development page.

The broader context is in our guide to building a B2B website.

The short summary

Creating an XML sitemap isn’t complicated, but the details decide the result.

Only list valuable, indexable end pages. Use robots.txt for crawling and noindex for visibility, without mixing them up. Then submit your sitemap through Search Console.

Most important: sort this out during the build and remove every temporary block at go-live.

Launching soon? Send us your staging URL and we’ll check sitemap, robots.txt and noindex before go-live.

Free website scan

Enter your website and get an automatic scan within minutes, with concrete technical and SEO improvements. No sales pitch.

Where should we send your report?

We only use your details for your scan. No spam, unsubscribe anytime.