XML sitemap and indexing for a new website: help Google understand your site

You create an XML sitemap by listing the pages you want Google to pick up, placing that file on your domain and submitting it through Google Search Console. For a new website, that matters more than most people think: without external links pointing to you, a sitemap is often the fastest way for Google to discover your pages. In this article you will learn which pages belong in your sitemap and which do not, how to configure robots.txt and noindex correctly during the build, and how to submit everything properly.

What is an XML sitemap and why do you need one?

An XML sitemap is a structured file that lists the most important URLs on your site for search engines. It tells Google: these are the pages that matter, feel free to come and fetch them.

One thing is essential to understand: a sitemap is a hint, not an order. Google uses it to discover your pages faster, but still decides for itself whether to crawl and include them. So a sitemap never forces indexing, and you can read exactly how that indexing process works in our article on how to get a website indexed.

For a new site, that discovery advantage is very real. An established site with plenty of inbound links gets found on its own. A fresh domain does not have those signals yet, so a sitemap you actively submit is one of your strongest tools for becoming findable.

Most platforms generate that sitemap automatically. Webflow creates one by default, and in WordPress plugins such as Yoast, Rank Math or All in One SEO handle it for you. So you rarely have to write the file by hand. The craft is in the contents: which pages do you leave in, and which do you leave out?

Which pages belong in your sitemap and which do not?

The rule is simple: your sitemap should only contain indexable, valuable end pages that you genuinely want to see in the search results.

Do include:

• Your homepage, service pages and important landing pages.
• Knowledge articles, blog posts and cases that bring in visitors and leads.
• Contact and about pages that help build trust.

Do not include:

• Thank-you and confirmation pages after a form submission.
• Login, account and checkout-style pages.
• Internal search results and filtered overviews that generate endless variations.
• Test, staging or duplicate pages.
• Pages with a noindex tag, or pages that return a redirect or an error.

A common mistake is a sitemap that dumps everything the CMS produces. That dilutes the signal: you make Google spend crawl budget on pages that deliver nothing. Keep the list clean and focused on pages that can turn visitors into leads. Which pages those are ties in closely with how you build your website architecture for SEO: a logical structure with clear internal links makes your sitemap tidy almost by itself.

Technically, each sitemap file is capped at 50,000 URLs and 50 MB uncompressed. For a B2B site you rarely hit those limits, but if you go over, split the file into several sitemaps and reference them from a sitemap index file.

How do you use robots.txt and noindex without mistakes?

Robots.txt and noindex look alike, but they do something fundamentally different, and that is exactly where things tend to go wrong. Robots.txt controls whether a bot may crawl a page. The noindex tag controls whether a page may appear in the index.

That distinction is crucial because of one trap. If you block a page in robots.txt, Google cannot visit the page, and therefore cannot read the noindex tag on it either. The consequence: if you really want to keep a page out of the search results, you put a noindex tag on it and deliberately leave it crawlable. If you block it in robots.txt instead, it can still surface in the results without you being able to get it out.

A few practical guidelines for a new site:

• During the build you do not want Google picking up the site. The safest approach is to put the entire staging environment behind a password or HTTP authentication. Do not rely on robots.txt alone to keep a site hidden, because it blocks crawling, not visibility.
• Do not forget to remove blocks at launch. The classic disaster is a site that goes live while a site-wide noindex or a blocking robots.txt from the build phase is still in place. Your new site looks great, but shows up nowhere. This is also one of the risks during a move, so read up on how to migrate a website without losing rankings.
• Use noindex deliberately for pages you need but that do not belong in Google, such as thank-you pages and thin tag overviews.
• Point your robots.txt at your sitemap with a line that states the sitemap’s location. That helps search engines find the file.

Make sure the pages in your sitemap and your noindex choices are consistent. A page you set to noindex but leave in your sitemap sends contradictory signals.

How do you submit your sitemap in Google Search Console?

You submit your sitemap by first verifying your site in Google Search Console and then adding your sitemap URL in the sitemaps report. It is a free step, and for a new site one of the first things you do.

Broadly speaking:

1. Create a free Google Search Console account and add your domain as a property.
2. Verify that you are the owner, usually through a DNS record or a verification file. Many CMSes have a built-in option for this.
3. Go to the sitemaps report and enter the path to your sitemap, usually something like sitemap.xml.
4. Check back after a while to see whether Google could read the sitemap and how many URLs it discovered.

After that, you can keep tracking in Search Console which pages get included, which do not, and why. If you want a specific new page picked up faster, you can request indexing through the URL inspection tool. Do bear in mind that Google makes the final call: submitting speeds up discovery, but forces nothing.

When do you sort this out during the build of a new site?

The right moment is during the build, not as aftercare once you have launched. Your sitemap, robots.txt and indexing choices are not a technical side issue: they decide whether your investment in a new site becomes findable at all.

In practice that means: during the build you keep the staging environment shielded, you plan which pages may be indexed, and you put temporary blocks on a checklist to be removed at go-live. On launch day you check that the site-wide noindex is gone, that robots.txt allows crawling, that your sitemap contains the right pages, and that you have submitted it in Search Console. Those few checks prevent the most painful mistake in web development: a beautiful new site that stays invisible for weeks.

At Customer Impact we build conversion-focused B2B sites, and we see this configuration as part of solid craftsmanship, not as a separate extra. A site that has to deliver leads must first be found correctly. You can read how we approach that technical foundation on our website development page, and the broader context is in our guide to building a B2B website.

The short summary

Creating an XML sitemap is not complicated, but the details determine the result. Only put valuable, indexable end pages in your sitemap, use robots.txt for crawling and noindex for visibility without mixing them up, and submit your sitemap through Search Console. Most important of all: sort this out during the build and remove every temporary block at go-live, so your new site is understood by Google straight away.

Want to be sure your new site is technically sound and actually delivers leads? Book your free intake and we will look at it together.