GPTBot robots.txt: how to let GPTBot, ClaudeBot and PerplexityBot read your site correctly

A GPTBot robots.txt line will not get you there on its own. Allowing AI crawlers on your website means controlling your entire fetch chain: robots.txt, server, CDN, firewall and the way your pages render. Only when all of those layers cooperate can bots like GPTBot, ClaudeBot and PerplexityBot read your content in full and interpret it correctly. This article shows which crawlers exist, where they get stuck in practice and how to configure your site so AI models genuinely see you.

This is the mirror image of the blocking question. If you are still unsure whether you want to let bots in at all, read blocking or allowing AI bots first. Here we assume you want to be found, and we look at the technology behind it. For the wider context, this fits within our guide to having a B2B website built.

Which AI crawlers exist and what do they do?

Every major AI provider runs not one bot, but a small fleet of crawlers, each with a different job. That distinction matters, because it determines which kind of access actually earns you visibility in AI answers.

Broadly speaking, they fall into three categories:

• Training crawlers fetch content to improve a future model. OpenAI’s GPTBot and Google-Extended are the best-known examples.
• Search and index crawlers build the live index from which an AI search system composes its answers. Examples are OAI-SearchBot and PerplexityBot.
• User-triggered fetch bots retrieve a page at the moment someone asks a question about it, such as ChatGPT-User and Perplexity-User.

This distinction has a practical consequence: for B2B visibility, the search and fetch bots are your infrastructure. They decide whether your brand appears in ChatGPT Search, in a Claude answer or in a Perplexity result. Allowing training crawlers is more of a long-term play, about how models learn to know your category. The names of these bots change regularly, so treat every list as a snapshot and check your server logs periodically for new user agents.

How do you correctly allow AI crawlers in robots.txt?

You allow an AI crawler by not blocking it in robots.txt, or by explicitly granting it access. Every bot is a separate user agent, so a rule for one does not touch the other.

A minimal, open setup looks like this:

User-agent: GPTBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: ClaudeBot
Allow: /

In practice, an empty or missing Disallow is already enough: whatever is not blocked may be fetched. The mistake we see most often is an overly broad Disallow: / that was once set up as a test and never removed, or a block that accidentally hits all user agents (*). So do not only check whether you are allowing the right bots, but above all whether a general rule is not quietly shutting them out.

If you want to steer per content type, leave your public knowledge and service pages open and close off only genuinely restricted paths (client portals, paid reports). We worked out that path-by-path trade-off in the blocking article. Do remember: robots.txt is a polite instruction, not a lock. Truly sensitive content belongs behind a login, not just behind a rule.

Why is opening up robots.txt not enough?

Because a crawler only reads your page once it gets past your server, CDN and firewall, and that is exactly where many sites get stuck unintentionally. Your robots.txt can be perfectly open while a security layer still holds the bot back.

The most common silent blocks:

• Firewall or WAF rules that turn bots away based on user agent or behaviour. Many security services and CDNs have default rules that flag AI crawlers as “unwanted traffic”. The bot then gets a 403 or a challenge instead of your content.
• Bot management that throws up a challenge, such as a JavaScript challenge or CAPTCHA. An AI crawler will not solve it and gives up.
• Rate limiting set too strictly, so a bot gets cut off after a few requests and never fetches your site in full.
• Geoblocking or IP restrictions that turn away traffic from certain regions or data centres, precisely where many crawlers come from.

The way to check this is your server logs. Search for user agents like GPTBot or PerplexityBot and look at which status code they get back. Plenty of 200 codes means they are getting in; plenty of 403s or challenges means a layer above robots.txt is blocking them. This is part of the broader website architecture and technology that determines whether you are findable at all.

Why do AI crawlers sometimes read only half my page?

Because most AI crawlers do not execute JavaScript. They read the HTML your server returns in the first response, and if your most important content only appears after client-side rendering, they see a largely empty page.

For B2B sites, this is the most underestimated problem. A modern single-page application or a build that leans heavily on JavaScript can look perfect in a browser, while a crawler only receives a skeleton: an empty <div> and a pile of scripts. Classic search engines like Google can render JavaScript to some extent, but with today’s AI crawlers you cannot assume that.

What keeps your content crawlable:

• Server-side rendering (SSR) or pre-rendering, so your core copy, headings and links are already in the initial HTML.
• Statically generated pages for content that does not differ per user, such as knowledge articles and service pages.
• Real HTML elements: text in <p>, headings in <h1> through <h3>, navigation in real <a> links instead of JavaScript handlers.
• No critical content behind interactions such as tabs, accordions or “read more” buttons that only load with JavaScript.

Whether you build in WordPress, Webflow or a custom/headless stack: it is not about the platform, but about whether the final page delivers its content server-side. A headless CMS can be perfectly crawlable, provided you add an SSR or static layer and do not assemble everything in the browser. In every website development project we look at how the pages render, not just how they look in a browser.

How do you make your content not just readable but usable?

Access and rendering make sure a crawler gets in and sees your text. Whether you are then understood and cited correctly depends on how clearly your content is structured.

A few things that make the difference:

• Fast, stable pages. Slow or error-prone pages get fetched less completely. Good Core Web Vitals help not only your visitors but also the bots working within a time budget.
• A clear heading structure. A logical hierarchy of H2s and H3s, phrased as the questions your audience asks, makes your content easier to parse and quote.
• Structured data. With schema markup you give models explicit context about who you are, what you offer and how your pages connect.
• A clean sitemap and internal links. That way crawlers find all your relevant pages and understand how your site fits together.

This is exactly where technology and visibility meet. The crawl layer is a precondition; what you do with it substantively belongs to generative engine optimization. Granting access therefore does not automatically mean you get cited, it means you are eligible in the first place.

The short summary

Allowing AI crawlers is not a switch but a chain. Open your robots.txt to the bots that feed your visibility, check that your server, CDN and firewall are not quietly blocking them, and make sure your most important content sits server-side in the HTML rather than appearing only after JavaScript. Only when those three layers line up do GPTBot, ClaudeBot and PerplexityBot read you in full. The rest, clean structure and fast pages, determines whether you are also useful enough to be cited.

Want to know whether your site is technically ready to be read correctly by AI crawlers, and whether that can actually generate leads? Book your free intake call and we will take an honest look with you.