Using Screaming Frog for a Technical SEO Audit

Screaming Frog is a desktop program that crawls your website the way a search engine does and puts the technical problems into a clear list: broken links, missing titles, duplicate content and more. The free version is enough for a small B2B site (up to 500 URLs), the paid licence costs around 245 euros per year. In this article you will read what the tool does exactly, how to set up a crawl and which problems it uncovers. One thing to know upfront: the tool delivers the diagnosis, prioritising and fixing stays your job.

What is Screaming Frog?

Screaming Frog is a British SEO tool, in full “Screaming Frog SEO Spider”, that came to market in 2010. It is a crawler: you enter a starting address and the program analyses the HTML of that page (titles, headings, images, meta descriptions, structured data) and then follows every internal link to the next pages. That way it walks through your entire site, exactly as the bots of Google and Bing do, in line with Google Search’s guidelines.

The result is a detailed table with every URL and what is wrong on each page. That makes it one of the best-known tools for a technical SEO audit, a process that Semrush explains step by step. Unlike many modern cloud-based audit tools, Screaming Frog does not run in the cloud but on your own computer. That has advantages: you do not have to log in anywhere and your crawl data stays local. The downside is that a large site puts a heavy load on your machine, and that the learning curve is steep if you rarely deal with technical SEO.

Do you want the basics of search engine optimisation first? Then read our explanation of SEO. And if you would rather hand over the whole technical side instead of crawling yourself: that is exactly what our SEO service covers.

How do you set up a crawl?

The barrier is low: you can get going almost immediately. Broadly, it works like this.

1. Download and install. Screaming Frog is free to download for Windows, macOS, Ubuntu and Fedora. It is a fairly large file, so the installation can take a while.
2. Enter your URL. At the top there is a field labelled “Enter URL to Spider”. Type your website in there. By default the tool crawls all subdomains (your entire site), but you can also limit it to a specific folder or subdomain.
3. Start the crawl. Click the green Start button. The URLs appear in real time as the tool encounters them, so you get a rough picture before the scan has finished.
4. Read your report. As soon as the progress bar reaches 100 percent, you can filter per tab and export everything to CSV, Excel or Google Sheets.

Those four steps form a fixed sequence together, from installation to a report you can filter and export tab by tab.

How long a crawl takes depends on the number of URLs and the speed of your computer. A large site with thousands of pages can take several hours, a typical B2B site is usually done in minutes.

One important detail about that 500-URL limit in the free version: every single link counts as a URL, not just pages. Your images, CSS files and JavaScript files add up too. A site with 80 pages can therefore hit the limit quickly. For most small B2B sites, 500 is enough for a first review; for larger sites you need the paid version.

Which problems do you find with it?

This is where the tool proves its worth. After a crawl, the most common technical problems sit in a handful of tabs. The main ones:

A few things that pay off most in practice:

• Tracking down broken links. Under Response Codes you filter on 4XX and see every broken link. Click an error and open the Inlinks tab at the bottom: it shows which page the broken link sits on, so you can repair it in a targeted way.
• Checking titles and descriptions. You can filter on missing, duplicate, too long or too short. On a B2B site that has grown over time, dozens of pages without a unique title often surface here.
• Analysing internal links. Via the Internal tabs you see, for each page, which other pages link to it (Inlinks) and which links it contains itself (Outlinks), including anchor text and whether a link is follow or nofollow. Good internal links keep visitors on your site longer and help Google index your whole site.

In the paid version you can additionally connect tools via Configuration > API Access, for example PageSpeed Insights for load speed or Google Search Console and Google Analytics for visitor data. That way you combine technical data with real figures on what pages deliver.

What does the tool not do?

This is the part missing from most guides. Screaming Frog gives you a list of problems, but no judgement on what is important. A fresh crawl of an average site easily spits out hundreds of “errors”. If you blindly try to clear them all, you will be busy for weeks with things that gain nobody anything.

The tool simply does not know which pages bring in your customers. A broken link on your contact page or a slow quote page directly costs you enquiries. A missing alt text on an image buried deep in your archive costs you nothing. Both show up equally red in the report.

A crawl is a diagnosis, not a treatment plan. The skill is not in finding errors, but in choosing which ones you fix.

That is why our approach is simple: we run the crawl, connect it to which pages actually deliver leads and revenue, and tackle the problems that cost money first. The rest of the list is often noise. A tool is an aid, not a strategy. Do you want a structured approach? Then use our technical SEO checklist as a framework alongside the crawl data.

Free versus paid, and alternatives

The free version of Screaming Frog is surprisingly complete. You get:

• Broken links and redirects
• Page titles and meta descriptions
• Meta robots and directives
• Hreflang attributes
• Exact duplicate pages
• Generation of an XML sitemap
• Visualisations

The paid licence (around 245 euros per year, per person and not per device) unlocks unlimited crawling, scheduled reports, JavaScript rendering, validation of structured data, connections with Search Console and Analytics, and technical support. For anyone who reviews larger sites regularly, that pays for itself quickly.

Screaming Frog is not the only option. Other crawlers, each with its own emphasis:

• Sitebulb. More expensive, but both desktop and cloud, and it explains problems in plain language through a “Hints” system.
• Visual SEO Studio. Closest to Screaming Frog in look and price, but a bit more accessible.
• Sitebulb, Lumar, Botify. Lumar and Botify aim at large enterprises; Botify is the most technical and prioritises for you which actions to take first.
• Xenu’s Link Sleuth. Free and simple, but it only checks broken links.

Which one you choose matters less than what you do with the outcome. For a broader overview of what exists, see our article on SEO tools.

Frequently asked questions about Screaming Frog

Is Screaming Frog free? Yes, there is a free version that handles up to 500 URLs per crawl. Remember that every link counts (including images, CSS and scripts), not just pages. For unlimited crawling and extra features you pay around 245 euros per year.

How often should I crawl my site? For an average B2B site, at least once a year is enough, more often if your site is large or complex. Run a crawl in any case after major changes, for example a new URL structure or a migration to a new site.

Do I need technical knowledge? Setting up a basic crawl works without a developer background. But interpreting the data and knowing which errors are truly a priority takes experience. That is where the learning curve sits, and that is where do-it-yourselfers often go wrong.

Does it also work for competitor research? Yes. You can crawl a competitor’s site and see, via the Page Titles, H1 and H2 tabs, which words and themes they use. Handy as an addition to your own research, though it does not replace a real keyword strategy.

Is a crawl the same as an SEO audit? No. A crawl is the technical diagnosis. A full SEO audit also looks at content, keywords, links from outside and what pages deliver. The crawl is the starting point, not the end point.

Would you rather have the diagnosis and the solution?

Screaming Frog shows you what is technically wrong on your site. Running a crawl is the easy part; choosing and fixing the right errors is the work that counts. We are a small Belgian B2B team that does not steer on a report full of red ticks, but on the few problems that cost you customers and revenue. No webshop approach, no vanity metrics, just honest advice on what is worth repairing.

Curious what a review of your site would deliver? Plan your free intake