Search Console API: exporting and automating your SEO data

The Google Search Console interface is fine for looking something up quickly, but the moment you want to work with your data seriously, you hit a wall: every export stops at 1,000 rows. For a B2B site with hundreds of pages and thousands of long-tail queries, that is a fraction of the story. In this article you will learn how to pull in all of your data through the Search Console API and the BigQuery bulk export, and how to automate that export instead of clicking through it manually every month.

Why the 1,000-row limit is a problem

The Performance report in Search Console shows you clicks, impressions, CTR and average position per query and per page. Handy, until you hit the export button and only 1,000 rows come along. On a small site you will not notice, but the queries that push your site to the top are exactly the ones in the long tail: a great many searches that each occur a couple of times a month.

That long tail is gold for B2B companies. These are often the specific, purchase-ready searches your best leads come from. If you cannot see them, you are steering blind. You optimise for the few popular terms that happen to sit at the top of the exported list, while the real opportunities are cut off at the bottom. An SEO specialist needs complete data, not a sample of the top thousand.

The interface has a second limitation: you can look back a maximum of 16 months. For seasonal patterns or a multi-year content strategy, that is too short. You solve both problems by pulling the data outside the interface.

Two routes beyond the limit

There are two ways to get all your data out of Search Console, and they complement each other.

The Search Console API lets you fire queries at your property programmatically. You request, for example, all queries for a specific page, filtered by country or device, and get back far more than 1,000 rows. The API runs on generous, free quotas and is perfect for scheduled, repeatable exports: every Monday morning, last week’s numbers into a Google Sheet or a dashboard.

The BigQuery bulk export is a daily, automatic dump of your raw Search Console data into a BigQuery dataset. You switch it on once and from then on the full, unsampled data lands in your data warehouse every day, without the 16-month limit. From the moment you activate the export, you build a historical dataset that reaches back as far as you let it run.

Rule of thumb: use the API for targeted, ad-hoc and scheduled exports, and switch on the BigQuery export as soon as you want to keep data over longer periods and connect it to other sources.

The Search Console API in practice

To get started with the API you need three things: a Google Cloud project, the Search Console API enabled within it, and a service account or OAuth access that you then add as a user to your Search Console property. That last part is often forgotten: the API only gets to your data once the associated account explicitly has rights in the property.

A typical export query goes through the Search Analytics endpoint. You specify a date range, choose your dimensions (query, page, country, device, date) and set a row limit of up to 25,000 per request. If you want more, you paginate with a start row that you increase per request. In a handful of requests you pull in the complete dataset, including the long tail the interface hides.

In practice you write this once in a small script (Python with the Google client library is the common choice) and put it on a schedule. The API costs nothing within the standard quotas, so you can run it daily or weekly without worry. You send the output to whichever destination works for you: a sheet for a quick glance, or straight into a dashboard your team already uses.

Important to know: the API applies the same thresholds and anonymisation as the interface. Rare queries that do not meet a privacy threshold stay hidden, even through the API. That is not a bug but a deliberate choice by Google to protect individual users.

When to choose BigQuery

You switch on the bulk export to BigQuery in your property settings. You link a Cloud project, give the right service account write access to your dataset, and from the next day onwards the data flows into two tables: one with query data and one with URL impressions. From that point on you have fresh, unsampled data every day.

The power lies in the joining. In BigQuery you combine your Search Console data with other sources: CRM data on closed deals, advertising costs, or your analytics. That lets you answer questions that are impossible in the interface, such as which organic searches ultimately led to a quote request. That is exactly the kind of analysis that lifts your SEO from a vanity number to a pipeline driver.

The downside is that BigQuery has a query-based cost model and takes some technical setup. For most B2B sites the costs stay low as long as you filter your queries neatly by date and do not needlessly scan the full table. Start small, expand once the value proves itself.

From a one-off export to a fixed reporting pipeline

The biggest mistake we see is companies treating the export as a goal in itself: pull one big file, look at it, and let it sit. Data only becomes valuable once it lives in a repeatable process that answers the same questions time after time.

A working pipeline looks roughly like this. The API or BigQuery export delivers raw data daily. A transformation step cleans it up and connects queries to themes or to your content clusters. A dashboard shows the numbers that matter for your growth: not “how many queries”, but whether your most important commercial pages are gaining visibility and whether the right long-tail intents are growing. That way Search Console becomes a sensor in your growth engine instead of a report you open now and then. How you then read that complete dataset and pull opportunities out of it, you can read in analysing Search Console data in depth.

For that, the underlying foundation does have to be right. An API export is of little use if pages are not properly indexed or if your Search Console property is not set up correctly. The data export is the measurement layer; the optimisation and the technology underneath determine what there is to measure.

This is where data and strategy meet. An experienced SEO specialist reads in the full query export not only what worked, but also which untapped intents are there, which pages are cannibalising each other and where a new page could deliver pipeline. The export is the starting point; the interpretation is where the difference lies between measuring and steering.

Getting started

Start with the API if you quickly want more than 1,000 rows and want to automate your reporting. Add the BigQuery export as soon as you want to keep data over longer periods and connect it to revenue. And from day one, build towards a fixed pipeline, not towards one-off exports.

Do you want your Search Console data to genuinely drive pipeline instead of gathering dust in a sheet? Get in touch and we will look together at how to cast your measurement and optimisation layer into an orchestrated growth engine, aimed at leads and revenue, not at vanity rankings.