Server log analysis for SEO: what Googlebot really crawls

Search Console samples and summarises; server logs record every request. With a few shell commands or a spreadsheet you can see exactly which templates Googlebot spends its time on, which pages it ignores, and which errors it keeps hitting.

DeskSEO editorial team · Nov 8, 2026 · 6 min read

A signalwoman watches the tracks from a sunlit signal box lined with levers

Every request to your web server leaves a line in an access log: who asked, for what, when, and what the server answered. For SEO, those lines are the ground truth about crawling. Search Console tells you what Google chose to summarise. Third-party crawlers tell you what could be crawled. Only logs tell you what search engine bots actually fetched, how often, and what they got back.

Log analysis sounds like a big-site discipline, and for sites with millions of URLs it is essential. But even a site with a few thousand pages can learn useful things from a month of logs: that Googlebot spends a third of its visits on a calendar widget, that a redirect chain is hit thousands of times a day, or that the newest product pages have not been crawled once.

Getting the logs

Where the logs live depends on your stack.

Your own server (nginx, Apache): usually under /var/log/nginx/ or /var/log/apache2/, rotated daily or weekly and often compressed. Ask for at least 30 days.
Managed hosting: many hosts expose raw access logs in the control panel, sometimes only for a few days. Download them regularly if retention is short.
CDN in front of the site: if the CDN caches HTML, many crawler requests are answered at the edge and never reach your server. You need the CDN's request logs for a complete picture.
Platforms such as hosted shop builders typically do not provide raw logs at all. In that case, the Crawl Stats report in Search Console is the best you can get.

Make sure the log format includes the user agent and, ideally, the response time. A typical combined log line looks like this:

66.249.66.1 - - [03/Nov/2026:06:12:44 +0000] "GET /hiking-boots/?sort=price HTTP/1.1" 200 48211 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" 0.412

Logs contain IP addresses, which are personal data in many jurisdictions. Strip or hash client IPs for non-bot traffic before handing logs to anyone outside the team, and keep them only as long as you need them.

Verify the bots before you count anything

The user-agent string is trivial to fake. Scrapers routinely call themselves Googlebot, and on some sites fake Googlebots outnumber real ones. Counting them inflates crawl figures and can hide real problems.

There are two ways to verify:

Reverse and forward DNS. Look up the hostname for the IP (host 66.249.66.1). Real Googlebot resolves to a name under googlebot.com, google.com or googleusercontent.com. Then look up that hostname and confirm it returns the same IP.
Published IP ranges. Google, Bing and several AI crawler operators publish the IP ranges their bots use as JSON files. Matching each IP against those lists is faster for millions of lines.

Do the same for any other bot you care about, such as Bingbot or the AI crawlers covered in AI crawlers and robots.txt.

The questions logs answer

Once you have verified bot requests, group the URLs into templates (product pages, category pages, filter URLs, blog posts, static assets, redirects, and so on) using URL patterns. Then the useful questions are mostly simple counts.

Question	How to compute it	Warning sign
Where does crawl go?	Bot requests per template, as a share of the total	Parameter or filter URLs take a larger share than products or articles
What do bots get back?	Requests per status code per template	Many 3xx (internal links to redirects), 404s on URLs still linked, any 5xx
How fresh is the crawl?	Days since last bot hit, per important URL	Key pages not crawled for 30+ days
What is never crawled?	URLs in your sitemap or database with zero bot hits in the period	New or deep pages with no hits at all
How fast are responses to bots?	Median and 95th percentile response time per template	One template much slower than the rest
What changed?	Daily bot hits over time	Sudden drops after a deploy or robots.txt change

The "never crawled" list is often the most valuable output. Join your sitemap URLs to the log data and sort by "last crawled". If a third of your products have not been fetched in a month, that explains slow indexing better than any audit score, and it points at internal linking or crawl budget as the place to work.

Hands holding a brass hand lens over layered sandstone in bright sun

Doing it without special software

Dedicated log analysers exist and are worth it for large sites, but a lot can be done with command-line tools on a sample. Some examples, assuming the combined format above:

# requests per status code from anything calling itself Googlebot
grep 'Googlebot' access.log | awk '{print $9}' | sort | uniq -c | sort -rn

# top 30 URLs fetched by Googlebot
grep 'Googlebot' access.log | awk '{print $7}' | sort | uniq -c | sort -rn | head -30

# share of Googlebot requests that contain a query string
grep 'Googlebot' access.log | awk '{print $7}' | grep -c '?'

These counts use the user agent only, so treat them as a first look and verify IPs before drawing conclusions. For anything more, load the parsed lines into a spreadsheet or a small database and build a pivot table by template and status code. A month of logs for a mid-size site fits comfortably in either.

Patterns worth recognising

A spike in crawling after a release often means new URLs appeared: a new filter, a changed URL format, or links with tracking parameters. Check which template the extra requests went to.
Bots hammering a redirect means internal links still point to the old URL. Fix the links; the redirect alone is not enough.
Many requests for URLs that 404 are normal after a cleanup, and fade over weeks. If they persist, something still links to them, possibly an old sitemap.
Googlebot Smartphone dominating is expected. Since mobile-first indexing, most crawling uses the mobile crawler; see mobile-first indexing.
A sudden fall in crawling together with slower response times or 5xx errors means Google is backing off to protect your server. Fix server health first; see HTTP status codes for SEO.
Heavy hits on JavaScript and CSS files are part of rendering. If file names change on every deploy, bots refetch everything; stable, versioned file names reduce the load.

Looking beyond Googlebot

Logs also show you everyone else. Bingbot matters directly for Bing and for the products built on its index. AI crawlers such as GPTBot, ClaudeBot and PerplexityBot fetch pages for training or for answering questions, and user-triggered fetchers such as ChatGPT-User visit when someone asks an assistant about a specific page. Counting each of them over time answers questions Search Console cannot: whether the AI crawlers you allowed in robots.txt actually come, which pages they read, and whether a crawler you blocked is still showing up under a different name.

Logs also expose aggressive scrapers and SEO tools that crawl harder than any search engine. If one unverified bot accounts for a large share of all requests and slows the site for everyone, rate-limiting it at the server or CDN is a legitimate fix. Just make sure the rule cannot catch verified search engine crawlers, and check afterwards that their request counts did not fall.

Turning findings into actions

A log analysis is only useful if it ends in a short list of changes. A typical outcome for a mid-size shop might be: disallow two sort parameters, update 120 internal links that point to redirects, add a "new arrivals" block to category pages so recent products are linked from crawled pages, and cache filter-free category HTML to cut response time for the template Googlebot fetches most. Each item should name the metric that will show whether it worked: crawl share by template, last-crawled date for new products, median response time.

Repeat the analysis a month after the changes, with the same template groups, and compare. Combined with a regular technical check, such as a weekly DeskSEO check that records response time and status, it gives you an early warning when crawling starts to drift again.

Questions

How do I know a request is really from Googlebot?

Do a reverse DNS lookup on the IP address: it should resolve to a host ending in googlebot.com, google.com or googleusercontent.com, and a forward lookup of that host should return the same IP. Google also publishes the IP ranges its crawlers use as JSON files, which is easier to check at scale.

How much log data do I need?

Two to four weeks is a good minimum for patterns. For large sites, even a few days gives a useful picture of crawl distribution. Keep at least one complete month if you want to compare before and after a change.

My site is behind a CDN. Are my server logs enough?

Often not. If the CDN serves cached pages, those requests never reach your origin server, so its logs undercount crawling. Use the CDN's own request logs where available.

Is Search Console's Crawl Stats report a substitute?

It is a useful summary, with breakdowns by response code, file type and purpose, but it groups data and shows only example URLs. Logs give you every request, which you need for per-template and per-URL analysis.