technical seo · 2026-07-10 · Yaaseen Saleh-Mohamed

How to read your server logs like an SEO

Google Analytics tells you what people did on your site. Your server logs tell you what Googlebot did — every URL it requested, when, and what your server gave it back. That's not a nice-to-have. It's the only first-party record of how a search engine actually experiences your site.

Most SEOs never look, because the tooling seems intimidating. It shouldn't be. Here's the practical version.

Step 1: Get the logs

On shared hosting (cPanel), look for Raw Access Logs or Metrics → Raw Access. Download a few weeks' worth. Each line looks something like this:

66.249.66.1 - - [10/Jul/2026:02:14:15 +0200]
"GET /services/dry-needling HTTP/1.1" 200 15320
"-" "Mozilla/5.0 (compatible; Googlebot/2.1)"

That's an IP, a timestamp, the requested URL, the status code your server returned, the bytes sent, and the user agent. Everything below comes from these six things.

Step 2: Ask the four questions that matter

1. What is Googlebot actually crawling?

Group requests by URL and count. On almost every site I've analysed, the distribution is shocking: a large share of crawl activity goes to URLs that don't matter — parameter variants, old paginations, dead assets, staging leftovers. Every request spent there is a request not spent on the pages that make you money.

2. What status codes is it getting?

Count crawler hits by status code. You want to see almost entirely 200s and a modest number of 301s. Red flags:

• Lots of 404s — Googlebot is wasting budget on dead URLs, usually from stale internal links or old sitemaps.
• Any 5xx — your server failed the crawler. If this happens regularly, Google slows down crawling and can drop pages.
• Redirect chains — a 301 to a 301 to a 200 means every hop cost a request.

3. Which important pages is it ignoring?

Take your money pages — the services, the locations, the products — and check when each was last crawled. A page that hasn't been requested in weeks is a page Google doesn't consider worth revisiting. That's usually an internal-linking problem: the page is buried too deep or too few pages point to it.

4. How fast are you serving the crawler?

If your log format includes response time, watch it. Slow responses at crawl time throttle how much of your site Google gets through — and they usually mean human visitors are getting the same sluggish experience.

Step 3: Do it without expensive tools

You don't need an enterprise platform for a site under a few thousand pages. A few honest options:

• Command line — grep Googlebot access.log | awk '{print $7}' | sort | uniq -c | sort -rn | head -25 gives you the top 25 crawled URLs in one line.
• Screaming Frog Log File Analyser — the free tier handles small sites, and it does bot verification for you.
• BigQuery — for ongoing analysis, load logs into a table and query them like any other dataset. This is how I run it for retainer clients, joined against Search Console data.

What this looks like in practice

A typical finding: a practice website where a third of Googlebot's requests were going to calendar-plugin URLs — infinite date pages nobody would ever search for. Blocking those in robots.txt redirected that crawl attention to service pages, whose crawl frequency picked up within weeks. No new content, no new links; just removing the noise.

That's the general shape of log-file work: it rarely tells you to add something. It tells you what to stop wasting.