Your Server Log Files Have the Answers!
+ FREE Python script!
Hello Riddlers!
📌 Your support keeps this newsletter FREE —> [☕Buy Me a Coffee☕]💡 Want more from me? [SEO Strategy Course] | [Build Your Own SEO Tools (with Python & Agentic AI)] | [Premium SEO Mastermind]
My new Free Log Files Tool
Not sure if you’ve heard but I recently launched a Log File Analyzer Tool and it’s free! Also it’s super cool, give it a test drive.
Talking about server log files feels especially relevant right now. While everyone’s focused on “AI this” and “AI that,” log files are sitting there quietly holding some of the most valuable data you can get.
Honestly, they’re a goldmine.
If you want to see what I mean, check out my example report—it’ll give you a clear picture of the kind of insights you can uncover.
Anyways I decided to make today’s newsletter about Log files analysis, plus I’m giving you a raw python script so you can build your own workflows on top of this code!
Also if you don’t know what log files are, don’t worry, I’ve got you:
What are server log files for SEO?
Server log files for SEO record every request made to a website’s server, including pages crawled, response codes, and bot activity. You can use server log files to analyze how search engine and AI bots crawlers access your site, identify crawl errors, and optimize crawl budget and indexing efficiency.
We don’t have the data we need
As annoying as this sounds, it’s the truth. We are missing a lot of the data we need in SEO. Google search console, our main source of data, has a lot of issues. Last year when I called it buggy, that ruffled some feathers, I still stand by that… it’s a buggy tool in general.
Google search console data is:
incomplete
inaccurate (e.g. impressions have been inflated for at least the last 12 months)
and depending on your website structured, maybe hard to segment in a clean way.
and that’s just the data part, let’s not get into the technical issues.
And that leaves us as SEO hanging, and looking for the best third party tool to get some more information. But again, third party tools come with their own limitations, and data assumptions, I would treat them as directional.
I guess this is a great motivation to build your own tool, and building your own log file analysis tool can give you answers you may not find elsewhere.
So… build your own data advantage
In a time were all we talk about is AI bots, log files cannot be any more important. From your log files you can find more concise answers to questions like:
How much referral traffic is sent to our website via chatgpt/gemini/perplexity/ect…
How many times does AI bots access the content on our websites.
Which pages on our websites are accessed more often by AI bots (this is a major optimization opportunity)
and many more…
But the log files analysis does not stop there, you can also understand the behavior of Google bot on your website:
How often does Google bot visit your website
What pages are visited regularly and what pages are rarely visited (important optimization opportunity right here!)
Are there any redundant page requests that make no sense (maybe we should noindex those pages?)
etc…
You can find pages to optimize, and opportunities to reduce your server costs as well. It’s a gold mine of insights.
Build your own log file workflow (Python script included)
So… as part of my Build Your Own SEO Tools (with Python & Agentic AI) course, I added this simple Colab script that generates this dashboard for you:
You can use this script as a starting point, and build your own workflow!
And That’s a Wrap (Almost 😄)
As someone who’s been working with data in SEO for a while, analyzing, learning, and trying to finding meaning and connections between the numbers we have and the things we do, I find log files very valuable.
If you’re looking for something new to try out this week, let this be at the top of your list. Ping your dev team today and ask them for few rows of data (1000+) from your log files and run the script (or my free tool) so you can understand how different bots interact with your website.
That’s that for today folks and see you next newsletter 🙏
Support the Riddler!
Sign up for my newsletter if you’re not already.
Share the newsletter and invite your friends to signup. Help me reach 2k signups by end of 2025 please 🙂
Provide feedback on how I can make this newsletter better!!!
If you’re an SEO tool or an SEO service provider, consider sponsoring my newsletter. I’m also open to other partnership ideas as well.
Disclaimer: LLMs were used to assist in wording and phrasing this blog.