Running Control Experiments in SEO

This week's newsletter is sponsored by me 😄

Check out my Python for SEO course

More often than not, there's no definitive answer for some of the issues we face in SEO. Specifically when you're trying to figure out "why" something is happening. Why did my traffic drop? why are pages not getting indexed? etc...

Add to this the fact that many SEO tools can cause more confusion, GSC included😄. It's common to find yourself asking: is this an SEO issue on Google, on my website, or is the tool just flagging something that's not an issue?

That's actually why I always recommend using more than one tool when doing a technical SEO audit. But there's also a good approach, having an SEO control experiment.

Don't know what that is? you're in the right place then, in this post I'm going to discuss how you can use the concept of "control" experiment to help diagnose your SEO issues.

TL;DR

• An SEO control experiment is a test where all factors are kept constant except for the issue being tested/investigated.
• An SEO control experiment starts by asking Why did [SOMETHING HAPPEN]?
• Next after brainstorming some potential causes you'd like to investigate, you should rephrase your question to be in the form of: Is [SOMETHING HAPPEN] caused by [THIS]?
• Yes/no questions are great start for your control experiment which will help eliminate potential causes.
• I listed 3 simple examples below, I really think are worth your time!

What is an SEO control experiment?

An SEO control experiment is a test where all factors are kept constant except for the issue being tested/investigated.

In more simple words:

If you want to find out whether something is causing an SEO problem, you need to compare how your website performs when that factor is there and when it’s not.

The goal of this is to help you understand the "why" something is happening. Be it a technical issue, an SEO performance issue on the website, an issue from Google's side (e.g. update or bug), a market issue, or even an SEO tool issue.

How to setup a control experiment

The best way to create a control experiment, is to start with a "why" question.

Why did [SOMETHING HAPPEN]?

Examples:

• Why did [my traffic drop]?
• Why did [the migration decrease traffic]?
• Why are [my pages not getting indexed]?

This [SOMETHING] you're trying to explain is an SEO issue you observed.

Once you have this question, as a good SEO you'll start thinking this could be caused by [THIS] issue. In other words, you'll come up with some potential causes you'd like to look into.

You can use those potential causes this to create your control experiment setup in the format of:

Is [SOMETHING HAPPEN] caused by [THIS]?

for example:

• Is [TRAFFIC DECLINE] caused by [GOOGLE UPDATE]?

Basically start by asking Why something is happening, then reframe your question to a yes/no question: is it caused by this?

Finding the right medium

The idea is to find a situation where you can test the outcome with and without the potential cause you selected.

Let's use this example: Is [TRAFFIC DECLINE] caused by [GOOGLE UPDATE]?

Here's how I would tackle this:

• Test: Compare your performance in GSC with that in Bing Webmaster tools.
• Insight: If your traffic declined only in GSC, then yes it's a Google Issue, if your traffic declined in both, it's not a Google update.

Let's try another example!

One time, I was running a crawl on screaming frog using Googlebot as the agent and to my surprise, the tool could not crawl the website. It's easy to dismiss this and move on, but I wanted to make sure that Googlebot is not facing any issues in crawling this website.

So I asked the first question:

--> Why is [Screaming Frog as Googlebot unable to crawl the website]?

Then came some thoughts:

--> Is [Screaming Frog as Googlebot unable to crawl the website] caused by [the tool itself]?

I know GSC is crawling the website just fine, but that didn't provide a definitive answer, because I was wondering is there some obstacle Googlebot has to overcome to crawl our website, this tool could not imitate?

• Test: I tested the same Screaming Frog setup for other similar websites.
• Result: Screaming Frog was able to crawl those websites normally as expected.
• Insight: So the conclusion was, it is not an SEO tool issue, it's something we're doing on our website. [This test idea was not mine, but was suggested to me by someone]

A third example, because the more the merrier 😄

Let's try the question of the hour:

Is [TRAFFIC DECLINE] caused by [AI OVERVIEWS]?

Here's how I would go about this:

• Test: Collect performance data for the keywords on your website that don’t have AI Overviews. Then, compare their performance trends (going up, going down, or staying the same) with the trends of keywords that do have AI Overviews.
• Insight: If both trends are the same, then it's not AI Overviews!

And That’s a Wrap (Almost 😄)

What makes one SEO better than the other, is their ability to diagnose issue and figure where the culprit is. And while there's no one way to diagnose SEO issues on a website, it's definitely useful to have different approaches under your sleeves.

By breaking down “why” something is happening into simple testable causes and comparing outcomes with and without those factors you eliminate guesswork and get closer to the real answer.

Hope you found that useful 🤩

Thanks for reading this far and see you next newsletter!

Like what you read and want to support me?

• Sign up for my newsletter if you're not already.
• Share the newsletter and invite your friends to signup. Help me reach 2k signups by end of 2025 please 🙂
• Provide feedback on how I can make this newsletter better!!!
• Buy me coffee.
• If you're an SEO tool or an SEO service provider, consider sponsoring my newsletter. I'm also open to other partnership ideas as well.

Disclaimer: LLMs were used to assist in wording and phrasing this blog.