How to run a simple before-and-after SEO test, and why correlation misleads

A before-and-after test compares a page against its own recent past, and it only means something if you control for what else changed.

You change a title, and two weeks later clicks are up. Did the change work? Maybe. A before-and-after test is the simplest way to find out, but it is easy to fool yourself, because things move for many reasons at once.

The basic method#

1. Pick one page and one change. Changing five things at once means you cannot say which mattered.
2. Write down the prediction first: what you changed, what you expect to move, by how much and by when. A prediction made after the result proves nothing.
3. Record the before. In Search Console, note clicks, impressions, CTR and position for that page over the previous 28 days.
4. Make the change and note the date.
5. Wait. Give it at least two to four weeks. Search results take time to update.
6. Record the after over an equal length of time and compare.

Why correlation misleads#

Two things happening together does not mean one caused the other. Traffic can rise because of:

• Seasonality, the normal rise and fall of your market through the year.
• A Google update that shifted results for everyone.
• A competitor changing or leaving.
• A campaign, a press mention or a new link.
• Plain random variation, especially on pages with few visits.

Making the test fairer#

• Compare against pages you did not change. If they rose by the same amount, your change did not cause it.
• Compare the same weeks last year, if you have the data, to account for the season.
• Use impressions and position as well as clicks. If clicks rose with no change in impressions, the title likely helped. If impressions rose first, the cause is probably outside the page.
• Look at the shape. A step change on the date of your edit is more convincing than a slow drift that started earlier.

Common mistakes#

• Calling it after three days.
• Testing on a page with a handful of visits.
• Keeping only the successes. Log the changes that did nothing as well.

For a ready-made way of keeping the log, see run an experiment and log hypotheses. For how PageSoar weighs a result, read how a change is measured.