A Test That Cannot Fail Proves Nothing About Your SEO

The spreadsheet is always the same shape. A title tag gets rewritten on fourteen pages on the second of the month, organic sessions rise eleven per cent by the twenty first, and the tab is labelled “result”. Nobody had written down what a failure would have looked like. That is not a test. That is a decision that got promoted to a finding somewhere between the edit and the meeting.

The test that cannot fail

Look at the weak points and they are not subtle. This month is compared with last month, which is comparing two different businesses. The change went live everywhere at once, so there is no control group left to compare against. The three week window is inside the noise band of any site without serious traffic.

Every one of those choices makes the conclusion easier to believe. That is the tell. The setup was built to produce a story rather than a finding.

Write the falsification before you ship

The cheapest upgrade is one paragraph, written before the change goes live, that a colleague can open later.

• The metric. Qualified sessions to this template. Not sessions, not impressions, not rankings.
• The threshold. 15 per cent, minimum. A number you would be embarrassed to call a win.
• The window. 45 days, decided in advance so it cannot be extended once the data arrives.
• The failure condition. What we will say and do if it comes in under. Revert, or keep it and admit the reasoning was the justification, not the result.

The moment that paragraph exists, half of the sloppy tests in circulation disappear, because most of them cannot name a threshold the author is confident of clearing.

The study that says what it measured

Compare that with a study Seer Interactive published in April, which tracked 53 brands, 5.47 million queries and 2.43 billion impressions from January 2025 through February 2026. It found the click through rate on AI Overviews bottoming at 1.3 per cent in December 2025 and climbing to 2.4 per cent by February 2026. Pages cited inside an overview also get two to five times the organic click rate of pages left uncited on the same results page.

You can disagree with the conclusion. What you cannot do is argue about what was measured, because the sample, the window and the method are stated. That is the difference between a study and an anecdote with a chart, and it has nothing to do with the size of the company that produced it.

Seasonality will happily answer for you

Search demand has cycles and the cycle does not check with you before it becomes your result. Retail, education, travel and anything tied to a financial year are the obvious cases, and service businesses have quieter versions of the same shape. Two habits fix most of it: compare against the same period twelve months ago as well as last month, and watch the segments you did not touch.

Untouched segments are the most underrated diagnostic available. If the pages you never edited also went up eleven per cent, you have not measured your change. You have measured the weather.

Changing something is not testing something

Question	It was a change	It was a test
Is there a control group?	No, we shipped it everywhere	Yes, and we know which pages are in it
Can we undo it cheaply?	Maybe, which is why we shipped fast	Reverting is scheduled if the threshold fails
What would have counted as failure?	Nobody wrote it down	Written before the change, with a number
Who needs to believe the result?	The client, which is why the story matters	Anyone who reads the method

Plenty of good decisions are not experiments and pretending they are makes them worse. If a page is unclear, fixing the clarity is a judgement call by somebody who knows the audience, and it can be right without being provable. Just stop calling it proof afterwards. Ship the improvement, describe it as an improvement, and keep the causal claim out of the deck.

Why the AI visibility conversation is the worst offender this year

Watch how quickly the industry narrative flipped. Through 2025 the line was that AI Overviews were permanently destroying clicks. By April 2026 the line was that click rates were recovering, on the same underlying data, with the same companies quoted. The dated evidence on AI visibility sits under AI search and getting cited.

Neither story is dishonest on its own, and the swing is the clearest illustration of the problem. The findings were stable. The framing moved with the mood. The durable anchors in that conversation are the dated, documented ones, like Google’s February 2026 Discover update, which said it would reduce sensational content and favour original, in depth work. That one is a fact with a date on it, and you can hold a site against it. “AI is changing everything” is a mood with a chart.

What a small site can measure

• Large effects over long windows, which rules out most of what people want to know and keeps what matters.
• A template starting from zero impressions and getting some. Binary, unglamorous, real.
• A cluster appearing for queries it never ranked for, tracked quarterly.
• Click through rate on one template before and after a pattern change across 40 pages, watched for a quarter.

When a clean test is impossible, write the expectation down

Most in-house teams cannot run a clean experiment on organic search, and the honest response is to accept that and change the standard of evidence rather than lowering it quietly. Reason from mechanism. Ship the change because you believe it. Write the expectation down. Check in 90 days whether it held, and say so either way.

What you must not do is build a retrospective story where a decision you were always going to take becomes a finding you discovered. That habit is expensive. It teaches an organisation that its beliefs are validated when they are merely unfalsified. The same habit runs through query fan-out and through posts that never get cited by AI, where a plausible mechanism gets repeated until it sounds like a result.