How we hold every change against a control
Isolating a single collection-page rewrite from season, algorithm and everything else shipping that week — the design of the holdout, and what it costs us to keep it honest.
A collection page gets rewritten on a Tuesday. Three weeks later organic revenue on it is up nineteen per cent. Nobody in the room can tell you why — and every one of them will offer a reason.
Why attribution fails
Search traffic moves for reasons that have nothing to do with the work. Seasonality, an algorithm update, a competitor going out of stock, a promotion running on another channel. Any of them will produce a chart that looks exactly like a successful optimisation.
The usual answer is a before-and-after window. It is the weakest possible design: it compares a page to its own past, in a period where everything else also changed.
Before-and-after is not measurement. It is a chart with a story attached.
Designing the holdout
Every write the engine makes is registered against a control group of pages on the same site that were deliberately left alone. The control is picked to match the treated pages on the things that matter and on nothing else:
- Same template. A collection page is only ever compared to other collection pages.
- Comparable pre-period demand, so a long-tail page is never measured against a hero page.
- Same crawl and indexation status going in — an unindexed control tells you nothing.
- Held out for the full read window, even when the change looks like an obvious win.
That last one is the expensive part. A page in the control group is a page not getting the improvement, and holding it back has a real cost. We pay it because the alternative is a number nobody can defend.
What we actually compare
The comparison is the difference in the change: how the treated pages moved, minus how the control moved over the same days. If the whole category rose eleven per cent and the rewritten pages rose thirty, the change earned nineteen — not thirty.
Below that sits the read window. Fourteen days is the floor; pages with thin traffic wait longer rather than report early. A change with no readable signal is reported as exactly that, not rounded up into a win.
Reverting the losers
Roughly one change in nine underperforms its control. Those get reverted automatically, and the revert is logged in the same ledger as the write, with the numbers that triggered it.
This is the part most tools have no answer for, because it requires being willing to say a recommendation was wrong. The engine has no ego about it.
What it costs us
Holding controls slows headline numbers. It means we ship improvements to fewer pages in the first month of an account, and it means our case studies are smaller than they could be.
It also means that when we tell you a change earned something, that figure survives being asked how we know.