Level 5 · The test that won by peeking

One look, one promise

When you set the threshold at p < 0.05, you accept that a test with no real difference will be called a winner 5% of the time. That promise assumes you analyse once, at a sample size you chose in advance. Nothing in the maths covers “we checked every Monday and stopped when it looked good”.

What peeking does

Running p-values on a growing test is a random walk. Early on, with few sessions, the rates jump around. Each look is another draw, and if you stop at the first draw under 0.05, you keep the lucky ones and never see them fall back.

Priya replayed the even/odd A/A split, two halves of identical traffic, week by week through 2024:

checked on	sessions per arm	arm A	arm B	p
Jan 28	338	1.48%	3.56%	0.084
Feb 4	422	1.18%	3.32%	0.037
Feb 25	658	1.52%	2.74%	0.126
Mar 31	1,132	2.47%	2.74%	0.692

On February 4 it “won” with a 180% lift. Five weekly checks out of 53 that year came in under 0.05. A team that stopped at the first one would have shipped a change that did nothing.

How bad does it get? In a simulation of 2,000 A/A tests at a 3% conversion rate, checked every day for 30 days, about 30% crossed p < 0.05 at least once. Six times the promised 5%.

Doing it right

• Fix the sample size and end date before launch (the boss level shows how), then analyse once.
• Run whole weeks. Weekday and weekend visitors behave differently; stopping on a Tuesday mixes them unevenly.
• Look for safety, not for wins. Monitoring for broken checkouts is fine; deciding the winner early is not.
• If you must decide early, use a method built for it: sequential tests or alpha-spending rules split the 5% across planned looks.

Common mistakes

• Celebrating a huge early lift. 3.32% vs 1.18% on 422 sessions each is 14 orders against 5. Big early lifts are usually noise.
• “We’ll just run it until it’s significant.” Given enough looks, almost any A/A test will get there eventually.
• Extending a test that “almost” won. Adding days only when p is close to 0.05 is peeking in reverse, and inflates false positives the same way.
• Forgetting the A/A lesson. Even analysed once, a full-year A/A test can cross the line: the 2024 split ends the year at p = 0.047. One significant test is evidence, not proof.