Level 4 · The tablet result

The garden of forking paths

The overall result is flat: +0.27 points, p = 0.62. Nobody likes a flat result, so the slicing starts: by device, by traffic source, by new versus returning customers. Each slice is a fresh chance for noise to look like a signal. The agency tried eight cuts and found one that shines:

Cut	A	B	p-value
tablet	3.27%	5.36%	0.25
direct	6.35%	7.75%	0.43
mobile	3.35%	3.51%	0.81
desktop	4.75%	4.67%	0.94

and four more traffic sources, all with p above 0.75. Tablets look like a 64% lift. But this is the best of eight, and its own p-value is 0.25.

Why many looks make false wins

At the usual 5% threshold, one test of a banner that does nothing still shows “significance” 5% of the time. That is the deal you accept. Look eight times and the deal changes: the chance that at least one of eight independent cuts shows p < 0.05 is

1 - 0.95 ** 8     # 0.337

about one in three. Slice a flat test by enough segments and a “winning segment” is close to guaranteed. This is the multiple comparisons problem, and it is behind many dashboard “insights” that never replicate.

Small segments, big lifts

Small segments have noisy rates, and noisy rates produce extreme lifts. The tablet cut has 499 sessions and 21 conversions. Three conversions moving from B to A would flip the “64% lift” into a loss. The most dramatic lifts in any segment table almost always come from the smallest rows, which is the opposite of where you should put your trust.

Peeking is the same trap in time

Checking the result every morning and stopping the test the first day p dips below 0.05 is slicing by day instead of by device. Each daily look is another chance for noise to cross the line. In this test the daily p-value never went below 0.05, but on plenty of tests with no effect it does, early, for a day or two. Fix the length before launch, using a sample size calculation, and read the result once at the end.

Common mistakes

• Reporting the best segment as the finding. If you show a segment result, show how many cuts you looked at, and label it exploratory.
• Ignoring the counts. Lifts on 20 conversions swing wildly. Put sessions and conversions next to every rate in the table.
• Never looking at segments. Looking is fine, and checking that no segment was badly hurt is good practice. The mistake is treating the most exciting slice as proof.
• Stopping early on a good day. Decide the test length up front and stick to it.