Level 4 · Test the test with an A/A split

Two rates, one question

An A/B test ends with two conversion rates. The question is whether the gap between them is bigger than the gap you would expect from luck alone. The two-proportion z-test answers it by measuring the gap in standard errors:

z = (rate_B - rate_A) / se

A z of 0 means no gap. A z around ±2 means the gap is about two standard errors wide, which happens by chance only about 5% of the time.

The pooled standard error

The test assumes the null hypothesis (no difference), so it estimates one shared rate from both arms together, the pooled rate, and uses it for the standard error of the difference:

pooled = (conv_a + conv_b) / (n_a + n_b)
se = math.sqrt(pooled * (1 - pooled) * (1 / n_a + 1 / n_b))

On the A/A split each arm has 20,000 sessions. Arm A converts 577 times (2.89%), arm B 616 times (3.08%). The pooled rate is 2.98% and the standard error of the difference about 0.17 points.

From z to a p-value

Python’s standard library has the normal curve built in. NormalDist().cdf(z) is the share of the curve below z, so the chance of a gap at least this extreme in either direction is:

from statistics import NormalDist
p_value = 2 * (1 - NormalDist().cdf(abs(z)))

Here z ≈ 1.15 and p ≈ 0.25: a gap like this between two identical halves shows up in a quarter of splits. Nothing to see, which is exactly right, because nothing was different.

Why run an A/A test

An A/A test checks the machinery: assignment, logging, the analysis code. Over many A/A tests about 5% should come out with p < 0.05, because that is what the 0.05 threshold means. It really happens: take the same even/odd split for 2024 only and the full year comes out at p = 0.047. Two halves of identical traffic, “significant”. That is not a bug, it is the 1-in-20 the threshold accepts, and the reason one significant test is evidence, not proof.

Common mistakes

• One-sided by accident. Leaving out the 2 × halves the p-value. Use two-sided unless you decided the direction before the test started.
• Unpooled se in the test. Each arm’s own rate is right for a confidence interval of the difference; the pooled rate is the standard choice for the test itself.
• Rounding z before computing p. Round only the reported numbers.
• Sessions that aren’t independent. If one shopper’s visits land in both arms, the test is overconfident. Split by customer when you can.