Level 1 · Check the split before the result

Why check the split first

An A/B test promises one thing: the two groups differ only in what you changed. If the assignment is broken, that promise is gone and the result means nothing, however clean it looks. So before you look at conversion, you check that each arm got the share of traffic it was supposed to get. That check has a name: the sample ratio mismatch test, or SRM.

Harbor & Pine’s banner test split sessions by session_id: even ids saw the banner (arm B), odd ids did not (arm A). The plan was 50/50, so each arm should hold half of the sessions in the window.

test["arm"].value_counts()
# A    2604
# B    2604

The chi-square check

“Close to half” is not a test. The chi-square goodness-of-fit statistic measures how far the counts are from what the plan predicts:

expected = len(test) / 2
chi2 = (n["A"] - expected) ** 2 / expected + (n["B"] - expected) ** 2 / expected

With two arms there is one degree of freedom, and a chi-square with one degree of freedom is the square of a standard normal variable. So you can get its p-value from the normal distribution in Python’s standard library, no SciPy needed:

from statistics import NormalDist
p = 2 * (1 - NormalDist().cdf(math.sqrt(chi2)))

Here the arms are exactly equal, so chi2 is 0 and the p-value is 1. The split is healthy.

What a failing check looks like

Suppose arm B had lost 3% of its sessions to a slow banner script: 2,604 in A and 2,526 in B. Then chi2 is about 1.19 and p is about 0.28, which still passes. Scale the same 3% loss up to a test of 50,000 sessions per arm and p drops below 0.0001. SRM checks get stricter as tests get bigger, which is exactly why big companies run them automatically. Most teams treat a p-value under 0.001 as a failed split and stop the analysis.

Common mistakes

• Eyeballing the counts. 50.6% versus 49.4% looks fine and can be a real bug on a large test.
• Cutting the window wrong. started_at < "2025-12-31" drops the whole last day, because the timestamps on December 31 are later than midnight. Use < "2026-01-01" for an inclusive end date.
• Checking only after the result disappoints. Run the SRM check first, every time, whatever the result says. A check you only run on bad news is a way of choosing the answer you like.

Interview note

“What would you check before trusting an A/B test result?” The sample ratio is the first answer interviewers hope to hear, followed by the test window, the unit of randomisation and the metric definition.