Level 3 · What p = 0.03 actually says

The question a p-value answers

Every significance test starts from a boring hypothesis, the null hypothesis: the two versions perform exactly the same, and any gap in the numbers is luck. The p-value is the probability of seeing a gap at least as large as yours if that boring hypothesis were true.

A small p-value means your data would be unusual in a world with no difference. That is evidence against the null. It is not a direct measure of anything else.

Three things a p-value is not

• Not the chance the result is a fluke. “p = 0.03, so a 3% chance we’re wrong” flips the conditional. The p-value is P(data this extreme | no difference), not P(no difference | data). The second number depends on how often ideas like this one work at all, which the test never sees.
• Not the chance of winning next time. 1 − p = 0.97 is not a probability of anything useful.
• Not the size of the effect. With a big enough list, a 0.1-point lift gets p < 0.001. With a small list, a 20% lift can get p = 0.3. Significance measures surprise; the lift and its interval measure size.

Where 0.05 comes from

The 0.05 cut-off is a convention, not a law. Setting it means accepting that, when nothing changed, about 1 test in 20 will still come out “significant”. You will see that happen on Harbor & Pine’s own data in the next levels: an A/A split, where both halves saw the same site, can cross the line by luck.

Writing it up honestly

A sentence Lauren can send:

The new subject line had a [new]% open rate vs [old]% for the old one
(+[diff] points, 95% CI [low] to [high]). A gap this large would be
unusual if the two performed the same (p = 0.03).

Fill in the brackets from the test; what matters is the shape. It gives both rates, the difference with its range, the p-value, and a plain-words meaning. Nobody reading it will think the lift is guaranteed or huge.

Common mistakes

• “97% confident it’s better.” That is not what the test computed.
• “Not significant, so there’s no difference.” A large p-value means the test could not tell. The interval shows which differences are still possible.
• Reporting only stars or “sig”. Without the lift and its interval, nobody can judge whether the result matters for the business.
• Running many tests and reporting the one under 0.05. With twenty variants, one will pass by luck alone.