Level 3 · Is the lift real?

The question behind the test

Even if the banner did nothing, the two arms would not convert at exactly the same rate. Who lands in which arm is random, and with about 100 conversions per arm, luck alone moves the rates by tenths of a point. So the real question is: is the gap bigger than luck usually makes? Two tools answer it, and you want both in a write-up.

The confidence interval

The standard error (SE) of a difference between two rates is

se = math.sqrt(p_a * (1 - p_a) / n_a + p_b * (1 - p_b) / n_b)   # 0.0054

and a 95% confidence interval is the difference ± 1.96 standard errors:

diff - 1.96 * se, diff + 1.96 * se    # (-0.0078, 0.0132)

Read it as: values from −0.78 to +1.32 percentage points are all consistent with what we saw. The interval contains zero, so “no effect” is consistent with the data. It also contains +1 point, a big win, so “no effect” is not proven either. The test simply was not precise enough to tell.

The p-value

The z-test asks the same question from the other side. Assume there is no effect: then both arms share one conversion rate, the pooled rate (203 conversions out of 5,208 sessions). Measure how many standard errors apart the arms are under that assumption:

pooled = arms["conv"].sum() / (n_a + n_b)
se0 = math.sqrt(pooled * (1 - pooled) * (1 / n_a + 1 / n_b))
z = diff / se0                                  # 0.50
p = 2 * (1 - NormalDist().cdf(abs(z)))          # 0.62

“Two-sided” means a gap this large in either direction counts, because you would also want to know if the banner hurt. A p-value of 0.62 means: if the banner did nothing, a gap at least this big would happen 62% of the time. That is not surprising at all.

Saying it honestly

What p = 0.62 does not mean: a 62% chance that the banner does nothing, or proof that it has no effect. The honest sentence is: “The banner group converted 0.27 points better, but the 95% interval runs from −0.78 to +1.32 points (p = 0.62). We found no evidence the banner changes conversion.”

Common mistakes

• Using the pooled SE for the interval. The pooled SE assumes no difference, which is right for the test and wrong for an interval around the difference you observed.
• One-sided p-values chosen after seeing the result. Halving p because the lift came out positive is a decision you must make before the test, not after.
• “Not significant” as “no effect”. Absence of evidence is not evidence of absence, especially on a small test. How small this one was is the subject of the boss.