Level 2 · Conversion per arm and the lift

Conversion rate is a mean

A session either converted or it didn’t. Store that as True/False and the conversion rate is simply the mean of the column, because pandas counts True as 1 and False as 0:

test.groupby("arm")["converted"].mean()
# A    0.0376
# B    0.0403

Arm B, the banner, converted 105 of 2,604 sessions; arm A converted 98 of 2,604. It is worth printing the counts next to the rates (.agg(["size", "sum", "mean"])), because 105 versus 98 is a much more honest picture of the evidence than 4.03% versus 3.76%.

Absolute and relative lift

There are two ways to say how much better B did, and people mix them up constantly:

• Absolute lift: rate_b - rate_a = 0.0027, or 0.27 percentage points.
• Relative lift: (rate_b - rate_a) / rate_a = 0.071, or 7.1%.

The relative number sounds more impressive, which is why it ends up in headlines. Both are correct; what matters is saying which one you mean. “Conversion rose 7%” read by someone who thinks in points sounds like 3.8% became 10.8%, which is not what happened.

Use points (pp) for differences of rates, and percent for relative change. Write it as “+0.27 pp (+7.1% relative)” and nobody can misread it.

Seven percent of what?

Seven extra orders in two months is the real-world size of this difference. With conversions this rare, a handful of sessions moves the rate a lot: if eight of B’s 105 buyers had left without ordering, B would be the losing arm. That is the question the next level answers properly: how much of this gap could be noise?

Common mistakes

• Rounding before dividing. Round 0.037634 and 0.040323 to 0.0376 and 0.0403 first and the relative lift comes out 0.0718 instead of 0.0714. Compute with full precision, round at the end.
• Dividing by the wrong arm. Relative lift is measured against the control (A). Dividing by B gives a different number that answers a different question.
• Counting rows instead of sessions with a conversion. count() counts non-missing values, so it counts every session. Use sum() for conversions, mean() for the rate.
• Declaring a winner from the lift alone. Every test has a winner if you only compare two numbers. The lift is the estimate; whether it is real is a separate question.