Chapter 3 · 6 levels
Comparing groups and A/B tests
Conversion rates, lift, confidence intervals, p-values explained honestly, the two-proportion z-test, peeking, and how much traffic a test really needs, on Harbor & Pine's web sessions.
Start“Did the new page work?” is the question analysts hear most after “how much did we sell?”. Answering it means comparing two groups, and comparing groups means deciding how much of a difference is real and how much is noise.
This chapter works on web_sessions: 40,000 visits to Harbor & Pine’s site, each with a device, a
traffic source and whether it ended in an order (converted). You start with the plain numbers,
conversion rate and lift, then put an interval around a rate so you know how sure it is. You learn
what a p-value says and what it does not, and run a z-test on a split where you know the true answer
in advance: two halves of the same traffic, an A/A test.
Then the traps: checking a test every few days and stopping when it looks good, and starting a test that could never detect the effect you care about. The boss plans a real test for the marketing team and finds out what the store’s traffic can and cannot prove.
Levels
- Desktop versus mobile conversion
from Daniel Foster, Mobile Product Manager
conversion rate, lift, relative difference, proportions
- How sure is the mobile rate?
from Daniel Foster, Mobile Product Manager
confidence interval, standard error, proportions, margin of error
- What p = 0.03 actually says
from Lauren Hayes, Email Marketing Manager
p value, null hypothesis, statistical significance, effect size
- Test the test with an A/A split
from Priya Shah, Head of Data
z test, a a test, p value, pooled proportion, standard error
- The test that won by peeking
from Priya Shah, Head of Data
peeking, false positive, stopping rules, a a test
- Boss: plan the banner test
from Jordan Reyes, Head of Marketing
sample size, power, minimum detectable effect, baseline rate, test duration