Level 4 · The email-only estimate

Random error and bias

A sample can be wrong in two different ways.

• Random error (sampling error) comes from chance: which rows happened to be drawn. It goes up and down from sample to sample, averages out, and shrinks as the sample grows.
• Bias comes from how the sample was chosen. If the method favours some kinds of rows, every sample leans the same way, and a bigger sample only makes the wrong answer look more precise.

Statistics formulas (standard errors, confidence intervals, p-values) measure random error only. None of them can detect bias.

A store example

Conversion by traffic source on Harbor & Pine:

web_sessions.groupby("source")["converted"].mean().round(3)

direct     0.050
email      0.094
organic    0.018
paid       0.016
social     0.006

Overall: 3.0%. Email traffic converts at more than three times the site rate, and the reason is who they are: 89% of email sessions belong to signed-in customers, against 35% for organic and 29% for social. Estimating “the share of visitors who buy” from email sessions measures the store’s most loyal customers and calls it everyone.

Where bias hides

• Convenience samples: the data that was easiest to collect (one channel, one region, the customers with an email address).
• Self-selection: surveys answered by the people who chose to answer, often the happiest and the angriest.
• Survivorship: looking only at customers who are still active, products still on sale, or orders that weren’t cancelled.
• Timing: a week in December says little about March.

How to get a fair sample

Draw it at random from the whole population (df.sample(n=..., random_state=...) on the full table), or measure everyone when you can. If you can only reach one group, report the number for that group under its real name: “email visitors convert at 9.4%” is true and useful.

Common mistakes

• Answering bias with more data. More rows from the same biased source give the same wrong answer with a smaller margin.
• Trusting the margin of error. A confidence interval on a biased sample is precise about the wrong number.
• Generalising a segment’s number. Name the group a number describes, every time.