Level 6 · Boss: how big should the sample be?

The standard error

Last level, one sample of 200 orders came out $28 below the true mean, and other samples landed anywhere from $385 to $581. The standard error (SE) puts one number on that wobble: it is the standard deviation of the sample mean across all the samples you could draw.

sd = completed["order_value"].std()   # 795.75, spread of single orders
se = sd / math.sqrt(100)              # 79.57, spread of 100-order averages

The formula is SE = sd / √n. Two things set it: how spread out the values are (sd) and how many you sample (n). It falls with the square root of n, so four times the sample only halves the error.

Checking it by simulation

A formula is easier to trust once you have watched it work. With the whole population in memory you can draw many samples and look at their means directly:

rng = np.random.default_rng(42)
values = completed["order_value"].to_numpy()
means = pd.Series([rng.choice(values, size=100, replace=False).mean() for _ in range(1000)])
means.std()   # close to the formula's 79.57

The list of 1,000 means is the sampling distribution of the mean. Its centre sits on the true mean, and its spread is the standard error. It is also far more bell-shaped than the order values themselves (skew 0.38 against 3.95): averages of many values smooth out the tail. That is the central limit theorem at work, and it is why a 1.96 multiplier works for means even when the data is skewed.

From standard error to margin of error

About 95% of sample means fall within 1.96 standard errors of the true mean, so the 95% margin of error of a sample mean is 1.96 × SE. For 100 orders that is 1.96 × 79.57 ≈ ±$156. Flip the formula around to plan a sample:

n = math.ceil((1.96 * sd / 25) ** 2)   # orders needed for ±$25

Round up: a sample of 3,892.1 orders doesn’t exist, and rounding down would miss the target.

What the standard error doesn’t cover

The standard error measures random sampling error only. A sample picked the wrong way (last level’s email visitors) has a small standard error and a big bias. Use it with random samples.

Common mistakes

• Confusing sd and SE. The sd describes single values and does not shrink with more data; the SE describes an average and does.
• Expecting linear gains. Doubling the sample cuts the error by only 29%.
• Mixing ddof. pandas .std() divides by n − 1, numpy’s np.std by n. Pick one and say so; here everything uses pandas.
• Rounding n down. Sample sizes always round up.