Level 4 · How spread out are our orders?

Why spread matters

Two stores can have the same median order and very different businesses: one where every order is close to $270, one where orders range from $10 to $10,000. The middle tells you where the data sits; the spread tells you how much to trust that any single order looks like it. Forecasts, sample sizes and A/B tests (chapter 3) all depend on it.

Standard deviation

The standard deviation is roughly “the typical distance from the mean”. pandas computes it by squaring each value’s distance from the mean, averaging the squares (dividing by n − 1), and taking the square root:

v = completed["order_value"]
v.std()      # 795.75

pandas divides by n − 1 (the sample standard deviation) by default; NumPy’s np.std divides by n unless you pass ddof=1. On 10,849 orders the difference is invisible, on 10 values it isn’t.

Squaring makes big distances count a lot. One $11,000 order is $10,500 from the mean, and its square outweighs hundreds of ordinary orders. That’s why the standard deviation of order value ($795.75) is bigger than the median order ($268.94).

Interquartile range

The IQR is the width of the middle half of the data: P75 minus P25.

q1, q3 = v.quantile(0.25), v.quantile(0.75)   # 93.99, 659.87
iqr = q3 - q1                                  # 565.88

Like the median, it is robust: the biggest orders can grow without limit and the IQR does not move. For skewed data, report the median with the IQR (“median $269, middle half between $94 and $660”), and the mean with the standard deviation.

The 1.5 × IQR rule

A common way to flag unusual values is Tukey’s fences: anything above P75 + 1.5 × IQR or below P25 − 1.5 × IQR. These are the dots beyond the whiskers on a box plot.

fence = q3 + 1.5 * iqr          # 1508.69
(v > fence).sum()               # 927 orders

On order values the lower fence is negative, so nothing is flagged on that side. On the top, 927 orders are flagged; 70% of them come from business accounts, and furniture and outdoor sets make up most of their value. The rule found the tail of a skewed distribution, which is not the same as finding mistakes. Level 5 is about telling the two apart.

Common mistakes

• Rounding too early. Round the final numbers, not the quartiles you build the fence from.
• Reading “outlier” as “error”. The fence is a flag for a closer look. Deleting 8.5% of orders, including your largest customers, would make every revenue figure wrong.
• Using the standard deviation to describe skewed data. “Mean $543 ± $796” suggests negative orders are normal. Use the median and IQR there.