The idea
A column of 10,000 numbers is too much to read, so we summarise it. Every summary answers one question. Where is the middle? is answered by the mean or the median. How spread out is it? is answered by the standard deviation or the IQR. What lies far from the rest? is answered by outlier rules. None of these questions has one correct number, and the answers change when the column is lopsided.
Harbor & Pine's order values are lopsided. Most orders are small and a few are very large. The mean and the median, which look like two spellings of "average", land at $543.30 and $268.94.
Figure 1
Completed orders by order value, in $250 bands
- Harbor & Pine's completed orders, grouped by what the customer paid for the items. Nearly half of all orders are under $250, and a long tail runs out past $2,000.
- The median is $268.94: sort every order by value and the one in the middle sits here. Half the orders are smaller, half are bigger.
- The mean is $543.30, double the median, because it adds every dollar. The 523 orders above $2,000 drag it up, so only 3 orders in 10 are as large as the mean.
Someone who quotes only one of them is making a choice, whether they know it or not.
Why it exists
Imagine reporting "average order value $543" to the CEO. She hears: a customer typically spends about $543. But half the orders are under $269, and only 3 in 10 reach $543. The number is correct, and the sentence she builds from it is wrong. A customer who spends $543 is a bigger spender than most of the people on the list.
Summary statistics exist so a reader can hold a column in their head. The craft is choosing the summary whose meaning matches the sentence you want the reader to take away. "Total revenue divided by number of orders" is a fine definition of the mean and the right number for forecasting revenue. "What a typical customer spends" is a question about the middle of the pile, which is the median.
How it works under the hood
The mean uses every value, the median uses only the order
The mean adds all the values and divides by how many there are, so a single $11,103.27 order moves it as much as thousands of small ones would. The median sorts the values and picks the middle one, so it cares about the ranking and not the size of the extremes. Make the biggest order ten times larger and the mean jumps by about $9 while the median does not move at all. That sounds small until the tail holds a few hundred such orders.
When a column is right-skewed, with a long tail of large values, the mean sits above the median. Money, waiting times, page views and file sizes are all like this. When the two agree, the column is roughly symmetric and either one will do.
Percentiles read the column by rank
The p-th percentile is the value below which p% of the rows fall. The median is the 50th. Q1 and Q3 are the 25th and 75th. Percentiles are how you set thresholds: if the 90th percentile of orders is $1,345.77, a free-shipping offer set there reaches the top tenth and no one else. They are also robust, since moving the maximum from $11,103.27 to a million does not change the 90th percentile at all.
Spread: standard deviation versus IQR
The standard deviation is the typical distance of a value from the mean. Order values have a standard deviation of $795.75, bigger than the mean itself, which is itself a sign of heavy skew. Because it squares the distances, the biggest orders dominate it, just as they dominate the mean. The IQR (Q3 minus Q1, here $565.88) measures the spread of the middle half and ignores both tails. Use the standard deviation with the mean, the IQR with the median, and do not mix the pairs.
To compare the steadiness of two groups with different averages, divide the standard deviation by the mean. This ratio, the coefficient of variation, is 1.51 for web orders and 1.39 for app orders: web order values are the more erratic relative to what a web order is worth.
Figure 2
The same orders as five landmarks, in dollars
- Percentiles read the column by rank. A quarter of orders are under Q1 = $93.99, three quarters under Q3 = $659.87. The distance between them, $565.88, is the IQR: the spread of the middle half.
- The common outlier rule draws a fence at Q3 plus 1.5 times the IQR: $1,508.69. It flags 927 orders. That is a rule about distance from the middle, not a verdict on whether those orders are real.
- The largest order is $11,103.27. It is far past the fence and still a legitimate business order. Whether a flagged value is an error is a question for the source, not the fence.
An outlier is a question, not an answer
An outlier rule only finds values that are far from the middle. It cannot tell you why. Three things can sit behind a flagged value: a legitimate rare event (a company buying in bulk), a data error (a quantity of 900 typed instead of 9), or a different population (a wholesale account mixed in with consumers). The first stays in. The second gets fixed or removed. The third gets analysed separately. Look at the flagged rows, then decide, and write the decision down.
Common misconceptions
"The average is the typical value"
The mean is the balance point of the column, not its most common or middle value. On skewed data it can describe almost no real row. Report the median for "typical", the mean for "total divided by count", and say which one you used.
"The standard deviation tells me how far values are from the mean, so a bigger one means more outliers"
The standard deviation grows with the overall scale of the data and with every skewed value. Two columns can have the same number of outliers and very different standard deviations just because one is measured in dollars and one in cents. Compare spread across groups with the coefficient of variation, or look at the IQR and the plot.
"Anything past 1.5 times the IQR is an error and should be removed"
The rule is a flag to inspect, not a deletion order. Here it flags 927 completed orders, and the largest of them is a real $11,103.27 purchase. Deleting it changes total revenue and the mean for the wrong reason. Remove a value only when you can show it is wrong.
Check yourself
One question. Answering it ticks this lesson off on the map; it never locks a level.
Not quite. Using every order is exactly why the mean is dragged up by a few big ones. 'Accurate' depends on the question: for a typical order the median is the honest answer.Try another answer.
Right. Half of all orders are below $268.94, so it describes a typical customer. The mean still belongs in the report because it is what total revenue divides into.
Not quite. Averaging two summaries gives a number that describes no order and no customer. Pick the statistic that answers the question.Try another answer.
Not quite. The fence flags 927 orders, but flagged is not wrong. Deleting real business orders would also delete real revenue from the total.Try another answer.
Lesson done: it is ticked on the map.