Sampling distribution and standard error explained with real data

The idea

Suppose you could not look at all 10,849 completed orders and wanted the average order value from a random sample of 100. You compute it and get, say, $478. Someone else draws their own 100 and gets $590. Both used the same method on the same store, and both are right about their sample. The average of a sample is a number that changes every time you draw one.

That wobble has a shape. If you drew the sample thousands of times and plotted the averages, you would see the picture below. Its centre is the true mean, and its width says how far a typical sample lands from the truth.

Figure 1

1,000 samples of 100 orders: where did their averages land?

Step 1 of 3 Draw 100 completed orders at random and compute their average. Do it again, and again, 1,000 times. Each draw gives a different average; this chart counts how many fell in each $50 band.

Draw 100 completed orders at random and compute their average. Do it again, and again, 1,000 times. Each draw gives a different average; this chart counts how many fell in each $50 band.
The averages pile up around the true mean of all orders, $543.30. Most draws are close to it, and a few land far away. Single orders are wildly skewed, but the averages form a rough bell.
About 95% of these sample averages fall between roughly $400 and $715. A single sample of 100 orders can easily be $100 or more away from the truth, and nothing in that one sample warns you.

The width of that picture is the standard error. It is the most useful idea in this chapter, because it turns "my number is probably close" into "my number is probably within this much".

Why it exists

An analyst almost never has all the data. Surveys cost money, a manual audit takes days, an experiment sees a few thousand visitors out of everyone who will ever visit. In all of these you compute a number from a sample and want to say something about the population behind it.

Without a way to measure the wobble, you have two bad options: report the sample number as if it were the truth, or refuse to say anything. The standard error gives a third: report the number together with how far it could plausibly be off, and know in advance how big a sample must be to make that small enough to act on.

How it works under the hood

Look at the shape of one column first

The distribution being sampled matters too. Plot Harbor & Pine's pages_viewed and 17,405 of the 40,000 sessions, 43.5%, are single-page visits. The mean is 3.66 pages and the median is 2. The histogram shows what neither number does: a huge spike at 1 and a long thin tail. A summary of a sample inherits that shape, so look at the histogram before trusting any single figure from it.

One sample, one average, one draw from a distribution

Each sample average is a single draw from the sampling distribution: the distribution of that statistic over all the samples you could have taken. You never see that distribution, only your one draw. But statistics lets you describe it from the sample itself.

The sampling distribution of an average has a useful property. Even when individual values are very skewed, as order values are, the averages of reasonably large samples form a bell shape centred on the true mean. That is the central limit theorem, and it is why one formula works for so many columns.

The standard error formula

The standard error of an average is the standard deviation of the column divided by the square root of the sample size:

se = sd / sqrt(n)

For order values, sd = $795.75 and n = 100, so se = $79.57. A simulation of 1,000 real samples gives a spread of averages very close to that. In practice you do not know the population sd, so you plug in the sample's own standard deviation, which is good enough once n is more than a few dozen.

A rough 95% margin is 1.96 standard errors. The $25 margin the finance team asks for needs 1.96 x 795.75 / sqrt(n) = 25, so n is about 3,893 orders.

Figure 2

Standard error of the average order value, by sample size

Step 1 of 3 The standard error is the standard deviation of the sample average: the typical distance between a sample's average and the truth. For order values it is $795.75 divided by the square root of n.

The standard error is the standard deviation of the sample average: the typical distance between a sample's average and the truth. For order values it is $795.75 divided by the square root of n.
Quadruple the sample, from 100 to 400 orders, and the error halves, from $79.57 to $39.79. It does not fall four times.
To reach $10 you need 6,400 orders. Every extra bit of precision costs more than the last, which is why sample size is a budget decision.

Precision is expensive

Because n is under a square root, halving the error costs four times the sample, and getting ten times more precise costs a hundred times more data. Before collecting anything, work out the error you can live with and solve for n. After the fact, the error is already decided and a report should state it.

Random error and bias are different problems

Everything above is about random error, the luck of which orders landed in your sample. It shrinks as n grows. Bias is a systematic miss caused by how the sample was picked, and it does not shrink at all. If you estimate site conversion only from email visitors, you get about 9.4%, while the true rate across all 40,000 sessions is 2.98%. A million email visitors would give the same wrong answer with a tiny standard error, which makes it look trustworthy.

The standard error tells you how precise a random sample is. It cannot tell you whether your sample was random. That part is a design question, answered before the data is collected.

Common misconceptions

"A bigger sample always fixes a wrong estimate"

A bigger sample fixes random error and leaves bias exactly where it was. The email-only estimate of conversion is 9.4% whether you take 500 sessions or 50,000. Ask first how the rows were chosen, and only then how many there are.

"The standard error is the same as the standard deviation"

The standard deviation describes single values: order values spread by $795.75. The standard error describes the average of n of them: $79.57 at n = 100. They answer different questions, and the second is always smaller. Quoting the standard deviation as the margin of error makes a sample look about ten times less precise than it is.

"If the data is skewed, I cannot use sample averages"

Skewed data means the individual values are skewed. Averages of a few dozen or hundreds of values are close to bell-shaped, so the standard error and a 95% margin still work. With very heavy tails or tiny samples, take more data or use a different statistic, but skew alone is not a reason to stop.

Check yourself

One question. Answering it ticks this lesson off on the map; it never locks a level.

An analyst wants the average order value within $25 of the truth, 95% of the time. A sample of 100 orders gives a standard error of $79.57. What happens if they survey 400 orders instead, still picked at random? The standard error drops to about $20, because four times the orders means a quarter of the error The standard error drops to about $40, half as large, because it shrinks with the square root of n The standard error stays near $80, because it depends on the spread of order values, not on how many you take The sample average becomes exactly the true mean, because 400 orders is a big enough sample