What a p-value and a confidence interval actually say

The idea

You change something on the site and conversion goes from 2.41% to 2.9%. Was it the change, or just which visitors happened to come that week? Every group comparison has this problem: the visitors in two groups differ by chance, even when you treat them the same.

Two tools answer it. A confidence interval says what range of true values is plausible given the data. A p-value says how surprising the observed gap would be if the true gap were zero. They come from the same calculation and carry the same information from two angles.

Harbor & Pine's web_sessions has 40,000 visits. Desktop converts at 3.87% and mobile at 2.41%. The intervals show how sure we can be of each number, and the gap between them is the evidence.

Why it exists

Without these tools, a team has two ways to decide. They can trust any visible difference, which means chasing noise and shipping changes that do nothing. Or they can trust only differences that "look big", which depends on who is looking. A p-value and an interval make the question precise: how large is the gap, and how much of that could be luck?

The practical payoff is a better conversation with the people who own the decision. Instead of "mobile looks worse", you can say "mobile converts between 2.21% and 2.62%, desktop between 3.56% and 4.19%, and the gap is about 1.5 points". A reader can act on that, and challenge it on its terms.

They also give a stopping point. If the interval for the lift runs from -0.8 to +1.3 points, the honest report is "we do not know the sign yet", and nobody should pretend otherwise.

How it works under the hood

Confidence intervals

For a conversion rate p measured on n sessions, the standard error is sqrt(p(1-p)/n). The 95% interval is the rate plus or minus 1.96 of those. Mobile: p = 2.41%, n = 21,549, standard error 0.10 points, interval 2.21% to 2.62%.

The correct reading is about the method: if you repeated the whole study many times and built an interval each time, 95% of those intervals would contain the true rate. Any one interval either contains it or does not. "There is a 95% chance the true rate is in this range" is a common shorthand, but it is a loose one.

The two-proportion z-test

To compare two groups, assume for a moment that they share one true rate, and measure how unusual the observed gap is. That is the procedure in the diagram.

Using 1.96 as the cut-off for a two-sided test at the 5% level, any |z| above 1.96 gives p below 0.05. Here z is almost 8, so the gap between desktop and mobile is not luck.

What p = 0.03 says, and what it does not

The p-value is a probability about the data, given that the null is true: "if there were no real difference, how often would a gap this large or larger appear?" It is not the probability that the null is true, not the probability the result is a fluke, and not a measure of effect size. A huge sample can give a tiny p for a lift too small to matter, and a small sample can give a big p for a lift that is real and valuable. Always read the p-value together with the size of the gap and its interval.

A/A tests, to learn what luck looks like

Split the same traffic in two with no change at all: sessions with an even id against an odd id. Even converts at 3.08% and odd at 2.89%, a gap of 0.2 points, z = 1.15, p = 0.25. Nothing is going on, and still the two numbers differ. That is the noise you have to beat.

Peeking and sample size

A p-value is only valid if you decide the sample size in advance and look once. Check a test every week and stop at the first p below 0.05, and every look gives chance another turn. Replaying Harbor & Pine's A/A split on 2024 with a weekly check, the arms crossed p < 0.05 on February 4 (B at 3.32% against A at 1.18%, p = 0.037). By March 31 the gap had gone, p = 0.69. A false winner.

The other trap is a test too small to find anything. To detect a 10% relative lift on a 2.98% base rate (2.98% to 3.28%) with 80% power at the 5% level, each arm needs about 53,500 sessions. The whole data set holds 40,000 sessions, all years together. That test cannot answer the question, no matter how carefully it is analysed.

Common misconceptions

"p = 0.03 means there is a 97% chance my change works"

The p-value assumes the change does nothing and asks how surprising the data is. It cannot give the probability the change works. What you can say is that the data is hard to explain by chance alone and show the interval for the size of the effect.

"No significant difference means the two versions are the same"

A large p-value means the data was not enough to separate them, not that they are equal. A test with 2,604 sessions per arm can only reliably detect a gap of about 1.5 points; anything smaller is invisible. Report the interval: "between -0.8 and +1.3 points" is a very different message from "no effect".

"Non-overlapping intervals are required for significance"

The reverse is the trap. Non-overlapping intervals imply a significant gap, but two intervals can overlap a little and the difference can still be significant, because the test works on the standard error of the gap itself. Run the test; do not eyeball the overlap.