The idea
Marketing ran the free-shipping banner on half of Harbor & Pine's visitors in November and December 2025 and wants to know if it works. The answer is a short chain of questions, and the order matters because each one only makes sense if the one before it was answered well.
Figure 1
The order of an A/B test analysis
-
check the split2,604 vs 2,604 sessions -
measure the lift3.76% vs 4.03% -
interval and p-value-0.78 to +1.32 pts, p = 0.62 -
what could it detect?about 1.5 pts -
write the verdictno proof either way
- First check the split. The banner test ran in November and December 2025 and 5,208 sessions took part, 2,604 in each arm. If one arm had been half the size, something broke the random assignment and no later number can be trusted.
- Then measure each arm. No banner (A): 98 orders from 2,604 sessions, 3.76%. Banner (B): 105 from 2,604, 4.03%. The lift is 0.27 percentage points, or 7.1% relative.
- Then ask how much of that is luck. The 95% interval for the difference runs from -0.78 to +1.32 points, and a z-test gives p = 0.62. A gap this size is entirely ordinary when nothing is going on.
- Then ask what the test could have seen. With 2,604 sessions per arm and a 3.9% base rate, the smallest gap it reliably detects is about 1.5 points, a 40% relative lift. A 7% lift was out of reach.
- The verdict states both facts: the test shows no evidence the banner works, and it was too small to show a modest effect either way. That is not 'the banner does nothing'.
The chain turns a pile of sessions into a verdict a person can act on. Skipping a link does not save time. It produces a number that looks like an answer.
Harbor & Pine's test is a good one to learn on, because the honest answer is a little deflating: the banner neither clearly works nor clearly fails. A write-up that can say that plainly is more useful to a marketing lead than one that forces a yes or a no.
Why it exists
A randomised A/B test is the strongest design an analyst has for cause and effect, because the only systematic difference between the arms is the thing you changed. That claim has conditions: the visitors were split fairly, the sample was big enough, and nobody picked a favourite result after looking.
A write-up is how those conditions get checked and recorded. A marketer reading "conversion rose 7%" will act on it. A good analyst's job is to make sure that number arrives with its uncertainty, its limits and a clear recommendation.
How it works under the hood
1. Check the split before the result
The assignment here is by session id: even ids saw the banner. If the split is healthy, both arms have about the same number of sessions, and they look alike in device mix and traffic source. An unexpected imbalance, called a sample ratio mismatch, means the assignment or the logging broke, and the comparison is no longer between like groups. Test the counts against the intended 50/50 split before looking at conversion at all. Here it is 2,604 and 2,604.
2. Measure conversion and the lift
Conversion per arm is orders divided by sessions: 3.76% for A and 4.03% for B. The absolute lift is the difference, 0.27 points. The relative lift divides that by the control rate, 7.1%. Quote both. Relative lifts sound big on a small base, and absolute lifts say what the change is worth in orders.
3. Put uncertainty on it
The 95% interval for the difference is -0.78 to +1.32 points. It includes zero, so the data cannot say whether the banner helps, does nothing, or hurts a little. The two-proportion z-test agrees: z = 0.50 and p = 0.62. Under "no difference", a gap this large would show up in about 62% of repeated tests.
4. Know what the test could detect
The minimum detectable effect is the smallest true difference a test of this size would catch reliably, with 80% power at the 5% level. For 2,604 sessions per arm around a 3.9% rate it is about 1.5 percentage points, roughly a 40% relative lift. The banner's observed 7% lift is far below that, so the test was too small to settle a modest effect either way. That sentence belongs in the write-up.
5. Resist the slices
When the headline is disappointing, someone slices by device, source, country, until something looks good. Every slice is another chance for luck. Here is the same test by device.
Figure 2
Conversion by device and arm during the banner test (%)
- Slice the same test by device. Desktop shows B slightly lower, 4.67% against 4.75%, and mobile shows B slightly higher, 3.51% against 3.35%. Both are small and consistent with noise.
- Tablet looks exciting: 5.36% with the banner against 3.27% without, a 2-point jump. But that is 12 orders from 224 sessions against 9 from 275.
- Its own test gives p = 0.25. Even taken alone the tablet gap is not significant, and it was one of three slices checked, so the chance that at least one slice crosses p < 0.05 by luck is about 14%.
Three device slices at the 5% level give about a 14% chance, 1 minus 0.95 cubed, that at least one crosses p < 0.05 with no real effect. A slice is worth following only if it was planned before the data arrived or the result is replicated in a new test. Otherwise, report it as a lead for a future test, not as a finding.
Before the verdict, one more habit: write down the decision rule in advance. Say what lift would make the banner worth keeping, how long the test runs and when you will look. A rule fixed before the data arrives is what makes the p-value and the interval mean what they claim, and it makes the later conversation about the result a good deal calmer.
6. Write the verdict
A good verdict has four parts: the result with its interval, what the test could and could not detect, the decision it supports, and the next step. For this test: the banner showed a 0.27-point lift with a 95% interval from -0.78 to +1.32 points; the test could only detect gaps of about 1.5 points; there is no evidence the banner works and none that it does not; to settle it, run a longer test sized for the lift that would justify keeping the banner.
Common misconceptions
"p above 0.05 means the banner does nothing"
It means the data were not enough to tell. Absence of evidence is not evidence of absence, and an under-sized test produces large p-values for real effects. Read the interval, and state the smallest effect the test could have caught.
"If the overall test is flat, the segment that won is the real result"
The tablet slice jumped from 3.27% to 5.36%, and its own p-value is 0.25. Cutting a flat result into pieces and picking the best one is a way of finding noise. Treat it as a hypothesis for a new test.
"A bigger lift in the sample means a bigger lift in reality"
A lift measured on a few thousand sessions has a wide interval, and the extreme estimates are the ones most likely to shrink on a repeat. The honest summary of 7% is "anything from a small loss to a sizeable gain", not "7%".
Check yourself
One question. Answering it ticks this lesson off on the map; it never locks a level.
Not quite. A large p-value means the data did not separate the arms. It does not show they are equal; the interval for the difference runs from -0.78 to +1.32 points.Try another answer.
Right. The interval includes zero and the test could not detect a gap smaller than about 1.5 points. A modest benefit and a modest harm both fit the data.
Not quite. A positive lift in the sample is what chance produces half the time. The p-value of 0.62 says this one is entirely ordinary.Try another answer.
Not quite. Repeating a test until it passes is peeking by another name. It inflates false positives and proves nothing about the banner.Try another answer.
Lesson done: it is ticked on the map.