Boss: the promo test write-up

Marketing ran a test. For the last two months of 2025, half of the visitors to Harbor & Pine’s site saw a banner promising free shipping, and half did not. Now Jordan Reyes, Head of Marketing, wants to know whether the banner works, and the answer will decide whether it goes on the site for good.

This chapter is the job from start to finish, the way a senior analyst does it. First you check that the split itself is healthy, because a broken split makes every later number meaningless. Then you measure conversion in each arm and the lift, put a confidence interval and a p-value on the difference, and look at what happens when someone starts slicing the results by device until something looks exciting.

The boss is a second test, run in the spring, that you analyse alone and finish with a verdict and a sentence on what the test could and could not have detected. That last part is what separates a write-up people can act on from a table of numbers.

Every code level gives you test: the sessions inside the test window, with an arm column ("A" = no banner, "B" = banner). Sessions with an even session_id saw the banner.