Level 5 · Write the conclusion

Three parts of a conclusion

The person reading your write-up will not read your code. They read one paragraph and decide. A paragraph they can act on has three parts:

• What: the main result in plain words. “The banner showed no measurable effect on conversion.”
• How much: the number that supports it, computed, not eyeballed. Both rates, the sample size, the interval, the p-value.
• So what: what it means for the decision. “We will not roll it out; if free shipping matters, we rerun it with more traffic.”

Without how much it is an opinion. Without so what it is a caption.

Every number comes from code

Write the paragraph after the numbers, from the numbers. Put the cell that computes them right above the text, so anyone can rerun it and get the same figures:

rate_a, rate_b = a["converted"].mean(), b["converted"].mean()
print(f"{rate_a:.2%} vs {rate_b:.2%}, diff {rate_b - rate_a:+.2%}, "
      f"95% CI {low:+.2%} to {high:+.2%}, p = {p:.2f}")

Rounded figures in the text (“4.03% vs 3.76%”) should match the printed output exactly. A number typed from memory is the one that ends up wrong in the slide deck.

Words that match the evidence

The wording carries as much weight as the numbers:

• A difference with p = 0.62 is “no measurable effect”, not “the banner worked” and not “the banner does nothing”. The test was too small to see an effect under about 40%.
• A correlation is “associated with”, never “causes” (chapter 4). Only a randomized test like this one can support a causal claim, and only when the result is clear.
• Say what the test could not show. That one sentence stops the next meeting from rerunning the same argument.

Decisions and recommendations

When you made a choice during the analysis, write it down in three parts: what you decided, why, and what it changes. “We excluded nothing: both arms had exactly 2,604 sessions, so the split was clean.”

A recommendation is an action verb, a concrete place and the number behind it: “Rerun the banner test in spring with about 50,000 sessions per arm, enough to detect a 10% lift.” Compare that with “consider testing more”, which nobody can schedule.

Common mistakes

• Celebrating a non-significant gap. +7.1% sounds like news; with p = 0.62 it is noise-sized.
• Turning “not significant” into “proven useless”. Absence of evidence from a small test is not evidence of absence.
• Describing the chart instead of concluding. “B is a bit higher” leaves the reader to do the analysis.
• Generalising past the test. One banner, on one site, in one season, says little about every free-shipping campaign.