Level 3 · Recommend: keep, change or test the coupons?

From numbers to a recommendation

Laura needs one paragraph that both Finance and Marketing can accept. The structure is the same as in any recommendation, what, how much, so what:

• What is the main finding? Coupon customers paid less for baskets that were about the same size as everybody else’s on the same days. The codes did not buy bigger orders.
• How much supports it? The cost: $121,785 of discounts over three years, on orders that brought 12.8% of net revenue. The fair comparison: $402 against $483 per order on Black Friday weekend, $571 against $658 in the summer sale.
• So what does it mean for the budget? Do not raise it on this evidence. Test, then decide.

Caveats belong in the paragraph

The biggest one is the counterfactual. A coupon pays for itself only through orders that would not have happened without it, and nothing in the data shows which those are. Two others are specific to this case:

• Selection. WELCOME10 only reaches first orders, so it is compared with nothing like it. Its AOV of $453.35 says as much about new customers as about the code.
• Missing cost. FREESHIP gives away shipping, not a percentage of the basket, so its cost is not in the discount totals.

Turning the open question into a test

When the data cannot answer a question, propose the cheapest way to get the answer. Here it is a hold-out test: send SPRING15 to a random half of the mailing list and keep the other half as a control. Because the halves are chosen at random, any difference in orders per customer between them is the code’s effect. After one spring you would know whether SPRING15 brings extra orders, and how many it would need to cover its discount.

The traps in the other options

• Revenue as return. “$751,822 of revenue for $121,785 of discounts” counts every coupon order as created by the coupon. That is the counterfactual problem dressed up as a ratio.
• The wrong comparison, plus cause. Comparing coupon AOV with the whole year’s AOV mixes the code with the season; then “bargain hunters” reads a cause into it.
• Over-reading small gaps. Coupon baskets were a little bigger before the discount on two codes. On a few hundred orders, gaps that size are not a reason to make every discount deeper.

The report structure behind this paragraph

In a written report this paragraph is the executive summary. Below it come the objective (the SMART question from the first level), the data and its limits (the code and discount check, FREESHIP, WELCOME10), the method (same-day comparison), the findings with the chart, and the next step: the hold-out test, with a date to report back.