The request
From Jordan Reyes, Head of Marketing
“Before the holiday test we ran a smaller banner test in the spring, March and April 2025, same setup. Nobody ever analysed it properly. Can you write it up, with a clear verdict I can forward to Catherine?”
Your goal
Step 1 of 4. Set result to a DataFrame with one row per arm (A, then B) and columns arm, sessions (count), conversions (sessions with converted True) and rate (conversions / sessions, rounded to 4).
- Use the
web_sessionsDataFramenot done yet - Show exactly
armsessionsconversionsratenot done yet - 2 rows, sorted as askednot done yet
Step 2 of 4. Set result to a dict with "abs_lift" (rate B minus rate A), "rel_lift" (that difference divided by rate A), and "ci_low", "ci_high": the 95% interval for the difference, ±1.96 unpooled standard errors. Use unrounded rates; round all four to 4 decimals.
- Use the
web_sessionsDataFramenot done yet - Store a dict in
resultnot done yet
Step 3 of 4. Set result to a dict with "z", the two-proportion z statistic for B minus A using the pooled standard error (rounded to 3), and "p_value", its two-sided p-value (rounded to 4).
- Use the
web_sessionsDataFramenot done yet - Store a dict in
resultnot done yet
Step 4 of 4. Set result to a dict with "mde_rel", the minimum detectable relative lift at 80% power and a two-sided 5% level, (z_0.975 + z_0.80) * sqrt(2 * p_a * (1 - p_a) / n_a) / p_a with p_a and n_a from arm A (rounded to 4), and "verdict": "ship" if p < 0.05 and the lift is positive, "stop" if p < 0.05 and the lift is negative, otherwise "inconclusive". Compute the verdict from the data, do not type it.
- Use the
web_sessionsDataFramenot done yet - Store a dict in
resultnot done yet
1,068 sessions in each arm. The banner arm converted less often. Your job is to say whether that gap means anything.
Show the numbers
| Arm | Sessions that converted |
|---|---|
| A: no banner | 3.8% |
| B: banner | 3% |
Step 1 · your code
Loading the editor…
Reference solution
| session_id | customer_id | started_at | device | source |
|---|---|---|---|---|
| 1 | NaN | 2023-01-01 23:02:24 | mobile | paid |
| 2 | 18 | 2023-01-01 15:07:49 | desktop | paid |
| 3 | NaN | 2023-01-01 16:55:46 | desktop | direct |
| 4 | NaN | 2023-01-01 12:43:28 | mobile | organic |
Mote checks your result
Press Run. Each row that matches the goal becomes a firefly in your lantern.
Step 2 · your code
Loading the editor…
Reference solution
| session_id | customer_id | started_at | device | source |
|---|---|---|---|---|
| 1 | NaN | 2023-01-01 23:02:24 | mobile | paid |
| 2 | 18 | 2023-01-01 15:07:49 | desktop | paid |
| 3 | NaN | 2023-01-01 16:55:46 | desktop | direct |
| 4 | NaN | 2023-01-01 12:43:28 | mobile | organic |
Mote checks your result
Press Run. Each row that matches the goal becomes a firefly in your lantern.
Step 3 · your code
Loading the editor…
Reference solution
| session_id | customer_id | started_at | device | source |
|---|---|---|---|---|
| 1 | NaN | 2023-01-01 23:02:24 | mobile | paid |
| 2 | 18 | 2023-01-01 15:07:49 | desktop | paid |
| 3 | NaN | 2023-01-01 16:55:46 | desktop | direct |
| 4 | NaN | 2023-01-01 12:43:28 | mobile | organic |
Mote checks your result
Press Run. Each row that matches the goal becomes a firefly in your lantern.
Step 4 · your code
Loading the editor…
Reference solution
| session_id | customer_id | started_at | device | source |
|---|---|---|---|---|
| 1 | NaN | 2023-01-01 23:02:24 | mobile | paid |
| 2 | 18 | 2023-01-01 15:07:49 | desktop | paid |
| 3 | NaN | 2023-01-01 16:55:46 | desktop | direct |
| 4 | NaN | 2023-01-01 12:43:28 | mobile | organic |
Mote checks your result
Press Run. Each row that matches the goal becomes a firefly in your lantern.
What Jordan gets, and why
DataFrames
Loaded for you as pandas DataFrames, plus pd and np.
web_sessions40,000 rows · 7 columns
session_idintegercustomer_idintegerstarted_attimestampdevicevarcharsourcevarcharpages_viewedintegerconvertedboolean
| session_id | customer_id | started_at | device | source | pages_viewed | converted |
|---|---|---|---|---|---|---|
| 1 | NaN | 2023-01-01 23:02:24 | mobile | paid | 1 | false |
| 2 | 18 | 2023-01-01 15:07:49 | desktop | paid | 1 | false |
| 3 | NaN | 2023-01-01 16:55:46 | desktop | direct | 7 | false |
| 4 | NaN | 2023-01-01 12:43:28 | mobile | organic | 3 | false |
| 5 | NaN | 2023-01-01 13:06:21 | mobile | social | 2 | false |
customers2,018 rows · 9 columns
customer_idintegerfirst_namevarcharlast_namevarcharemailvarcharcityvarcharstatevarcharsignup_datedatesegmentvarcharreferral_sourcevarchar
| customer_id | first_name | last_name | city | state | signup_date | segment | referral_source | |
|---|---|---|---|---|---|---|---|---|
| 1 | Owen | Hudson | owen.hudson@icloud.com | San Francisco | CA | 2022-06-02 | consumer | search |
| 2 | Douglas | Mendoza | dmendoza69@gmail.com | Atlanta | GA | 2022-06-02 | consumer | NaN |
| 3 | Henry | Brooks | hbrooks@lakesidecafe.com | Los Angeles | CA | 2022-06-04 | business | NaN |
| 4 | Brian | Graham | bgraham@bluecedarhomes.com | Seattle | WA | 2022-06-06 | business | |
| 5 | Kyle | Jimenez | kjimenez27@yahoo.com | Indianapolis | IN | 2022-06-07 | consumer | NaN |
products120 rows · 8 columns
product_idintegernamevarcharcategoryvarcharsubcategoryvarcharunit_pricedecimalunit_costdecimallaunched_ondateis_activeboolean
| product_id | name | category | subcategory | unit_price | unit_cost | launched_on | is_active |
|---|---|---|---|---|---|---|---|
| 101 | Vale Velvet Loveseat | Furniture | Sofas | 1659.00 | NaN | 2023-09-11 | true |
| 102 | Alder Oak Coffee Table | Furniture | Tables | 919.00 | 503.50 | 2024-05-16 | true |
| 103 | Orchard Rattan Dining Chair | Furniture | Chairs | 789.00 | 429.87 | 2023-05-27 | true |
| 104 | Upland Oak Bookcase | Furniture | Storage | 429.00 | 214.70 | 2022-12-20 | true |
| 105 | Avery Leather Bed Frame | Furniture | Beds | 449.00 | 232.70 | 2021-08-31 | true |
orders12,000 rows · 7 columns
order_idintegercustomer_idintegerorder_datetimestampstatusvarcharchannelvarcharshipping_feedecimalcoupon_codevarchar
| order_id | customer_id | order_date | status | channel | shipping_fee | coupon_code |
|---|---|---|---|---|---|---|
| 1001 | 125 | 2023-01-01 20:51:19 | completed | web | 0.00 | NaN |
| 1002 | 156 | 2023-01-02 07:58:46 | completed | web | 0.00 | NaN |
| 1003 | 22 | 2023-01-02 10:25:27 | completed | marketplace | 4.99 | WELCOME10 |
| 1004 | 48 | 2023-01-02 11:21:51 | completed | marketplace | 4.99 | WELCOME10 |
| 1005 | 82 | 2023-01-02 11:43:36 | completed | web | 0.00 | WELCOME10 |
order_items30,338 rows · 6 columns
order_item_idintegerorder_idintegerproduct_idintegerquantityintegerunit_pricedecimaldiscount_pctdecimal
| order_item_id | order_id | product_id | quantity | unit_price | discount_pct |
|---|---|---|---|---|---|
| 1 | 1001 | 121 | 9 | 31.99 | 0.00 |
| 2 | 1001 | 154 | 3 | 11.99 | 0.00 |
| 3 | 1002 | 154 | 4 | 11.99 | 0.00 |
| 4 | 1002 | 194 | 1 | 149.00 | 0.00 |
| 5 | 1002 | 193 | 2 | 19.99 | 0.00 |
returns1,673 rows · 5 columns
return_idintegerorder_item_idintegerreturned_ondatereasonvarcharrefund_amountdecimal
| return_id | order_item_id | returned_on | reason | refund_amount |
|---|---|---|---|---|
| 1 | 14 | 2023-01-10 | damaged | 83.69 |
| 2 | 42 | 2023-01-10 | wrong_item | 19.99 |
| 3 | 105 | 2023-01-14 | wrong_item | 10.99 |
| 4 | 24 | 2023-01-16 | changed_mind | 26.99 |
| 5 | 126 | 2023-01-16 | late | 20.99 |
employees60 rows · 8 columns
employee_idintegerfirst_namevarcharlast_namevarchardepartmentvarchartitlevarcharmanager_idintegerhire_datedatesalaryinteger
| employee_id | first_name | last_name | department | title | manager_id | hire_date | salary |
|---|---|---|---|---|---|---|---|
| 1 | Catherine | Hale | Executive | Chief Executive Officer | NaN | 2016-03-01 | 285000 |
| 2 | Robert | Kim | Executive | Chief Operating Officer | 1 | 2017-01-09 | 235000 |
| 3 | Megan | Collins | Sales | Head of Sales | 1 | 2018-04-16 | 172000 |
| 4 | Chloe | Gutierrez | Sales | Account Executive | 3 | 2020-01-21 | 72000 |
| 5 | Frank | Bell | Sales | Account Executive | 3 | 2023-05-21 | 73000 |
marketing_spend3,276 rows · 3 columns
spend_datedatechannelvarcharamountdecimal
| spend_date | channel | amount |
|---|---|---|
| 2023-01-01 | paid | 492.99 |
| 2023-01-01 | 37.38 | |
| 2023-01-01 | social | 200.53 |
| 2023-01-02 | paid | 709.13 |
| 2023-01-02 | 64.13 |
The lesson
The four numbers of a write-up
A test write-up answers four questions, in this order:
- Was the test healthy? The arm table: sessions per arm (the sample ratio check) and conversions.
- What happened? The lift, absolute and relative.
- How sure are we? The 95% confidence interval and the p-value.
- What could the test have seen? The minimum detectable effect, and the verdict.
You built the first three for the holiday test. The spring test is smaller: 1,068 sessions per arm against 2,604, which is exactly why the fourth question matters.
Minimum detectable effect
Power is the chance a test detects an effect that is really there. The minimum detectable effect (MDE) is the smallest effect the test detects with a chosen power, usually 80%, at the usual 5% level:
nd = NormalDist()
mde_abs = (nd.inv_cdf(0.975) + nd.inv_cdf(0.8)) * math.sqrt(2 * p_a * (1 - p_a) / n_a)
mde_rel = mde_abs / p_a # about 0.61On the spring sample, only a change of about 61% in conversion had a fair chance of showing up. Banners, button colours and copy changes typically move conversion by a few percent. This test was too small for its question before it started, and it is better to say so than to let a flat result read as “the banner does nothing”.
The same formula answers the planning question in reverse: fix the lift you care about and solve for the sessions you need. That is how you avoid running the next underpowered test.
Writing the verdict
Fix the decision rule before you look: ship on a significant gain, stop on a significant loss, otherwise call it inconclusive. Then the verdict is computed, not argued, and the write-up for Jordan fits in three sentences:
“The banner arm converted 3.00% versus 3.84% (−0.84 points, −22%); the 95% interval runs from −2.38 to +0.70 points, p = 0.28. Verdict: inconclusive. With 1,068 sessions per arm the test could only detect changes of about 61%, so it cannot rule out a typical banner effect either way.”
Common mistakes
- Leading with the relative lift. “−22%” alone sounds like a disaster. With its interval it is a shrug.
- Treating “inconclusive” as “no effect”. An underpowered test that finds nothing has learned very little. Say how big an effect it could have found.
- Typing the verdict. If the rule is written in code, anyone can rerun it and get the same call, even when they would have preferred a different one.
- Reusing the holiday conclusion. Season, traffic mix and sample size differ. Each test gets its own write-up.
Lantern lit!
Practised: ab testconfidence intervalp valueminimum detectable effectwrite up
- Solved
- No hints
- First run