The request
From Jordan Reyes, Head of Marketing
“Next year we want to test a new homepage banner against the current one, and we'd call it a win at a 10% lift in conversion. Can you tell me how many visitors we need and how long the test has to run? I was hoping for two weeks.”
Your goal
Step 1 of 3. Set result to a dict with "baseline", the 2025 conversion rate of all sessions rounded to 4 decimals, and "daily_sessions", the number of 2025 sessions divided by 365, rounded to 1 decimal.
- Use the
web_sessionsDataFramenot done yet - Store a dict in
resultnot done yet
Step 2 of 3. Set result to the sessions each arm needs (an int, rounded up with math.ceil) to detect a lift from p0 to p1 = p0 × 1.10 at two-sided alpha 0.05 and power 0.80: n = (z_alpha + z_beta)² × (p0(1−p0) + p1(1−p1)) / (p1 − p0)², with z_alpha = NormalDist().inv_cdf(0.975) and z_beta = NormalDist().inv_cdf(0.80). Use the unrounded p0 from the starter.
- Use the
web_sessionsDataFramenot done yet - Store a number in
resultnot done yet
Step 3 of 3. Set result to a dict with "days", the days needed to collect both arms (2 × the per-arm n from step 2, divided by 2025's average daily sessions, rounded up with math.ceil), and "years", days / 365 rounded to 1 decimal. Use unrounded values until the end.
- Use the
web_sessionsDataFramenot done yet - Store a dict in
resultnot done yet
Step 1 · your code
Loading the editor…
Reference solution
| session_id | customer_id | started_at | device | source |
|---|---|---|---|---|
| 1 | NaN | 2023-01-01 23:02:24 | mobile | paid |
| 2 | 18 | 2023-01-01 15:07:49 | desktop | paid |
| 3 | NaN | 2023-01-01 16:55:46 | desktop | direct |
| 4 | NaN | 2023-01-01 12:43:28 | mobile | organic |
Mote checks your result
Press Run. Each row that matches the goal becomes a firefly in your lantern.
Step 2 · your code
Loading the editor…
Reference solution
| session_id | customer_id | started_at | device | source |
|---|---|---|---|---|
| 1 | NaN | 2023-01-01 23:02:24 | mobile | paid |
| 2 | 18 | 2023-01-01 15:07:49 | desktop | paid |
| 3 | NaN | 2023-01-01 16:55:46 | desktop | direct |
| 4 | NaN | 2023-01-01 12:43:28 | mobile | organic |
Mote checks your result
Press Run. Each row that matches the goal becomes a firefly in your lantern.
Step 3 · your code
Loading the editor…
Reference solution
| session_id | customer_id | started_at | device | source |
|---|---|---|---|---|
| 1 | NaN | 2023-01-01 23:02:24 | mobile | paid |
| 2 | 18 | 2023-01-01 15:07:49 | desktop | paid |
| 3 | NaN | 2023-01-01 16:55:46 | desktop | direct |
| 4 | NaN | 2023-01-01 12:43:28 | mobile | organic |
Mote checks your result
Press Run. Each row that matches the goal becomes a firefly in your lantern.
What Jordan gets, and why
DataFrames
Loaded for you as pandas DataFrames, plus pd and np.
web_sessions40,000 rows · 7 columns
session_idintegercustomer_idintegerstarted_attimestampdevicevarcharsourcevarcharpages_viewedintegerconvertedboolean
| session_id | customer_id | started_at | device | source | pages_viewed | converted |
|---|---|---|---|---|---|---|
| 1 | NaN | 2023-01-01 23:02:24 | mobile | paid | 1 | false |
| 2 | 18 | 2023-01-01 15:07:49 | desktop | paid | 1 | false |
| 3 | NaN | 2023-01-01 16:55:46 | desktop | direct | 7 | false |
| 4 | NaN | 2023-01-01 12:43:28 | mobile | organic | 3 | false |
| 5 | NaN | 2023-01-01 13:06:21 | mobile | social | 2 | false |
customers2,018 rows · 9 columns
customer_idintegerfirst_namevarcharlast_namevarcharemailvarcharcityvarcharstatevarcharsignup_datedatesegmentvarcharreferral_sourcevarchar
| customer_id | first_name | last_name | city | state | signup_date | segment | referral_source | |
|---|---|---|---|---|---|---|---|---|
| 1 | Owen | Hudson | owen.hudson@icloud.com | San Francisco | CA | 2022-06-02 | consumer | search |
| 2 | Douglas | Mendoza | dmendoza69@gmail.com | Atlanta | GA | 2022-06-02 | consumer | NaN |
| 3 | Henry | Brooks | hbrooks@lakesidecafe.com | Los Angeles | CA | 2022-06-04 | business | NaN |
| 4 | Brian | Graham | bgraham@bluecedarhomes.com | Seattle | WA | 2022-06-06 | business | |
| 5 | Kyle | Jimenez | kjimenez27@yahoo.com | Indianapolis | IN | 2022-06-07 | consumer | NaN |
products120 rows · 8 columns
product_idintegernamevarcharcategoryvarcharsubcategoryvarcharunit_pricedecimalunit_costdecimallaunched_ondateis_activeboolean
| product_id | name | category | subcategory | unit_price | unit_cost | launched_on | is_active |
|---|---|---|---|---|---|---|---|
| 101 | Vale Velvet Loveseat | Furniture | Sofas | 1659.00 | NaN | 2023-09-11 | true |
| 102 | Alder Oak Coffee Table | Furniture | Tables | 919.00 | 503.50 | 2024-05-16 | true |
| 103 | Orchard Rattan Dining Chair | Furniture | Chairs | 789.00 | 429.87 | 2023-05-27 | true |
| 104 | Upland Oak Bookcase | Furniture | Storage | 429.00 | 214.70 | 2022-12-20 | true |
| 105 | Avery Leather Bed Frame | Furniture | Beds | 449.00 | 232.70 | 2021-08-31 | true |
orders12,000 rows · 7 columns
order_idintegercustomer_idintegerorder_datetimestampstatusvarcharchannelvarcharshipping_feedecimalcoupon_codevarchar
| order_id | customer_id | order_date | status | channel | shipping_fee | coupon_code |
|---|---|---|---|---|---|---|
| 1001 | 125 | 2023-01-01 20:51:19 | completed | web | 0.00 | NaN |
| 1002 | 156 | 2023-01-02 07:58:46 | completed | web | 0.00 | NaN |
| 1003 | 22 | 2023-01-02 10:25:27 | completed | marketplace | 4.99 | WELCOME10 |
| 1004 | 48 | 2023-01-02 11:21:51 | completed | marketplace | 4.99 | WELCOME10 |
| 1005 | 82 | 2023-01-02 11:43:36 | completed | web | 0.00 | WELCOME10 |
order_items30,338 rows · 6 columns
order_item_idintegerorder_idintegerproduct_idintegerquantityintegerunit_pricedecimaldiscount_pctdecimal
| order_item_id | order_id | product_id | quantity | unit_price | discount_pct |
|---|---|---|---|---|---|
| 1 | 1001 | 121 | 9 | 31.99 | 0.00 |
| 2 | 1001 | 154 | 3 | 11.99 | 0.00 |
| 3 | 1002 | 154 | 4 | 11.99 | 0.00 |
| 4 | 1002 | 194 | 1 | 149.00 | 0.00 |
| 5 | 1002 | 193 | 2 | 19.99 | 0.00 |
returns1,673 rows · 5 columns
return_idintegerorder_item_idintegerreturned_ondatereasonvarcharrefund_amountdecimal
| return_id | order_item_id | returned_on | reason | refund_amount |
|---|---|---|---|---|
| 1 | 14 | 2023-01-10 | damaged | 83.69 |
| 2 | 42 | 2023-01-10 | wrong_item | 19.99 |
| 3 | 105 | 2023-01-14 | wrong_item | 10.99 |
| 4 | 24 | 2023-01-16 | changed_mind | 26.99 |
| 5 | 126 | 2023-01-16 | late | 20.99 |
employees60 rows · 8 columns
employee_idintegerfirst_namevarcharlast_namevarchardepartmentvarchartitlevarcharmanager_idintegerhire_datedatesalaryinteger
| employee_id | first_name | last_name | department | title | manager_id | hire_date | salary |
|---|---|---|---|---|---|---|---|
| 1 | Catherine | Hale | Executive | Chief Executive Officer | NaN | 2016-03-01 | 285000 |
| 2 | Robert | Kim | Executive | Chief Operating Officer | 1 | 2017-01-09 | 235000 |
| 3 | Megan | Collins | Sales | Head of Sales | 1 | 2018-04-16 | 172000 |
| 4 | Chloe | Gutierrez | Sales | Account Executive | 3 | 2020-01-21 | 72000 |
| 5 | Frank | Bell | Sales | Account Executive | 3 | 2023-05-21 | 73000 |
marketing_spend3,276 rows · 3 columns
spend_datedatechannelvarcharamountdecimal
| spend_date | channel | amount |
|---|---|---|
| 2023-01-01 | paid | 492.99 |
| 2023-01-01 | 37.38 | |
| 2023-01-01 | social | 200.53 |
| 2023-01-02 | paid | 709.13 |
| 2023-01-02 | 64.13 |
The lesson
Plan before you launch
Every A/B test needs three numbers decided before it starts: the baseline rate (how the current version converts), the smallest lift worth detecting (the minimum detectable effect), and the sample size that gives a fair chance of seeing that lift if it is real. Skip this and you either stop too early (the peeking trap) or run a test that could never have found anything.
Power and alpha
Two error rates go into the plan:
- alpha = 0.05: the false-positive rate you accept when there is no real effect.
- power = 0.80: the chance of detecting the lift when it is real. The other 20% is a false negative: a real improvement the test misses.
Each becomes a z-value from the normal curve: NormalDist().inv_cdf(0.975) ≈ 1.96 for a two-sided
alpha of 0.05, and NormalDist().inv_cdf(0.80) ≈ 0.84 for 80% power.
The sample size formula
For two conversion rates p0 (control) and p1 (the rate you hope to see), the sessions needed per arm are:
n = (z_alpha + z_beta) ** 2 * (p0 * (1 - p0) + p1 * (1 - p1)) / (p1 - p0) ** 2The gap p1 − p0 is squared in the denominator. That is the whole story of A/B testing on small sites: low conversion rates make the gap tiny, and a tiny gap squared makes n enormous.
What the store can afford
With 2025’s 44.5 sessions a day and a 3.08% baseline, here is what different target lifts cost:
| lift to detect | sessions per arm | days with all traffic |
|---|---|---|
| 10% | 51,752 | 2,329 |
| 20% | 13,530 | 609 |
| 30% | 6,275 | 283 |
| 50% | 2,445 | 111 |
Even a 50% lift needs almost four months. This is normal for a small store, and the reason big companies can test button colours and small ones cannot.
Common mistakes
- Relative versus absolute lift. “10% lift” on a 3% baseline means 3.3%, not 13%.
- Per arm versus total. The formula gives one arm. The test needs both.
- Planning on a busy month. November and December convert better and bring more traffic; a plan built on holiday numbers underestimates the time a spring test needs. Use a full year.
- Lowering the bar to fit the calendar. Running two weeks “because that’s what we have” does not make a 10% lift detectable. It makes a false result likely, in either direction.
Lantern lit!
Practised: sample sizepowerminimum detectable effectbaseline ratetest duration
- Solved
- No hints
- First run