Portfolio project · Python, Statistics · Intermediate · 8 to 12 hours

A/B test analysis and write-up

A product team ran a test and wants to know whether to ship. You check the data, estimate the difference with a confidence interval, and write a decision a product manager can act on. The write-up matters as much as the test: it shows you can explain uncertainty in plain words.

The business question

Did the new landing page convert better than the old one, by enough to be worth shipping?

Turned into a SMART goal, so you know when you are done and a reviewer can see you scoped it like an analyst:

Specific
Conversion rate of control against treatment on one primary metric, chosen before looking at the results.
Measurable
Lift in percentage points with a 95% confidence interval, judged against a decision rule written first, for example: ship if the interval is above zero and the lift is at least 0.5 points.
Achievable
One public test file and a notebook with pandas plus scipy or statsmodels.
Relevant
The product team decides whether to roll the new page out to everyone.
Time-bound
Uses the whole test period in the file. Write-up done in one week.

Datasets

Public data, free to download. Pick one, read its license on the source page, and link that page in your README rather than uploading the files.

Kaggle

A/B testing (ab_data.csv)

About 290,000 rows: user, timestamp, group, landing page and whether the user converted. A good first test file because it has real data problems to find.

Some users appear more than once, and some rows put a control user on the new page or the reverse. Find and handle both before you test anything.

Kaggle

Marketing A/B Testing

About 588,000 users who were shown either an ad or a public service announcement, with a converted flag and when they saw the most ads.

The two groups are very different in size. Say why that is or is not a problem for this design.

Kaggle

Mobile Games: A/B Testing (Cookie Cats)

About 90,000 players of a mobile puzzle game, with the first gate at level 30 or level 40, and 1-day and 7-day retention flags.

The metric is retention, not conversion. Look at the game-rounds column before you summarize it; it has an extreme value.

Step by step

Each step links to AnalystQuest levels that teach the skill on the Harbor & Pine store data, so you can practise it before you do it on the real dataset.

  1. Write the plan before the result

    State the hypothesis, the one primary metric, a guardrail metric if the file has one, the smallest lift worth shipping, and the decision rule. Writing these first is what stops you from choosing the metric that happens to look good.

  2. Check the data and the split

    Look for duplicate users, rows where the group and the experience do not match, and gaps in the dates. Then check the split: if the test was meant to be 50/50, test whether the group sizes are consistent with that. A sample ratio mismatch can make the whole result unreliable.

  3. Estimate the lift

    Compute the conversion rate per group, the absolute difference in percentage points and the relative lift. Run a two-proportion z-test and report the 95% confidence interval for the difference, not only the p-value.

    A result that is not significant is not proof of no effect. Say how large an effect the test could have detected.

  4. Show it in one chart

    One chart: the conversion rate of each group with its confidence interval as error bars, the axis labeled in percent. If you look at segments (device, country, date), label them as exploratory; with enough slices, one will look significant by chance.

  5. Write the decision

    Open with the decision and the number behind it, then the interval in plain words, the data checks you ran, and the caveats. Close with a next step: ship, do not ship, or rerun with a sample size that can detect the lift you care about.

What to show

The same project, packaged three ways. Numbers in brackets are placeholders: replace them with your own results.

GitHub

README, in this order

  1. Decision in one line
  2. Question, hypothesis and decision rule
  3. Data checks: duplicates, mismatches, split
  4. Result: rates, lift, interval, p-value
  5. The chart
  6. Caveats and next test

Files

  • notebooks/ab_test.ipynb: checks, test and chart, top to bottom
  • plan.md: the plan you wrote before the analysis, dated
  • images/conversion_ci.png: the chart for the README
  • data/README.md: where to download the data

LinkedIn

Post outline

  1. Open with the decision: "The new page did not beat the old one, and here is how sure we can be."
  2. One line on the setup: two groups, one metric, the rule written first.
  3. The data problem you found and how you handled it.
  4. The result as an interval in plain words.
  5. The chart and the repo link; ask how others handle uneven splits.

CV

Two bullets

  • Analyzed a [290k]-user landing-page A/B test in Python: removed [n] duplicate and mis-assigned rows, checked the split, and estimated a lift of [−0.2] points (95% CI [−0.5, 0.1]).
  • Recommended [not shipping] the new page and sized a follow-up test to detect a [0.5]-point lift with [80]% power in [3] weeks.

Start with a verb, include a number, and name the tool. Every number must be one you can explain in an interview.

Write-up template

Copy it into your README or a document and replace each bracket. Keep the executive summary to what a busy manager would read.

# [Project title]: A/B test write-up

## Executive summary
Decision: [ship / do not ship / rerun]. [Treatment] converted at [x]% against [y]% for control, a difference of [d] points (95% CI [a] to [b]).

## Objective
Did [the new landing page] raise [conversion] by at least [0.5] points?
Decision rule, set before the analysis: [..].

## Data and limits
Source: [dataset, link]. Users: [n] control, [n] treatment. Period: [dates].
Removed: [n] duplicate users, [n] rows where group and page did not match.
Split check: [expected 50/50; observed [..]; p = [..]].
Limits: [e.g. short test, no revenue data, novelty effect possible].

## Method
Two-proportion z-test on [metric]; 95% confidence interval for the difference.
Segments reviewed as exploratory only: [..].

## Findings
1. [Rates, difference and interval in plain words.]
2. [What the test could and could not detect.]
3. [Any exploratory pattern, labeled as such.]

## Recommendations
[The decision and the reason, in two sentences.]

## Next steps
[e.g. a rerun sized for the lift that matters, or a guardrail metric to add.]

Common mistakes

  • Choosing the metric after seeing which one moved.
  • Checking results every day and stopping when p dips under 0.05.
  • Reporting only the relative lift ("+12%") without the absolute rates.
  • Reading "not significant" as "no effect".
  • Slicing into many segments and reporting the one that won.
  • Skipping the duplicate and split checks.

How long it takes

About 8 to 12 hours for a first version, spread over one to three weeks of evenings.

Plan
1 hour
Data checks
2 to 3 hours
Test and interval
2 to 3 hours
Chart
1 to 2 hours
Write up
2 to 3 hours

Other portfolio projects

Excel, Power BI · 10 to 13 hours

E-commerce sales dashboard

One page a head of sales can read in a minute: revenue, orders, average order value and where they moved.

SQL, pandas · 12 to 16 hours

Customer cohort retention analysis

Do first-time buyers come back? Monthly cohorts and a retention grid, built in SQL and checked in pandas.

Cookie settings

Pick what may run on this site. You can change this any time from "Cookie settings" at the bottom of every page.