Chapter 8 · 6 levels

Cleaning messy data

Handle missing values, fix inconsistent text, find duplicate customers, work with dates and types, and remove bad rows before reporting.

Start

Every dataset you’ll meet on the job is messy, and Harbor & Pine is no exception. Customers typed their city as ' dallas ' and 'NEW YORK'. Some states are lowercase or padded with spaces. Eighteen people signed up twice, sometimes with the same email in different capitals. Nine products have no recorded cost. Two dozen order lines have a quantity of zero or less.

None of these problems throws an error. They just make your numbers wrong: a state counted three times under three spellings, a “new customer” who already has 40 orders, revenue reduced by negative quantities. Cleaning is how you make sure the numbers mean what you say they mean.

This chapter covers the core toolkit: missing values (isna, fillna, dropna), text cleaning with the .str accessor, duplicates, and dates and types. The boss level produces a clean monthly revenue report and a deduplicated customer count, the kind of thing that ends up in a board deck.

Levels

  1. Where do customers come from?

    from Jordan Reyes, Head of Marketing

    nan, isna, fillna, dropna, value counts

    Code

  2. Fix the state codes

    from Megan Collins, Head of Sales

    str accessor, strip, upper, title, normalisation

    Code

  3. Find duplicate customers

    from Priya Shah, Head of Data

    duplicated, drop duplicates, subset, keep, normalisation

    Code

  4. Which weekday sells most?

    from Jordan Reyes, Head of Marketing

    to datetime, dt accessor, dayofweek, day name, astype, to numeric

    Code

  5. What does mean() do with missing costs?

    from Laura Bennett, Head of Finance

    nan, mean, skipna, isna

    Quiz

  6. Boss: the clean 2025 report

    from Catherine Hale, Chief Executive Officer

    data quality, filtering, dates, merge, groupby, drop duplicates

    Boss · 3 steps