Level 1 · Where do customers come from?

What missing looks like

In plain Python a missing value is None. Inside a pandas column it shows up as NaN (for numbers and text loaded from CSV) or NaT (for dates): an empty cell in the CSV becomes NaN. Either way, missing is not zero and not an empty string; it means “we don’t know”.

Finding them is the first step:

customers.isna().sum()

customer_id          0
...
city                14
...
referral_source    666

isna() returns a True/False table, and .sum() counts the True values per column. notna() is the opposite. customers["referral_source"].isna().mean() gives the share missing, here about a third.

Missing is information

Ask why values are missing before deciding what to do. Here, referral_source is missing because many customers skipped the “how did you hear about us?” question. That’s natural, not an error, and it’s a sizeable group, so it belongs in the report as its own category.

city is missing on 14 rows, likely a data-entry gap. products.unit_cost is missing for 9 products whose cost wasn’t recorded. Each case calls for a different treatment.

fillna: replace with a value

customers["referral_source"].fillna("unknown")
products["unit_cost"].fillna(products["unit_cost"].median())   # an estimate, use with care

fillna returns a new Series. To change the DataFrame, assign it back to the column: customers["referral_source"] = customers["referral_source"].fillna("unknown"). Avoid fillna(..., inplace=True) on a selected column; it’s a form of chained assignment that may not change the original DataFrame.

Fill text categories with an explicit label like "unknown". Be much more careful with numbers. Filling missing costs with 0 would make those products look 100% profitable, and filling with an average invents data. Often the honest choice is to leave numbers missing and exclude them from that specific calculation, saying so in a footnote.

dropna: remove rows

customers.dropna(subset=["city"])          # drop rows with no city
products.dropna(subset=["unit_cost"])      # only products with a known cost

Always pass subset=. Without it, dropna() drops any row with any missing value, and in customers that would throw away a third of the table because of referral source.

Dropping rows changes your population. A referral breakdown computed after dropna() describes only customers who answered, which overstates every channel’s share.

How pandas treats NaN in calculations

• sum, mean, count, max and friends skip NaN by default.
• value_counts and groupby leave out NaN keys by default; use dropna=False to keep them.
• Comparisons with NaN are always False, even np.nan == np.nan. Use isna() to test.
• Arithmetic with NaN gives NaN: unit_price - unit_cost is NaN when the cost is missing.

So the shortcut for this level is also customers["referral_source"].value_counts(dropna=False), which counts NaN as its own row. Using fillna("unknown") first gives that row a readable label, which is better for anything you’ll show people.

Common mistakes

• Reporting shares after value_counts() without dropna=False or fillna.
• Calling dropna() without subset and losing most of the table.
• Filling numeric gaps with 0 by reflex.
• Testing == np.nan, which is never True.

If you know SQL

fillna is COALESCE(referral_source, 'unknown'), and isna() is IS NULL.