Level 5 · What does mean() do with missing costs?

NaN in aggregations

pandas aggregations (sum, mean, median, min, max, std, count) skip missing values by default. So:

products["unit_cost"].count()   # 111, non-missing values only
products["unit_cost"].mean()    # sum of the 111 known costs / 111
products["unit_cost"].mean(skipna=False)   # NaN

This default is convenient and usually right, but it quietly changes the population. “Average unit cost” really means “average unit cost of products whose cost we know”. If the missing products are special (all expensive furniture, say), the average is biased. Check which rows are missing before trusting the number:

products.loc[products["unit_cost"].isna(), ["product_id", "name", "category", "unit_price"]]

Plain NumPy behaves differently: np.mean on an array with NaN returns NaN. That’s one reason to stay with pandas methods for analysis, and to know which library you’re calling.

Sums and counts

sum() of a column that’s all NaN returns 0, not NaN. That can make “no data” look like “zero revenue” on a dashboard. When it matters, check count() alongside the sum.

NaN never equals anything

np.nan == np.nan                     # False
products["unit_cost"] == np.nan      # all False, even on missing rows
products["unit_cost"].isna()         # the right test

By definition, a missing value isn’t equal to anything, including another missing value. Always use isna() and notna() to test for it.

Arithmetic spreads NaN

products["unit_price"] - products["unit_cost"]    # NaN where cost is missing

That’s the correct behaviour: the margin of a product with unknown cost is unknown. If you then take the mean margin, those products are skipped, so your margin average covers 111 products. Report it that way: “average margin across the 111 products with a recorded cost”.

Missing values in groups

The same skipping happens inside groupby. If you compute the average cost per category, each category’s mean uses only its products with a known cost, so one category might be based on 20 products and another on 17. Add a count next to the mean, for example with named aggregation n_with_cost=("unit_cost", "count"), so readers can see how solid each number is.

The analyst’s rule

For every number you report from a column with gaps, say how many rows it’s based on. It takes one line of text and prevents the most common follow-up question in any meeting.