7. Lighthouse

Level 5 · Outlier or error?

From Megan Collins, Head of Sales

“Someone on my team wants to 'remove the outliers' before we report order value: the negative orders and the giant ones. That makes me nervous. What should we actually do?”

Your goal

Pick the right way to handle the extreme order values.

Among 10,849 completed orders, 4 have a negative order value (the lowest is −$420.06, from a line with quantity −1). The largest is $11,103.27: a business account buying a daybed, a bookcase, a patio dining set and two smaller items. Both sit outside the 1.5 × IQR fences. What should go into the report?

Pick one answer, then check it.

The lesson

Two different things

An outlier is a value far from the rest. An error is a value that is wrong: it does not describe what happened. Some outliers are errors, most are not, and some errors are not outliers at all (a $250 order typed as $205 hides in the middle). Statistics can flag outliers; only knowledge of the business can tell you which are errors.

The test: could it have happened?

Harbor & Pine’s lowest completed orders:

completed["order_value"].nsmallest(4)    # -420.06, -336.01, -52.97, -19.01

Each one contains a line with a negative quantity, like −1 or −4. A completed sale with a negative quantity is impossible; the data notes call these data-entry errors. They are errors, whatever their distance from the median.

The largest orders, $11,103, $10,730 and $10,551, all come from business accounts and contain several large items. Possible, plausible, and important: business customers place 27% of completed orders and bring in far more than their share of revenue.

What to do with each

  • Errors: exclude them (or correct them if the source system can), and write down how many and why. “Excluded 4 orders with negative values (data-entry errors)” is one line in the notes.
  • Real extremes: keep them. Revenue, totals and means need every real order. To describe the typical customer, switch to the median and percentiles rather than deleting the tail.
  • Unsure: check the source. One query showing the order lines, or one message to the team that owns the system, beats a guess.

Why not trim or cap?

Trimming (dropping the top and bottom 5%) and capping (winsorizing: replacing values above a limit with the limit) are real techniques, used to stop a few points from dominating a model or a test statistic. In a sales report they quietly change what customers paid. If you use one, name it (“trimmed mean, top and bottom 5% removed”) so nobody mistakes it for the average order value.

Common mistakes

  • Deleting everything outside the fences. On skewed data that removes your biggest customers.
  • Keeping impossible values because “removing data is bad”. An impossible value is not data about your customers.
  • Cleaning silently. Every exclusion changes a number someone else will try to reproduce.

Cookie settings

Pick what may run on this site. You can change this any time from "Cookie settings" at the bottom of every page.