Level 6 · Metric trap: every device got better

The total is a weighted average

The overall conversion rate is total conversions over total sessions. Split by device, that is the same as a weighted average of the device rates, where each device’s weight is its share of sessions:

overall = share_desktop * rate_desktop + share_mobile * rate_mobile + share_tablet * rate_tablet

So the total can move for two reasons: the device rates changed (rate effect), or the shares changed (mix effect).

What happened from 2023 to 2025

Every device converted better in 2025: desktop from 3.78% to 4.13%, mobile from 2.30% to 2.52%, tablet from 2.93% to 2.94%. But the weights moved too. Mobile went from 47.8% of sessions to 58.0%, and desktop from 41.9% to 32.4%. More of the traffic now sits on the device that converts worst.

Hold the mix fixed to see the rate effect on its own: weight the 2025 rates by the 2023 shares.

0.419 * 4.13 + 0.478 * 2.52 + 0.103 * 2.94    # ≈ 3.24%

With the old mix, 2025 would have converted 3.24%. The real 3.08% is 0.16 points lower, and that difference is the mix effect. Both teams’ numbers are right.

Why the simple average is the wrong check

The average of the three device rates is 3.20%. It gives tablet, with about 10% of sessions, the same weight as mobile with 58%. That is the classic average of ratios versus ratio of totals trap. The overall rate is the ratio of totals: 500 / 16,225 = 3.08%.

When this shows up

Mix shift is behind many “every segment improved but the total fell” puzzles, and it has a famous extreme form, Simpson’s paradox, where the total moves in the opposite direction to every segment. It appears whenever a ratio is reported for a population whose composition changes: conversion as mobile grows, margin as the product mix moves, average order value as new customers arrive.

Interview note

“Our conversion rate dropped but every channel’s rate is flat or up. What’s going on?” is a favourite case question. Answer: check the mix, since traffic probably moved toward a low-converting segment. Then show the rate-at-old-mix calculation, as above, to size the two effects.