Better in every group, worse overall?

Suppose two hospitals each treat 100 patients for the same illness. Hospital A cures 58 of them; hospital B cures 72. Which hospital would you choose? Almost everyone says B. Now suppose you learn that A was better than B for mild cases and better than B for severe cases. That sounds impossible: if A wins in each part, surely it wins in the whole?

It does not have to. When a comparison reverses as you split the data into groups (or merge them back), statisticians call it Simpson’s paradox, after Edward Simpson, who described it in 1951. It is not a quirk of a few odd numbers. It turns up in medicine, in admissions data, in sport and in economics, whenever the groups being compared are made up differently.

Make it happen yourself

In the figure below, two treatments, A and B, are each given to a mix of easy cases and hard cases. For each treatment you set how many easy and hard patients it got, and how often it succeeds with each kind. The bars show the success rate (the share of patients for whom the treatment worked) in each group and overall.

As it starts, A beats B among easy cases (90% against 80%) and among hard cases (50% against 40%), but loses overall (58% against 72%). Try moving A’s hard cases down towards 10, and watch the paradox disappear. Then try to make B win both groups and lose overall.

You will have found a pattern. The paradox needs two things at once. First, one kind of case must be much harder than the other: here, hard cases succeed about half as often as easy ones, whichever treatment they get. Second, the two treatments must get very different mixes: A got mostly hard cases, B mostly easy ones. Give A and B the same mix and the paradox cannot happen.

A real case: kidney stones

In 1986 Clive Charig and colleagues at the Institute of Urology in London compared ways of removing kidney stones. Two of them were open surgery, an operation through a cut in the side, and percutaneous nephrolithotomy (PCNL), in which a surgeon reaches the kidney through a small hole using a tube and a camera. Each was used on 350 patients. The patients were not assigned at random: doctors chose the treatment.

The figure shows every one of those 700 patients as a dot. A filled dot is a success, an empty one is not. First look at the overall picture. Then press By stone size, which splits each treatment’s patients into small stones (under 2 cm across) and large stones (2 cm or more).

Overall, PCNL looks better: 83% against 78%. Split by stone size, open surgery is better for small stones (93% against 87%) and for large stones (73% against 69%). Same patients, same outcomes, opposite verdicts.

Look at the shapes of the blocks. Open surgery’s big block is large stones (263 of its 350 patients); PCNL’s big block is small stones (270 of its 350). Large stones are harder for both treatments. So open surgery was mostly handed the hard cases, exactly as A was in Figure 1.

Why: an overall rate is a weighted average

Here is the whole explanation in one line. A treatment’s overall success rate is the average of its two group rates, each weighted by the share of its patients in that group:

overall = (1 − w) × small-stone rate + w × large-stone rate

where w is the fraction of that treatment’s patients who had large stones. For open surgery w = 263/350 = 75%, so its overall rate is 0.25 × 93.1% + 0.75 × 73.0% = 78.0%. For PCNL w = 80/350 = 23%, giving 0.77 × 86.7% + 0.23 × 68.8% = 82.6%.

A weighted average of two numbers always lies between them, and where it lies depends only on the weight. In the next figure, each treatment is a line from its small-stone rate (left) to its large-stone rate (right). Its overall rate is a dot that sits on that line at its own mix of patients. Open surgery’s line is above PCNL’s everywhere, but its dot sits far to the right, down the slope. Drag the dots, or use the sliders, and give both treatments the same mix.

With the real mix, PCNL’s dot is higher. With the same mix for both, open surgery’s dot is higher: 83.3% against 77.9% at 49% large stones. Wherever you put the two dots, as long as they are at the same place, open surgery wins, because its line is above everywhere. The reversal comes entirely from the dots being at different places along their lines.

So which number should you trust?

The numbers do not answer this on their own. You have to know how the data came about.

Here, stone size came first. A patient arrived with a small or large stone; that size influenced which treatment the doctor chose, and it also influenced the chance of success. A variable that pushes on both the choice of treatment and the outcome is called a confounder. With a confounder, the overall comparison mixes up the effect of the treatment with the effect of the case mix. The fair comparison is like with like: within each stone size. On these data, open surgery had the better success rate for both sizes, by about 6 percentage points for small stones and 4 for large ones.

But splitting is not always right. Imagine instead that a drug lowers blood pressure, and that lower blood pressure is part of how it helps. If you split patients by their blood pressure after treatment, you would be hiding part of what the drug does. When the grouping variable is a consequence of the treatment, the overall number is the one to trust. The arithmetic is identical in both cases; only the story of cause and effect differs. Judea Pearl has argued that this is why the paradox once seemed baffling: the answer depends on causes, which the table alone does not show.

The figure below draws the two stories as arrows of cause and effect. Switch between them and watch which path the arrows open up between treatment and outcome.

A second famous example points the same way as the kidney stones. In 1973 the graduate division of the University of California, Berkeley, admitted about 44% of 8,442 men who applied but only about 35% of 4,321 women. Peter Bickel, Eugene Hammel and William O’Connell looked department by department and found little sign of bias against women; if anything, a small bias in their favour. Women had applied mostly to departments that were hard for anyone to get into. The authors did not conclude that there was no problem, only that it lay earlier, in what steered women toward those fields.

Simpson’s paradox does not mean that statistics can prove anything. It means that an overall average can hide a third variable, and that the right comparison depends on what that variable is doing.