Regression to the mean, the most misread pattern in sport and school Image: Unknown photographer, 1904, public domain

Regression to the mean, the most misread pattern in sport and school

A school identifies the 50 pupils with the lowest reading scores and gives them a term of extra lessons. At the end of term it tests them again, and their average has risen. A football club's player of the season signs a new contract and has an ordinary year. A speed camera is installed at a junction that had an unusually bad year for accidents, and accidents fall. In each case a cause is announced, and in each case most or all of the change would have happened with no lessons, no contract, and no camera. The pattern is called regression to the mean, and it is the most common statistical illusion there is.

Galton's heights

Francis Galton found it in 1886. He had collected the heights of 928 adult children and their 205 pairs of parents, and he plotted the children's heights against the average height of the two parents, scaling the mothers up by 8 per cent to put them on the same footing as the fathers. Tall parents had tall children, as expected. But the children of the tallest parents were, on average, less tall than their parents, and the children of the shortest parents were less short. For every inch the parents stood above the average, the children stood about two thirds of an inch above it. Galton called this regression towards mediocrity, and the name regression stuck to the whole method of fitting lines to data. His paper, with the original table, is online.

He worried that the finding meant the population was converging on the average and that in a few generations everyone would be the same height. It does not mean that, and seeing why is the key to the whole subject. The children of the tallest parents are shorter than their parents; but the tallest children have parents who are, on average, shorter than they are. The regression runs both ways. Pick the extreme group by either variable and the other variable is less extreme. The spread of heights in the population stays exactly the same.

Why it happens

Any measured quantity can be thought of as a true, stable part plus a chance part. A pupil's reading score is reading ability plus the luck of that day's test: which passages came up, how well she slept, whether she guessed right on the questions she did not know. An athlete's season is talent plus the luck of injuries, bounces, and opponents. Now pick the pupils with the lowest scores. You have selected, in part, for bad luck, because a pupil who scored very low is more likely to have been unlucky than lucky on the day. On the retest the ability is still there and the bad luck is not, so the group's average rises. Nothing was done to them. The same is true of the top group in reverse, which is why the best performers on any test are, on average, slightly worse next time.

How much the extreme group moves back depends on how much of the measurement is chance. If the test were a perfect measure of ability, there would be no regression; if it were pure luck, the retest average would fall all the way back to the overall mean. In between, the fraction of the distance from the mean that is kept equals the correlation between the two measurements. Galton's two thirds was the correlation between parents' and children's heights. A reading test that correlates 0.7 with itself a term later will, on average, give the bottom group 70 per cent of its deficit back and forgive the other 30.

The sophomore slump and the cover jinx

Sport is full of this. The rookie of the year has a worse second season so often that the sophomore slump has a name, and the explanation is that the rookie of the year was chosen for having the best first season, which selected for talent and for luck together. Only the talent returns. The Sports Illustrated cover jinx, the belief that appearing on the magazine's cover brings a bad spell, is the same effect: athletes appear on the cover at the peak of a run that was partly luck.

Daniel Kahneman told the story that made the pattern famous outside statistics. Teaching flight instructors in the Israeli Air Force in the 1960s, he said that praise improves performance more than punishment. An instructor objected: when he praised a cadet for an excellent manoeuvre, the next one was usually worse, and when he shouted at a cadet for a bad one, the next was usually better. The instructor was right about what he saw and wrong about why. An excellent manoeuvre is partly luck and is followed by a more ordinary one whether you praise it or not; a bad one is followed by a better one whether you shout or not. The instructors had been taught by experience to punish, because experience, unaided by statistics, rewards punishment with an apparent improvement every time.

Where it does damage

Medicine is the field where the illusion costs most. Patients see a doctor when they feel worst, which is an extreme measurement, and they would feel better afterwards without treatment. Part of what is called the placebo effect is regression to the mean: not the pill, not even the belief in the pill, but the selection of the day on which the pill was taken. Blood pressure measured once and found high is lower on the second visit in most people, which is why guidelines require repeated readings before a diagnosis.

Policy has the same problem. Interventions are aimed where things are worst: the junction with the most accidents this year, the school with the lowest scores, the police district with the most crime. The following year is better, and the intervention takes credit for an improvement that was coming anyway. Studies of speed cameras that compared camera sites with their own previous record found large falls in accidents; studies that compared them with similar sites without cameras found falls about half as large, the other half being regression.

How to tell it from a real effect

There is only one reliable way, and it is the same answer as always: a control group selected in the same way. Take the 100 lowest-scoring pupils, give extra lessons to a random half, retest everyone. Both halves will improve by regression; only the difference between the halves is the effect of the lessons. Without the control half, the before-and-after comparison measures the lessons plus the regression and cannot separate them. Every drug trial has a placebo group for this reason, and every claim that an intervention helped the worst cases should be met with the question of what happened to similar cases that received nothing.

Where a control group is impossible, the best you can do is to estimate the regression from the correlation between repeated measurements and subtract it, which is what the better studies of road safety do. What you cannot do is read an improvement in the extreme group as evidence of anything, however large it is.

Unit 8 of the Statistics Masterclass derives the regression line from Galton's height data and shows why the slope in standardized units is the correlation, which is the same statement as "the extreme group keeps the fraction \(r\) of its distance from the mean". The unit's treatment of before-and-after comparisons is the reason it carries causation in its title, and the article on correlation and causation continues the argument.

Learn statistics properly

The Statistics Masterclass: weekly units, quizzes, a final exam, and a certificate.
About the program