A laboratory technician at a computer whose screen shows a DNA microarray, a grid of thousands of coloured spots, one per gene
Unit 14 · Advanced + Statistical Learning · 75 min

Ridge, Lasso, and Regularization

Shrinkage, ridge regression, the lasso, soft thresholding, and choosing the penalty by cross-validation
Image: National Cancer Institute, Bill Branson, public domain

Least squares comes apart when predictors are correlated or more numerous than the observations: the coefficients swing to absurd values, and with more predictors than patients there is no unique fit at all. This unit adds a price to the size of the coefficients. Ridge regression charges for squared coefficients and shrinks all of them, and the lasso charges for absolute values and sets some exactly to zero, so that it selects predictors while it fits; the diabetes data of 442 patients and the fuel consumption of 32 cars show both at work. The penalty is chosen by cross-validation with the one-standard-error rule, the penalties turn out to be Bayesian priors in disguise, and a worked example shows how many of 5,000 unrelated genes will look correlated with an outcome by chance alone.

Preview: the beginning of the unit

Unit 10 fitted a regression with two or three predictors and read the coefficients off a table. Modern datasets offer hundreds or thousands of predictors at once, a patient’s genes, a customer’s clicks, every word of a document, and many of them measure nearly the same thing. Least squares then does something unpleasant: the fitted line still predicts, but its coefficients swing to absurd values, and with more predictors than observations it has no unique solution at all. The repair of this unit is to add a price to the size of the coefficients, which pulls them towards zero. Ridge regression, which grew out of a chemical company’s process data and was published in 1970, charges for squared coefficients and shrinks all of them; the lasso, invented in 1996, charges for absolute values and sets some of them exactly to zero, so that it chooses the predictors while it fits. The strength of the penalty is chosen by the cross-validation of Unit 13, and the penalties turn out to be the priors of Unit 12 under another name.

1 Many predictors and the trouble with least squares

The running example of this unit is the dataset with which the lasso’s own inventors demonstrate it. Bradley Efron, Trevor Hastie, Iain Johnstone, and Robert Tibshirani published it in 2004 with their paper on least angle regression. It records, for 442 diabetes patients, ten variables measured at a baseline visit, namely age, sex, body mass index (BMI), average blood pressure, and six blood serum measurements, and the response, a quantitative score of how far the disease had progressed one year later. The six serum measurements are total cholesterol, low-density lipoprotein (LDL), high-density lipoprotein (HDL), the ratio of total cholesterol to HDL (TCH), the logarithm of the serum triglyceride level (LTG), and blood glucose. The score has mean 152 and standard deviation 77 and carries no unit of measurement in the paper, so the unit writes “points”.

1.1 Standardizing the predictors

Before any penalty is applied, the predictors are put on a common scale, because a penalty on the size of a coefficient only makes sense when the coefficients are comparable. A coefficient of 10 points per year of age and a coefficient of 10 points per unit of BMI have nothing in common, and a penalty that treated them alike would penalize the choice of units. This unit standardizes each predictor in two steps: subtract its mean and divide by its standard deviation (the \(z\)-score of Unit 2), and then divide by \(\sqrt{n - 1}\) as well, so that every column of predictors has mean zero and a sum of squares equal to one, because the squared \(z\)-scores of a column add up to \(n - 1\). The second step is a convention of the field (it is how the diabetes data were published) that keeps every formula of this unit free of stray factors. A coefficient on this scale is \(\sqrt{n - 1}\) times the coefficient per standard deviation of the predictor, because a predictor that has been divided by \(\sqrt{n - 1}\) needs a coefficient \(\sqrt{n - 1}\) times larger to produce the same fitted values. For \(n = 442\) the factor is \(21.0\), so the least-squares coefficient \(520\) of BMI in Table 1 means \(520/21.0 = 24.8\) points per standard deviation of BMI. The response is centred by subtracting its mean, and then the regression needs no intercept, because the least-squares intercept of centred data is zero. In practice the intercept is always left out of the penalty, since it only sets the level of the response and has nothing to do with the size of the predictors’ effects.

1.2 Collinearity

Table 1 shows the least-squares fit of the diabetes score on the ten standardized predictors, computed as in Unit 10. BMI, blood pressure, LTG, and sex have coefficients several times their standard errors. The two cholesterol measurements are another matter. Total cholesterol has the largest coefficient of all, \(-792\), and LDL the next largest, \(+477\), with opposite signs; both have standard errors of several hundred, and neither is clearly different from zero. The reason is that the two predictors measure nearly the same thing: their correlation across the 442 patients is \(0.897\). When two predictors move together, the data can hardly tell which of them the response follows, so least squares is free to give one a large positive coefficient and the other a large negative one, as long as the two nearly cancel. The sum of the two coefficients is well determined; each one alone is not. This condition is called collinearity, and the standard errors measure the damage. Unit 10 introduced the variance inflation factor, the factor by which the variance of a coefficient exceeds what it would be if the predictor were uncorrelated with the others; for total cholesterol it is \(59\) and for LDL \(39\), so their standard errors are \(\sqrt{59} = 7.7\) and \(\sqrt{39} = 6.2\) times larger than they would be for uncorrelated predictors, and that is why the two coefficients can wander so far.

The full unit is part of the program

Want to see a complete unit first? Unit 5, Sampling and the Central Limit Theorem, is free to read.