Multiple Regression and Linear Models
Almost every response worth modelling depends on several predictors at once. This unit extends least squares to the multiple regression model, teaches what a coefficient means when the other predictors are held fixed and why it changes when one is left out, puts categorical variables into the model as indicators, and tests coefficients and whole models with t and F statistics. It ends with the residual checks that decide whether a fit deserves belief and with prediction intervals for a new case.
Unit 8 fitted a straight line through one predictor and one response, and its residual plot for Galton’s heights ended with a warning: the sons and the daughters formed two separate bands, so a second variable belonged in the model. Almost every question that is worth a regression has this shape. A car’s fuel economy depends on its weight and on its engine, and a penguin’s body mass depends on its size and on its species. This unit extends least squares to several predictors at once. The arithmetic follows the same idea as before, which is to make the sum of squared residuals as small as possible; the new work lies in reading the coefficients, which now describe each predictor with the others held fixed, and in checking whether the model that produced them deserves to be believed.
1 Several predictors at once
The multiple regression model writes each observation of the response \(y\) as a sum of contributions from \(k\) predictors \(x_1, \ldots, x_k\) and an error term, \begin{equation} y_i = \beta_0 + \beta_1 x_{i1} + \beta_2 x_{i2} + \cdots + \beta_k x_{ik} + \varepsilon_i , \qquad i = 1, \ldots, n , \label{eq:model} \end{equation} where \(x_{ij}\) is the value of predictor \(j\) for case \(i\). The coefficients \(\beta_0, \ldots, \beta_k\) are unknown constants, and the errors \(\varepsilon_i\) collect everything that moves \(y_i\) away from the sum. With \(k = 1\) this is the model of Unit 8. The word “linear” in linear model, the family name of every model of this shape, refers to the coefficients, which enter as plain multipliers; the predictors themselves may be anything that was measured or computed, such as the square of an age or the logarithm of an income, and Section 3 puts a species into the model as a predictor.
The clause “while every other predictor is held fixed” is the whole point of the unit, and it deserves a slow reading. Two cars of the same engine power that differ by 1,000 pounds in weight differ in expected fuel economy by \(\beta_{\text{weight}}\); two cars of the same weight that differ by one horsepower differ by \(\beta_{\text{power}}\). Neither number is the slope that Unit 8 would have fitted to a plot of fuel economy against weight, because in that plot the heavier cars also had the larger engines, and the simple slope counted the effect of the engine as if it belonged to the weight. Statisticians call \(\beta_j\) a partial coefficient and say that the regression “adjusts for” the other predictors; the first worked example shows the adjustment at work.