The Bootstrap and Cross-Validation
The first unit of the Advanced + Statistical Learning masterclass turns from estimating parameters to predicting new data, and lets the computer stand in for formulas. Polynomials of rising degree fitted to the Old Faithful eruptions show why the training error flatters a model and the test error does not, and the bias-variance decomposition explains the shape of the test error curve. The bootstrap gives a standard error and an interval for a median and a correlation by resampling the data, with the result that a bootstrap sample holds about 63 per cent of the observations. Ten-fold cross-validation chooses the degree of a polynomial, the one-standard-error rule leans towards the simpler model, and the sleep data of Unit 7 are tested again by flipping the signs of the ten differences.
Twelve units have asked what a sample says about a parameter: a mean, a proportion, a slope, a posterior. The last four units ask a second question, namely how well a model fitted to one set of data will predict the next set, and they answer it with a tool that the first twelve units used only for illustration, namely the computer. Two ideas from the 1970s carry this unit. The bootstrap, published by Bradley Efron in 1979, replaces the formula for a standard error by a thousand new samples drawn from the data themselves, and so provides a standard error and a confidence interval for statistics that have no formula. Cross-validation hides part of the data, fits a model to the rest, and measures the prediction errors on the hidden part, which is the only honest measure of how a model will do on data it has not seen. Both are used every day in every field that fits models to data.
1 Prediction and the test error
The phrase statistical learning covers the methods whose aim is to predict a response \(y\) from one or more predictors \(x\), and with less interest in the individual coefficients than Units 8, 10, and 11 had in them. The fitted rule is written \(\hat{f}(x)\), and it may be a line, a polynomial, a logistic curve, or one of the trees of Unit 15. The data it is fitted to are called the training set, and fitting is often called training. How good is the rule? A loss function puts a price on each mistake. For a quantitative response the standard choice is the squared error \((y - \hat{f}(x))^2\), which is the quantity that least squares minimizes (Unit 8). For a categorical response the rule names a category and is either right or wrong, so the loss is 1 for a wrong category and 0 for a right one, and its average is the misclassification rate, the fraction of wrong predictions.
The price can be averaged over two different sets of observations, and the whole unit turns on the difference between them.
The training error always flatters the rule. The same observations were used to choose the coefficients and to judge them, and the coefficients were chosen to make exactly this quantity as small as possible. A more flexible rule has more coefficients to adjust, and so its training error is at least as small as that of a less flexible rule, whatever the data. For example, a polynomial of degree 5 can do everything a polynomial of degree 4 can do, by setting its last coefficient to zero, so its training error cannot be larger than that of the degree-4 polynomial. The test error cannot be pushed down in this way, because a new observation had no say in the fit. A rule that has bent itself around the accidents of its training set pays for that on the test set, and the name for this is overfitting: the rule has learned the noise of the training set together with the signal, and the noise does not repeat.