Image: Martin Disteli, public domainThe bootstrap in one page
You have measured the bill lengths of 50 penguins and computed their median, 44.5 mm. How far is that median likely to be from the median of all the penguins on the island? For the mean there is a formula, \(s/\sqrt{n}\). For the median there is no simple formula, and for most statistics anyone actually wants, a ratio of two means, a correlation, the coefficient of a complicated model, there is none at all. Until 1979 the answer was to find a mathematician or to give up. Then Bradley Efron proposed that the data could answer the question themselves.
The idea
The uncertainty of a statistic comes from sampling: a different sample of 50 penguins would have given a different median. If you could draw a thousand new samples from the population and compute the median of each, the spread of those thousand medians would be the standard error you want. You cannot draw new samples from the population, because you have only the 50.
Efron's move is to treat the sample as if it were the population. Draw 50 penguins from your 50 at random, with replacement, so that some are drawn twice or three times and others not at all. Compute the median of this resample. Do it again, 1,000 times. The 1,000 medians scatter around the original median, and their standard deviation is the bootstrap estimate of its standard error.
The justification is that the sample is the best information you have about the population, and that the way resamples vary around the sample mimics the way samples vary around the population. Efron proved that for a wide class of statistics the mimicry is good enough, and in the half century since it has been checked in thousands of settings. He named the method after the expression about pulling oneself up by one's own bootstraps, which is what it appears to do.
A worked example
Take nine measurements of something: 3, 5, 5, 6, 8, 9, 12, 14, 20. Their median is 8. Here are three bootstrap resamples, each nine values drawn with replacement from the nine, sorted:
- 3, 3, 5, 6, 8, 8, 12, 14, 20, median 8
- 5, 5, 6, 6, 9, 12, 12, 14, 14, median 9
- 3, 5, 5, 5, 6, 6, 9, 20, 20, median 6
Repeat 1,000 times and the medians land mostly on 5, 6, 8, 9, and 12, with a standard deviation of about 2.1. That is the standard error of the median of this sample. For a 95 per cent interval, sort the 1,000 bootstrap medians and take the 25th and the 975th: this is the percentile interval, and here it runs from about 5 to 12. The interval is wide because nine observations are few and the data are skewed, and the bootstrap has told you so without a single formula.
The same recipe works for any statistic. Replace the median with the correlation between two columns, with the ratio of two group means, with the slope of a regression fitted by whatever method, and the three steps are the same: resample the rows, recompute, read the spread. That universality is why the method spread from the statistics journals into every empirical science within a decade.
The 63 per cent
One number about the bootstrap is worth knowing by heart. In a resample of \(n\) from \(n\), the chance that a particular observation is not drawn on one draw is \(1 - 1/n\), and the chance that it is not drawn in any of the \(n\) draws is
\[ \left(1 - \frac{1}{n}\right)^{n} \to \frac{1}{e} = 0.368. \]So about 36.8 per cent of the original observations are missing from any given resample, and about 63.2 per cent appear at least once, some of them more than once. For \(n = 50\) the figure is already 36.4 per cent. The resamples are not shuffles of the data; each is a genuinely different subset, weighted by how often each point was drawn, and that is where the variation comes from.
The observations left out are useful. A random forest, which grows each of its trees on a bootstrap resample, tests every tree on the 37 per cent of the data that tree never saw, and so measures its own prediction error without a separate test set. This out-of-bag estimate is the subject of the article on random forests.
What it cannot do
The bootstrap mimics sampling from the population by sampling from the sample, so it inherits everything that is wrong with the sample. A biased sample gives a biased bootstrap with a confident-looking interval around the wrong answer; the method measures variability, not bias. With very small samples, a handful of observations, the resamples contain too little variety and the intervals come out too narrow. For statistics that depend on the extremes, the maximum or the 99th percentile, the plain bootstrap fails, because a resample can never produce a value beyond the largest one observed. And for data that are not independent, time series or clustered observations, the rows cannot be resampled one at a time; blocks or clusters must be resampled together.
Within those limits it is the most useful general tool statistics has produced since the computer made it possible. Efron's paper, Bootstrap Methods: Another Look at the Jackknife, appeared in the Annals of Statistics in 1979 and is free to read; the jackknife of its title was an earlier resampling idea of Maurice Quenouille and John Tukey from the 1950s, which leaves out one observation at a time. In 2019 Efron received the International Prize in Statistics, the field's highest award, for the bootstrap, and the citation says that it transformed the practice of science.
Doing it yourself
The method needs nothing beyond a random number generator and patience, and a spreadsheet will do for small datasets. The Palmer penguins, the dataset of the week for Unit 13 of the Statistics Masterclass, has 344 rows under a CC0 licence, and bootstrapping the median bill length of one species takes a quarter of an hour. Unit 13 works through it, derives the 63.2 per cent, and then introduces cross-validation, the bootstrap's cousin for measuring how well a model predicts, with Efron as its research spotlight.


