Formula sheet

Key Formulas

The key formulas of the masterclass, collected unit by unit. Each one is derived or motivated in its unit, with the conditions under which it holds and a worked example on real data.

Statistics states its results as formulas so that they can be applied to new data and checked against old. A standard error written as a formula tells you at once why quadrupling a sample only halves the uncertainty; Bayes' theorem written down shows which three numbers a positive test result depends on; the least-squares slope, once written, can be computed from any two columns of a table. This page collects the key formulas of the masterclass in the order of the units: the mean and the standard deviation, the rules of probability and Bayes' theorem, expectation and variance, the standard error and the central limit theorem, the confidence intervals and the test statistics, the correlation coefficient and the regression line, the likelihood, the logistic model, the posterior, the bootstrap standard error, the ridge and lasso objectives, and the false discovery rate.

Each card gives the formula the way the Key Formula box of its unit does, names its symbols, and states the condition an expert would insist on: a simple random sample, independent observations, a large enough sample, or a model whose errors are normal. Population parameters are Greek letters, sample statistics are Latin letters, and estimates wear a hat, as in every unit. The page grows as units are published, so a formula you cannot find yet belongs to a unit that is still to come.

Data, Variables, and the Statistician's Toolkit

The summation sign and its two rules

\[ \sum_{i=1}^{n} c\,x_i = c \sum_{i=1}^{n} x_i , \qquad \sum_{i=1}^{n} (x_i + y_i) = \sum_{i=1}^{n} x_i + \sum_{i=1}^{n} y_i . \] A constant factor \(c\) that multiplies every term may be taken in front of the sum, and a sum of sums may be split into two sums. A constant added \(n\) times gives \(\sum_{i=1}^{n} c = nc\).

The sample mean

\[ \bar{x} = \frac{1}{n}\sum_{i=1}^{n} x_i \] The mean \(\bar{x}\) of a sample of \(n\) values \(x_1, \ldots, x_n\) of a quantitative variable. It carries the same units of measurement as the data, and it is the value every case would have if the total were shared out equally.

Proportion, percentage, and rate

\[ \text{proportion} = \frac{\text{count}}{\text{total}} , \qquad \text{percentage} = 100 \times \text{proportion} , \qquad \text{rate} = \frac{\text{events}}{\text{exposure}} \times \text{base} \] The total of a proportion is the whole of which the count is a part. The exposure of a rate is the number of people, houses, years, or kilometres in which the events could happen, and the base (1,000, 10,000, or 100,000) is chosen so that the rate is a readable number.

Absolute and relative change

\[ \text{absolute change} = x_{\text{new}} - x_{\text{old}} , \qquad \text{relative change} = \frac{x_{\text{new}} - x_{\text{old}}}{x_{\text{old}}} \] For two risks, a treated group with risk \(p_{\mathrm{t}}\) and a control group, which received no treatment or a dummy treatment, with risk \(p_{\mathrm{c}}\), the same two ideas are called the absolute risk reduction \(p_{\mathrm{c}} - p_{\mathrm{t}}\) and the relative risk \(p_{\mathrm{t}}/p_{\mathrm{c}}\). The number needed to treat is \(1/(p_{\mathrm{c}} - p_{\mathrm{t}})\), the number of people who must be treated for one of them to be spared the event, on average.

The logarithm and its rules

\[ \log_{10} x = y \;\Longleftrightarrow\; 10^{y} = x , \qquad \log_{10}(ab) = \log_{10} a + \log_{10} b , \qquad \log_{10}\!\left(\frac{a}{b}\right) = \log_{10} a - \log_{10} b \] The rules hold for positive numbers \(a\) and \(b\). Since \(a^{k}\) is \(a\) multiplied by itself \(k\) times, the second rule applied \(k\) times gives \(\log_{10}(a^{k}) = k \log_{10} a\). Doubling a number adds \(\log_{10} 2 = 0.301\) to its logarithm, and multiplying it by ten adds exactly 1.

Describing Distributions

Mean and median

\begin{align*} \bar{x} &= \frac{1}{n}\sum_{i=1}^{n} x_i ,\\ \text{median} &= x_{((n+1)/2)} \quad \text{for odd } n , \qquad \text{median} = \tfrac{1}{2}\bigl(x_{(n/2)} + x_{(n/2+1)}\bigr) \quad \text{for even } n , \end{align*} where \(x_{(1)} \le x_{(2)} \le \cdots \le x_{(n)}\) are the \(n\) observations sorted from smallest to largest. The mean uses every value and its size; the median uses only the order of the values.

Variance and standard deviation

\[ s^2 = \frac{1}{n-1}\sum_{i=1}^{n} (x_i - \bar{x})^2 , \qquad s = \sqrt{\frac{1}{n-1}\sum_{i=1}^{n} (x_i - \bar{x})^2} \] The sample variance \(s^2\) and the sample standard deviation \(s\) of \(n\) observations with mean \(\bar{x}\). The standard deviation is measured in the same unit as the data and is roughly the typical distance of a value from the mean. The population versions, written \(\sigma^2\) and \(\sigma\), describe a whole population or a probability distribution rather than a sample, and Unit 4 defines them.

Interquartile range and the fences for outliers

\[ \mathrm{IQR} = Q_3 - Q_1 , \qquad \text{lower fence} = Q_1 - 1.5\,\mathrm{IQR} , \qquad \text{upper fence} = Q_3 + 1.5\,\mathrm{IQR} \] A value below the lower fence or above the upper fence is marked as an outlier in a box plot. The whiskers end at the smallest and the largest value inside the fences, never at the fences themselves. The factor 1.5 is Tukey’s convention and has no deeper justification than long experience.

Standardized value

\[ z = \frac{x - \bar{x}}{s} \] The \(z\)-score of an observation \(x\) in a sample with mean \(\bar{x}\) and standard deviation \(s\) says how many standard deviations \(x\) lies above (positive \(z\)) or below (negative \(z\)) the mean. The \(z\)-scores of a whole sample have mean \(0\) and standard deviation \(1\).

Median absolute deviation

\[ \mathrm{MAD} = \operatorname{median}_{i}\, \bigl|\, x_i - \operatorname{median}_{j}(x_j) \,\bigr| \] The median of the absolute distances of the observations from their median, a measure of spread with breakdown value one half. Multiplied by \(1.4826\) it estimates the same spread as \(s\) when the data are bell-shaped.

Probability and Conditional Probability

Complement rule and addition rule

\[ \mathrm{P}(A^{c}) = 1 - \mathrm{P}(A), \qquad \mathrm{P}(A \cup B) = \mathrm{P}(A) + \mathrm{P}(B) - \mathrm{P}(A \cap B) \] The first rule gives the probability that \(A\) does not happen. The second gives the probability that at least one of \(A\) and \(B\) happens, for any two events; when they are mutually exclusive, \(\mathrm{P}(A \cap B) = 0\) and the rule reduces to the third axiom.

Binomial coefficient

\[ \binom{n}{k} = \frac{n!}{k!\,(n - k)!} \] The number of different sets of \(k\) objects that can be chosen from \(n\) distinct objects when the order of choosing does not matter. It is symmetric, \(\binom{n}{k} = \binom{n}{n - k}\), because choosing the \(k\) objects to take is the same as choosing the \(n - k\) to leave.

Conditional probability and the multiplication rule

\[ \mathrm{P}(A \mid B) = \frac{\mathrm{P}(A \cap B)}{\mathrm{P}(B)} \qquad\text{and therefore}\qquad \mathrm{P}(A \cap B) = \mathrm{P}(B)\,\mathrm{P}(A \mid B) = \mathrm{P}(A)\,\mathrm{P}(B \mid A) \] The probability of \(A\) given that \(B\) has occurred, defined for \(\mathrm{P}(B) > 0\). The second form, the multiplication rule, gives the probability that both events occur as the probability of one times the conditional probability of the other.

Law of total probability

\[ \mathrm{P}(A) = \mathrm{P}(A \mid B)\,\mathrm{P}(B) + \mathrm{P}(A \mid B^{c})\,\mathrm{P}(B^{c}) \] The overall probability of \(A\) is the weighted average of its conditional probabilities in the cases \(B\) and \(B^{c}\), weighted by the probabilities of the cases; the same holds for any split into mutually exclusive cases that together cover \(S\).

Random Variables and the Normal Distribution

Expected value and variance of a discrete random variable

\[ \mu = \mathrm{E}[X] = \sum_{x} x\,\mathrm{P}(X = x) , \qquad \sigma^2 = \mathrm{Var}(X) = \sum_{x} (x - \mu)^2\,\mathrm{P}(X = x) = \mathrm{E}[X^2] - \mu^2 \] The sums run over all possible values \(x\). The expected value is the long-run average of the variable, and the standard deviation \(\sigma = \sqrt{\mathrm{Var}(X)}\) is the typical distance of a value from that average. The last form, with \(\mathrm{E}[X^2] = \sum_x x^2\,\mathrm{P}(X = x)\), is the shortcut used in calculations.

Rules for expected values and variances

\[ \mathrm{E}[aX + b] = a\,\mathrm{E}[X] + b , \qquad \mathrm{Var}(aX + b) = a^2\,\mathrm{Var}(X) , \] \[ \mathrm{E}[X + Y] = \mathrm{E}[X] + \mathrm{E}[Y] \ \text{ always}, \qquad \mathrm{Var}(X + Y) = \mathrm{Var}(X) + \mathrm{Var}(Y) \ \text{ if $X$ and $Y$ are independent}. \] Here \(a\) and \(b\) are constants. The rules extend to any number of terms, so for \(n\) independent random variables with the same mean \(\mu\) and variance \(\sigma^2\) the sum has mean \(n\mu\) and variance \(n\sigma^2\).

The binomial distribution

\[ \mathrm{P}(X = k) = \binom{n}{k}\,p^{k}\,(1 - p)^{\,n-k} , \qquad k = 0, 1, \ldots, n , \qquad \mathrm{E}[X] = np , \qquad \mathrm{Var}(X) = np(1 - p) \] \(X \sim \mathrm{Bin}(n, p)\) counts the successes in \(n\) independent trials that each succeed with probability \(p\). The formula needs all four parts of that sentence: a fixed number of trials, two outcomes per trial, the same \(p\) throughout, and independence between trials.

The Poisson distribution

\[ \mathrm{P}(X = k) = \frac{e^{-\lambda}\,\lambda^{k}}{k!} , \qquad k = 0, 1, 2, \ldots , \qquad \mathrm{E}[X] = \lambda , \qquad \mathrm{Var}(X) = \lambda \] \(X \sim \mathrm{Poisson}(\lambda)\) counts events in a fixed interval of time or region of space when the events occur independently of one another at a constant average rate \(\lambda\) per interval. Here \(e = 2.71828\ldots\) is Euler’s number from Unit 1 and \(k! = k \times (k-1) \times \cdots \times 1\), with \(0! = 1\). The mean and the variance are equal, which is the signature of the distribution.

The normal density

\[ f(x) = \frac{1}{\sigma\sqrt{2\pi}}\,\exp\!\left(-\frac{(x - \mu)^2}{2\sigma^2}\right) , \qquad X \sim \mathrm{N}(\mu, \sigma^2) \] The density of the normal distribution with mean \(\mu\) and standard deviation \(\sigma\). The curve is symmetric about \(\mu\), highest there, and its two points of inflection, where it stops curving downwards and starts curving upwards, lie at \(\mu - \sigma\) and \(\mu + \sigma\). Probabilities are areas under it.

Standardizing a normal variable

\[ z = \frac{x - \mu}{\sigma} , \qquad \mathrm{P}(a \le X \le b) = \Phi\!\left(\frac{b - \mu}{\sigma}\right) - \Phi\!\left(\frac{a - \mu}{\sigma}\right) \] For \(X \sim \mathrm{N}(\mu, \sigma^2)\), the probability of any interval is a difference of two values of the standard normal distribution function \(\Phi\), read from Table 2. For a one-sided question, \(\mathrm{P}(X > b) = 1 - \Phi\big((b - \mu)/\sigma\big)\).

Sampling and the Central Limit Theorem

Mean and standard error of a sample mean

\[ \mathrm{E}[\bar{X}] = \mu , \qquad \mathrm{SD}(\bar{X}) = \frac{\sigma}{\sqrt{n}} = \mathrm{SE}(\bar{x}) \] For a simple random sample of \(n\) independent draws from a population with mean \(\mu\) and standard deviation \(\sigma\), the sample mean \(\bar{X}\) has expected value \(\mu\) and standard deviation \(\sigma/\sqrt{n}\). This standard deviation of the sampling distribution is called the standard error of the mean. When \(\sigma\) is unknown, which is almost always, the sample standard deviation \(s\) takes its place and the result \(s/\sqrt{n}\) is the estimated standard error.

The sample size for a given standard error

\[ \frac{\mathrm{SE}_2}{\mathrm{SE}_1} = \sqrt{\frac{n_1}{n_2}} \qquad\text{and therefore}\qquad n_2 = n_1 \left(\frac{\mathrm{SE}_1}{\mathrm{SE}_2}\right)^{2} \] If a sample of size \(n_1\) gives the standard error \(\mathrm{SE}_1\), a sample of size \(n_2\) from the same population gives \(\mathrm{SE}_2\). To divide the standard error by a factor \(k\), the sample must be \(k^2\) times as large.

Standardizing a sample mean

\[ z = \frac{\bar{x} - \mu}{\sigma/\sqrt{n}} \] The number of standard errors by which an observed sample mean \(\bar{x}\) lies above or below the population mean. By the central limit theorem, \(z\) follows approximately the standard normal distribution when \(n\) is large enough for the shape of the population, so the probability of a \(z\) beyond \(\pm 2\) is about 5 per cent and beyond \(\pm 3\) about 0.3 per cent.

The sampling distribution of a proportion

\[ \mathrm{E}[\hat{p}] = p , \qquad \mathrm{SE}(\hat{p}) = \sqrt{\frac{p(1 - p)}{n}} , \qquad \hat{p} \;\approx\; \mathrm{N}\!\left(p,\ \frac{p(1 - p)}{n}\right) \] For a simple random sample of size \(n\) from a population in which a proportion \(p\) has the property, the sample proportion \(\hat{p}\) is unbiased, has the standard error \(\sqrt{p(1 - p)/n}\), and is approximately normal when \(n\) is large enough; the usual condition is that both \(np\) and \(n(1 - p)\) are at least 10. For \(p\) near one half the standard error is close to \(0.5/\sqrt{n}\).

Estimation and Confidence Intervals

Standard error of the sample mean

\[ \mathrm{SE}(\bar{x}) = \frac{\sigma}{\sqrt{n}} \approx \frac{s}{\sqrt{n}} \] The standard deviation of the sample mean over repeated simple random samples of size \(n\) from a population with standard deviation \(\sigma\). When \(\sigma\) is unknown, the sample standard deviation \(s\) is used in its place, and the result is called the estimated standard error.

Confidence interval for a mean, \(\sigma\) known

\[ \bar{x} \pm z_{0.975}\,\frac{\sigma}{\sqrt{n}} , \qquad z_{0.975} = 1.96 \] A 95 per cent confidence interval for the population mean \(\mu\) from a simple random sample of size \(n\), when the population standard deviation \(\sigma\) is known and \(n\) is large enough for the central limit theorem (or the population is normal). For 90 per cent confidence replace 1.96 by \(z_{0.95} = 1.645\), for 99 per cent by \(z_{0.995} = 2.58\).

Confidence interval for a mean, \(\sigma\) estimated

\[ \bar{x} \pm t_{0.975,\,n-1}\,\frac{s}{\sqrt{n}} \] A 95 per cent confidence interval for \(\mu\) from a simple random sample of size \(n\) with sample standard deviation \(s\). Exact when the population is normal; a good approximation for other shapes once \(n\) reaches a few dozen, unless the data contain extreme outliers or strong skewness. For \(n \ge 30\) the quantile is within 0.1 of 1.96.

Confidence interval for a proportion

\[ \hat{p} \pm 1.96\,\sqrt{\frac{\hat{p}(1 - \hat{p})}{n}} \] A 95 per cent confidence interval for the population proportion \(p\) from a simple random sample of size \(n\) with sample proportion \(\hat{p}\). The approximation is adequate when the sample contains at least 10 successes and at least 10 failures; for smaller counts use the Wilson interval of the derivation below. This formula is known as the Wald interval.

Sample size for a given margin of error

\[ n = \left(\frac{1.96\,\sigma}{m}\right)^{2} \ \text{for a mean}, \qquad n = \left(\frac{1.96}{2m}\right)^{2} \ \text{for a proportion near one half} \] The sample size of a simple random sample whose 95 per cent margin of error is \(m\) (measured like the data for a mean, as a fraction for a proportion). The first formula needs a guess of \(\sigma\) from a pilot study or an earlier survey; the second takes \(p = 0.5\), the worst case, for which \(\sqrt{p(1 - p)/n}\) becomes \(1/(2\sqrt{n})\), and gives slightly more than is needed for any other \(p\). Round \(n\) up to the next whole number.

Confidence interval for a paired difference

\[ \bar{d} \pm t_{0.975,\,n-1}\,\frac{s_d}{\sqrt{n}} \] A 95 per cent confidence interval for the mean difference \(\mu_d\) between two measurements on the same \(n\) subjects, where \(\bar{d}\) and \(s_d\) are the mean and the sample standard deviation of the \(n\) differences. The differences are assumed to be a simple random sample from a population of differences that is roughly normal, or \(n\) is large.

Hypothesis Tests and p-Values

One-sample z and t statistics

\[ z = \frac{\bar{x} - \mu_0}{\sigma/\sqrt{n}} , \qquad t = \frac{\bar{x} - \mu_0}{s/\sqrt{n}} \] The distance of the sample mean \(\bar{x}\) from the value \(\mu_0\) stated by the null hypothesis, measured in standard errors. Under \(H_0\), \(z\) is approximately standard normal and \(t\) follows Student’s \(t\) distribution with \(n - 1\) degrees of freedom. Both require a simple random sample and, for small \(n\), a population that is not far from normal.

Two-sample t statistic (Welch’s form)

\[ t = \frac{\bar{x}_1 - \bar{x}_2}{\sqrt{\dfrac{s_1^2}{n_1} + \dfrac{s_2^2}{n_2}}} \] The difference of two independent sample means measured in standard errors, with the sample standard deviations \(s_1\) and \(s_2\) and sample sizes \(n_1\) and \(n_2\). Under \(H_0\colon \mu_1 = \mu_2\) it follows a \(t\) distribution with the degrees of freedom \(\nu\) of the derivation below, approximately. It does not assume that the two populations have the same spread.

Sample size for a two-group comparison

\[ n = \frac{2\,(z_{1-\alpha/2} + z_{1-\beta})^{2}\,\sigma^{2}}{\delta^{2}} \quad\text{per group} \] The number of observations in each of two equal groups needed to detect a true difference of means \(\delta\) with power \(1 - \beta\) in a two-sided test at level \(\alpha\), when the measurements have standard deviation \(\sigma\). With \(\alpha = 0.05\) and a power of 80 per cent, \(z_{0.975} = 1.96\) and \(z_{0.80} = 0.842\), and the numerator is \(2 \times 7.85\,\sigma^2 = 15.7\,\sigma^2\).

Two-proportion z test

\[ z = \frac{\hat{p}_1 - \hat{p}_2}{\sqrt{\hat{p}(1 - \hat{p})\left(\dfrac{1}{n_1} + \dfrac{1}{n_2}\right)}} , \qquad \hat{p} = \frac{x_1 + x_2}{n_1 + n_2} \] The difference of two sample proportions from independent samples, measured in standard errors under \(H_0\colon p_1 = p_2\), where \(x_1\) and \(x_2\) are the numbers of successes. Under \(H_0\), \(z\) is approximately standard normal when each sample has at least about ten successes and ten failures.

The chi-squared statistic

\[ \chi^2 = \sum_{\text{all cells}} \frac{(O - E)^{2}}{E} \] \(O\) is the observed count in a cell and \(E\) the count expected under \(H_0\), summed over all cells. Under \(H_0\) the statistic approximately follows the chi-squared distribution with \(\nu\) degrees of freedom, given below for each use, when every expected count is at least about 5. Only large values count against \(H_0\), so the p-value is the upper tail.

Correlation, Regression, and Causation

Correlation coefficient

\[ r = \frac{1}{n - 1}\sum_{i=1}^{n} \frac{x_i - \bar{x}}{s_x}\cdot\frac{y_i - \bar{y}}{s_y} = \frac{s_{xy}}{s_x\,s_y} = \frac{\sum (x_i - \bar{x})(y_i - \bar{y})}{\sqrt{\sum (x_i - \bar{x})^2\,\sum (y_i - \bar{y})^2}} \] The correlation coefficient of \(n\) pairs \((x_i, y_i)\) with means \(\bar{x}\), \(\bar{y}\) and standard deviations \(s_x\), \(s_y\). It measures the strength and direction of a linear relation and lies between \(-1\) and \(1\). The third form is the one used for a calculation by hand, because the factors of \(1/(n - 1)\) in the covariance and in the two standard deviations cancel out and need not be computed.

Least-squares slope and intercept

\[ \hat{\beta}_1 = \frac{\sum (x_i - \bar{x})(y_i - \bar{y})}{\sum (x_i - \bar{x})^2} = r\,\frac{s_y}{s_x} , \qquad \hat{\beta}_0 = \bar{y} - \hat{\beta}_1\,\bar{x} \] The slope and intercept of the line that minimizes the sum of squared residuals, for \(n\) pairs with correlation \(r\) and standard deviations \(s_x\) and \(s_y\). The line always passes through the point of the means \((\bar{x}, \bar{y})\), and its slope is the correlation rescaled from the units of \(x\) to the units of \(y\).

Coefficient of determination

\[ R^2 = 1 - \frac{\sum e_i^2}{\sum (y_i - \bar{y})^2} = r^2 \] The fraction of the variance of \(y\) that the least-squares line accounts for, between 0 and 1. The equality \(R^2 = r^2\) holds for a line with one predictor; the first form is the general definition and survives when there are several.

Standard error and t statistic of the slope

\[ s_e = \sqrt{\frac{\sum e_i^2}{n - 2}} , \qquad \mathrm{SE}(\hat{\beta}_1) = \frac{s_e}{\sqrt{\sum (x_i - \bar{x})^2}} , \qquad t = \frac{\hat{\beta}_1}{\mathrm{SE}(\hat{\beta}_1)} \] The residual standard deviation \(s_e\), the standard error of the slope, and the t statistic for the null hypothesis \(H_0\colon \beta_1 = 0\) of no linear relation, compared with Student’s t distribution with \(n - 2\) degrees of freedom. A 95 per cent confidence interval for the slope is \(\hat{\beta}_1 \pm t_{0.975,\,n-2}\,\mathrm{SE}(\hat{\beta}_1)\). Both assume that the errors \(\varepsilon_i\) of Equation  are independent, have the same spread at every \(x\), and are roughly normal or that \(n\) is large.

Regression to the mean

\[ \hat{z}_y = r\,z_x \] The predicted standardized value of \(y\) is the standardized value of \(x\) multiplied by the correlation. Because \(|r| < 1\) whenever the relation is not perfect, the prediction is always less extreme, in standard deviations, than the value it is based on.

Likelihood and Maximum Likelihood

Likelihood and log-likelihood

\[ L(\theta) = \prod_{i=1}^{n} f(x_i \mid \theta) , \qquad \ell(\theta) = \ln L(\theta) = \sum_{i=1}^{n} \ln f(x_i \mid \theta) \] The likelihood of the parameter \(\theta\) from \(n\) independent observations \(x_1, \ldots, x_n\), where \(f(x \mid \theta)\) is the probability mass function of a count or the density of a measurement under the model, and the log-likelihood \(\ell(\theta)\) is its natural logarithm. Both are functions of \(\theta\) with the data fixed, and both are defined up to an additive or multiplicative constant that does not involve \(\theta\).

Finding a maximum with the derivative

\[ f'(\hat{x}) = 0 \qquad\text{and}\qquad f''(\hat{x}) < 0 \] A smooth function \(f\), one whose graph has no jumps and no corners, has a maximum at \(\hat{x}\) when its derivative is zero there and its second derivative is negative. A zero derivative with a positive second derivative marks a minimum. The rule finds the highest point of each hump; a function with several humps needs the humps compared, and a maximum at the edge of the allowed range, such as \(p = 0\) or \(p = 1\) for a proportion, is not found this way and must be checked separately.

Maximum likelihood estimates of the three standard models

\[ \hat{p} = \frac{k}{n} , \qquad \hat{\lambda} = \bar{x} , \qquad \hat{\mu} = \bar{x} , \qquad \hat{\sigma}^2 = \frac{1}{n}\sum_{i=1}^{n}(x_i - \bar{x})^2 \] For \(k\) successes in \(n\) independent trials, for \(n\) independent Poisson counts, and for \(n\) independent normal measurements. The variance estimate divides by \(n\) and is smaller than the sample variance \(s^2\) of Unit 2 by the factor \((n - 1)/n\).

Observed information and the standard error

\[ I(\hat{\theta}) = -\ell''(\hat{\theta}) , \qquad \mathrm{SE}(\hat{\theta}) \approx \frac{1}{\sqrt{I(\hat{\theta})}} \] The observed information is the curvature of the log-likelihood at its maximum, and its reciprocal is the approximate variance of the maximum likelihood estimate in a large sample from a regular model. The approximate 95 per cent interval is \(\hat{\theta} \pm 1.96\,\mathrm{SE}\), as in Unit 6.

The likelihood ratio statistic

\[ W = 2\big[\ell(\hat{\theta}) - \ell(\theta_0)\big] \;\approx\; \chi^2_1 \quad\text{if } \theta = \theta_0 \] Twice the drop in the log-likelihood from its maximum to the hypothesized value is approximately chi-squared with one degree of freedom when the hypothesis is true, in a large sample from a regular model (Wilks’s theorem). The hypothesis is rejected at the 5 per cent level when \(W > 3.84\), the 95 per cent point of \(\chi^2_1\), which is \(1.96^2\). When several parameters are fixed at once, the degrees of freedom equal their number.

Multiple Regression and Linear Models

The multiple regression model and its coefficients

\[ \mathrm{E}[y \mid x_1, \ldots, x_k] = \beta_0 + \beta_1 x_1 + \cdots + \beta_k x_k \] The expected response is a sum of straight-line contributions, one per predictor. The coefficient \(\beta_j\) is the change in the expected response for an increase of one in \(x_j\) while every other predictor is held fixed. It is estimated by least squares as \(\hat{\beta}_j\), and it is a different quantity from the slope of \(y\) on \(x_j\) alone.

Standard error, \(t\) test, and confidence interval of a coefficient

\[ s_e = \sqrt{\frac{\sum e_i^2}{n - k - 1}} , \qquad t_j = \frac{\hat{\beta}_j}{\mathrm{SE}(\hat{\beta}_j)} , \qquad \hat{\beta}_j \pm t_{0.975,\,n-k-1}\,\mathrm{SE}(\hat{\beta}_j) \] From left to right, these are the residual standard deviation, the \(t\) statistic for the null hypothesis \(H_0\colon \beta_j = 0\) that predictor \(j\) adds nothing once the other \(k - 1\) predictors are in the model, and the 95 per cent confidence interval for \(\beta_j\). Under the assumptions of Section 2, \(t_j\) follows Student’s \(t\) distribution with \(n - k - 1\) degrees of freedom when \(H_0\) is true. The standard error of \(\hat{\beta}_j\) is \(s_e\) times the square root of the \(j\)-th diagonal entry of \((\mathbf{X}^{\top}\mathbf{X})^{-1}\), which software computes.

\(R^2\) and adjusted \(R^2\)

\[ R^2 = 1 - \frac{\mathrm{SS}_{\text{res}}}{\mathrm{SS}_{\text{tot}}} , \qquad R^2_{\text{adj}} = 1 - \frac{\mathrm{SS}_{\text{res}}/(n - k - 1)}{\mathrm{SS}_{\text{tot}}/(n - 1)} \] The fraction of the variance of \(y\) accounted for by the fitted model, with \(\mathrm{SS}_{\text{res}} = \sum e_i^2\) and \(\mathrm{SS}_{\text{tot}} = \sum (y_i - \bar{y})^2\). The adjusted version rises when a new predictor reduces the residual sum of squares by more than its share of the degrees of freedom and falls otherwise, so it can be used to compare models with different numbers of predictors. For one predictor \(R^2 = r^2\); with several, \(R^2\) is the squared correlation between \(y\) and \(\hat{y}\).

The \(F\) test of a regression

\[ F = \frac{(\mathrm{SS}_{\text{tot}} - \mathrm{SS}_{\text{res}})/k}{\mathrm{SS}_{\text{res}}/(n - k - 1)} \] The statistic for the null hypothesis \(H_0\colon \beta_1 = \beta_2 = \cdots = \beta_k = 0\) that none of the predictors matters. Under the assumptions of Section 2 and under \(H_0\) it follows the \(F\) distribution with \(k\) and \(n - k - 1\) degrees of freedom, and \(H_0\) is rejected for large values. The numerator is the regression mean square and the denominator is \(s_e^2\), the residual mean square.

Confidence and prediction intervals for a new case

\[ \hat{y}_0 \pm t_{0.975,\,n-k-1}\,\mathrm{SE}(\hat{y}_0) \qquad\text{and}\qquad \hat{y}_0 \pm t_{0.975,\,n-k-1}\,\sqrt{s_e^2 + \mathrm{SE}(\hat{y}_0)^2} \] The 95 per cent confidence interval for the expected response at the predictor values \(x_0\), and the 95 per cent prediction interval for a single new observation there, where \(\hat{y}_0\) is the fitted value at \(x_0\). The standard error of the fitted value is \(\mathrm{SE}(\hat{y}_0) = s_e\sqrt{h_0}\), with \(h_0 = \mathbf{x}_0^{\top}(\mathbf{X}^{\top}\mathbf{X})^{-1}\mathbf{x}_0\) the leverage of the new point; it is smallest near the centre of the data and grows as \(x_0\) moves away. The prediction interval can never be narrower than about \(\pm 2 s_e\), however large the sample.

Logistic Regression and Generalized Linear Models

Odds and log odds

\[ \text{odds} = \frac{p}{1 - p} , \qquad \text{logit}(p) = \ln\!\left(\frac{p}{1 - p}\right) , \qquad p = \frac{e^{\eta}}{1 + e^{\eta}} = \frac{1}{1 + e^{-\eta}} \] The odds and the log odds of an event of probability \(p\), and the probability recovered from a log odds \(\eta\). A probability of one half has odds 1 and log odds 0; \(p\) near 0 has large negative log odds, \(p\) near 1 large positive ones.

Logistic regression

\[ \ln\!\left(\frac{p}{1 - p}\right) = \beta_0 + \beta_1 x \qquad\Longleftrightarrow\qquad \mathrm{P}(Y = 1) = \frac{e^{\beta_0 + \beta_1 x}}{1 + e^{\beta_0 + \beta_1 x}} \] The probability of the event, \(p = \mathrm{P}(Y = 1)\), as a function of the predictor \(x\). The intercept \(\beta_0\) is the log odds at \(x = 0\), the slope \(\beta_1\) is the change in the log odds for an increase of one in \(x\), and the probability passes one half at \(x = -\beta_0/\beta_1\), where the log odds are zero. With several predictors the right side is \(\beta_0 + \beta_1 x_1 + \cdots + \beta_k x_k\), as in Unit 10.

Coefficients as odds ratios

\[ \mathrm{OR} = e^{\beta_1} \quad\text{per increase of one in } x , \qquad \mathrm{OR} = e^{c\,\beta_1} \quad\text{per increase of } c \] The odds ratio for an increase of one in the predictor is the exponential of the slope, and for an increase of \(c\) it is the exponential of \(c\) times the slope. A 95 per cent interval for \(\mathrm{OR}\) is found by taking exponentials of \(\hat{\beta}_1 \pm 1.96\,\mathrm{SE}(\hat{\beta}_1)\); it is never symmetric about the estimate.

Poisson regression with an offset

\[ \ln \lambda_i = \ln E_i + \beta_0 + \beta_1 x_i , \qquad \text{rate ratio} = e^{\beta_1} \] The expected count \(\lambda_i\) of a Poisson response, for an exposure \(E_i\) (a time, an area, or a number of people at risk) and a predictor \(x_i\). The offset \(\ln E_i\) has no coefficient to estimate, and \(e^{\beta_1}\) is the factor by which the rate, the count per patient-year, per kilometre, or per policy, is multiplied for an increase of one in \(x\). When all exposures are equal, the offset is a constant and can be dropped.

Deviance and the likelihood ratio test

\[ D = 2\bigl[\ell_{\text{sat}} - \ell_{\text{model}}\bigr] , \qquad D_{\text{small}} - D_{\text{large}} = 2\bigl[\ell_{\text{large}} - \ell_{\text{small}}\bigr] \;\approx\; \chi^2_{q} \] The deviance of a fitted generalized linear model, and the test of a smaller model inside a larger one with \(q\) more parameters: the drop in the deviance is compared with the chi-squared distribution with \(q\) degrees of freedom, and the 95 per cent point for \(q = 1\) is 3.84. The deviance of a single model for binary data is no test of its fit on its own.

Bayesian Inference

Bayes’ theorem for a parameter

\[ f(\theta \mid x) = \frac{f(x \mid \theta)\, f(\theta)}{f(x)} \propto f(x \mid \theta)\, f(\theta) \] The posterior density of the parameter \(\theta\) given the data \(x\) is the likelihood times the prior, divided by the marginal probability of the data, \(f(x) = \int f(x \mid \theta)\, f(\theta)\, \dd\theta\), which the text below calls the marginal likelihood. Since \(f(x)\) does not depend on \(\theta\), the posterior is proportional to likelihood times prior, and the missing constant is recovered at the end by requiring that the posterior has total area one.

Beta-binomial posterior

\[ \theta \mid k \sim \mathrm{Beta}(a + k,\, b + n - k), \qquad \mathrm{E}[\theta \mid k] = \frac{a + k}{a + b + n} = w\,\frac{k}{n} + (1 - w)\,\frac{a}{a + b}, \quad w = \frac{n}{a + b + n} \] A Beta(\(a\), \(b\)) prior and \(k\) successes in \(n\) trials give a Beta posterior whose parameters are the prior pseudo-counts plus the observed counts. The posterior mean is a compromise between the sample proportion and the prior mean, with the weight \(w\) on the data growing towards one as \(n\) grows.

Normal-normal posterior for a mean

\[ \mu \mid \bar{x} \sim \mathrm{N}\!\left(\frac{\mu_0/\tau^2 + n\bar{x}/\sigma^2}{1/\tau^2 + n/\sigma^2},\;\; \frac{1}{1/\tau^2 + n/\sigma^2}\right) \] With a normal prior \(\mathrm{N}(\mu_0, \tau^2)\) and \(n\) normal observations of known standard deviation \(\sigma\) and mean \(\bar{x}\), the posterior precision is the prior precision \(1/\tau^2\) plus the precision of the data \(n/\sigma^2\), and the posterior mean is the average of \(\mu_0\) and \(\bar{x}\) weighted by these two precisions.

Posterior predictive probability of a success

\[ \mathrm{P}(\text{next trial succeeds} \mid k \text{ successes in } n) = \int_0^1 \theta\, f(\theta \mid k)\, \dd\theta = \frac{a + k}{a + b + n} \] Under a \(\mathrm{Beta}(a, b)\) prior the probability that the next trial succeeds is the posterior mean of \(\theta\). With the flat prior it is \((k + 1)/(n + 2)\), Laplace’s rule of succession.

Metropolis acceptance rule

\[ \mathrm{P}(\text{accept } \theta') = \min\!\left(1,\; \frac{f(\theta' \mid x)}{f(\theta_t \mid x)}\right) = \min\!\left(1,\; \frac{f(x \mid \theta')\,f(\theta')}{f(x \mid \theta_t)\,f(\theta_t)}\right) \] A proposed value is always accepted when it has higher posterior density than the current one and accepted with probability equal to the ratio of the densities when it has lower. The marginal likelihood cancels in the ratio, so the posterior need be known only up to a constant.

The Bootstrap and Cross-Validation

Training error and test error

\[ \mathrm{MSE}_{\mathrm{train}} = \frac{1}{n}\sum_{i=1}^{n}\bigl(y_i - \hat{f}(x_i)\bigr)^2 , \qquad \mathrm{MSE}_{\mathrm{test}} = \frac{1}{m}\sum_{j=1}^{m}\bigl(y_j^{\mathrm{new}} - \hat{f}(x_j^{\mathrm{new}})\bigr)^2 \] The training error is the mean squared error of the rule on the \(n\) observations it was fitted to; the test error is its mean squared error on \(m\) new observations that played no part in the fit. For a categorical response the squared error is replaced by the count of wrong predictions, and both become misclassification rates.

Bootstrap standard error and percentile interval

\[ \widehat{\mathrm{SE}}_{\mathrm{boot}} = \sqrt{\frac{1}{B - 1}\sum_{b=1}^{B}\bigl(\hat{\theta}^{*}_b - \bar{\theta}^{*}\bigr)^2} , \qquad \bar{\theta}^{*} = \frac{1}{B}\sum_{b=1}^{B}\hat{\theta}^{*}_b \] The bootstrap standard error of a statistic \(\hat{\theta}\) is the standard deviation of its values \(\hat{\theta}^{*}_b\) over \(B\) bootstrap samples. The 95 per cent percentile interval runs from the 2.5th to the 97.5th percentile of the \(B\) values, that is, from the 25th to the 975th of 1,000 sorted values. About \(B = 200\) bootstrap samples are enough for a standard error, and \(B = 1{,}000\) or more are needed for an interval, because the ends of an interval depend on the few values in the tails of the bootstrap distribution, and those take more samples to pin down.

Observations in a bootstrap sample

\[ \mathrm{P}(\text{a given observation appears}) = 1 - \Bigl(1 - \frac{1}{n}\Bigr)^{n} \;\longrightarrow\; 1 - e^{-1} = 0.632 \] A bootstrap sample of size \(n\) from \(n\) observations contains, on average, \(63.2\) per cent of the distinct observations for large \(n\), and leaves \(36.8\) per cent out. For small samples the share is a little larger: \(75.0\) per cent for \(n = 2\), \(67.2\) for \(n = 5\), \(65.1\) for \(n = 10\), \(64.2\) for \(n = 20\), and \(63.4\) for \(n = 100\).

The \(k\)-fold cross-validation estimate

\[ \mathrm{CV}_k = \frac{1}{k}\sum_{j=1}^{k}\mathrm{MSE}_j , \qquad \mathrm{SE}(\mathrm{CV}_k) \approx \frac{s_{\mathrm{MSE}}}{\sqrt{k}} \] The cross-validation estimate of the test error of a model is the average of the \(k\) test errors \(\mathrm{MSE}_j\) of the folds. Its standard error is estimated from the standard deviation \(s_{\mathrm{MSE}}\) of the \(k\) fold errors as if they were independent, which they are not quite, since the training sets overlap; the approximation is the one the field uses. With \(k = n\) every fold is a single observation and the method is called leave-one-out cross-validation.

The permutation p-value

\[ \text{p-value} = \frac{\text{number of relabellings whose statistic is at least as extreme as the observed one}}{\text{number of relabellings}} \] Under the null hypothesis that the labels carry no information, every relabelling of the data was as likely as the observed one, and the p-value counts the fraction that are at least as extreme. When the relabellings are too many to list, \(B\) of them are drawn at random, and the p-value is estimated as \((b + 1)/(B + 1)\), where \(b\) is the number of random relabellings at least as extreme as the observed one; the added 1 counts the observed relabelling itself.

Ridge, Lasso, and Regularization

Ridge regression

\[ \hat{\boldsymbol{\beta}}^{\mathrm{ridge}} \text{ minimizes } \sum_{i=1}^{n}\Bigl(y_i - \sum_{j=1}^{p}\beta_j x_{ij}\Bigr)^2 + \lambda\sum_{j=1}^{p}\beta_j^2 , \qquad \hat{\boldsymbol{\beta}}^{\mathrm{ridge}} = (\mathbf{X}^{\top}\mathbf{X} + \lambda\mathbf{I})^{-1}\mathbf{X}^{\top}\mathbf{y} \] The predictors \(x_{ij}\) are standardized and the response is centred, so there is no intercept. The penalty parameter \(\lambda \ge 0\) sets the price of large coefficients: \(\lambda = 0\) is least squares, and as \(\lambda\) grows every coefficient is pulled towards zero. The closed form adds \(\lambda\) to the diagonal of \(\mathbf{X}^{\top}\mathbf{X}\) before inverting, and \(\mathbf{I}\) is the identity matrix with ones on the diagonal and zeros elsewhere.

The lasso

\[ \hat{\boldsymbol{\beta}}^{\mathrm{lasso}} \text{ minimizes } \frac{1}{2}\sum_{i=1}^{n}\Bigl(y_i - \sum_{j=1}^{p}\beta_j x_{ij}\Bigr)^2 + \lambda\sum_{j=1}^{p}|\beta_j| \] The predictors are standardized and the response is centred, as for ridge regression. The penalty \(\lambda\sum_j|\beta_j|\) shrinks the coefficients towards zero and, unlike the ridge penalty, sets some of them exactly to zero, so that the lasso selects predictors at the same time as it estimates. The factor one half in front of the sum of squares is a convention that keeps the rule of Section 3.3 free of a factor 2; books that omit it obtain the same estimates with \(\lambda\) twice as large. There is no closed form, and the solution is found numerically.

Soft thresholding

\[ \hat{\beta}_j^{\mathrm{lasso}} = \mathrm{sign}(\hat{\beta}_j)\,\bigl(|\hat{\beta}_j| - \lambda\bigr)_{+} \] For uncorrelated standardized predictors, the lasso coefficient is the least-squares coefficient \(\hat{\beta}_j\) moved towards zero by \(\lambda\), and set to zero if it was within \(\lambda\) of zero to begin with. The notation \((u)_{+}\) means \(u\) when \(u\) is positive and \(0\) otherwise. In the same case ridge regression gives \(\hat{\beta}_j/(1 + \lambda)\): ridge divides, the lasso subtracts.

The elastic net penalty

\[ \lambda\Bigl[\alpha\sum_{j=1}^{p}|\beta_j| + (1 - \alpha)\sum_{j=1}^{p}\beta_j^2\Bigr] \] The mixing parameter \(\alpha\) runs from \(0\) (ridge regression) to \(1\) (the lasso); values between keep the lasso’s ability to set coefficients to zero and the ridge’s tendency to keep correlated predictors together. Both \(\lambda\) and \(\alpha\) are chosen by cross-validation.

Priors and penalties

\[ \beta_j \sim \mathrm{N}(0, \tau^2) \;\Longleftrightarrow\; \text{ridge with } \lambda = \frac{\sigma^2}{\tau^2} , \qquad f(\beta_j) = \frac{1}{2b}\,e^{-|\beta_j|/b} \;\Longleftrightarrow\; \text{lasso with } \lambda = \frac{\sigma^2}{b} \] A normal prior on the coefficients makes the posterior mode the ridge estimate, and a Laplace prior makes it the lasso estimate. A tight prior (small \(\tau^2\) or \(b\)) is a large penalty, and a vague prior is a small one; \(\sigma^2\) is the error variance of the regression.

Trees, Random Forests, and Boosting

The split of a regression tree

\[ \mathrm{RSS}(t) = \sum_{i:\, x_i \le t}\bigl(y_i - \bar{y}_{\mathrm{L}}\bigr)^2 + \sum_{i:\, x_i > t}\bigl(y_i - \bar{y}_{\mathrm{R}}\bigr)^2 \] For a candidate cut point \(t\) of a predictor \(x\), the residual sum of squares is computed with the mean \(\bar{y}_{\mathrm{L}}\) of the responses on the left (\(x_i \le t\)) and the mean \(\bar{y}_{\mathrm{R}}\) of those on the right. The tree chooses the predictor and the cut point with the smallest \(\mathrm{RSS}(t)\), and then repeats the search inside each of the two regions.

Gini index and entropy of a node

\[ G_m = \sum_{k}\hat{p}_{mk}\bigl(1 - \hat{p}_{mk}\bigr) , \qquad D_m = -\sum_{k}\hat{p}_{mk}\ln\hat{p}_{mk} \] The Gini index \(G_m\) and the entropy \(D_m\) of node \(m\) are both zero for a pure node and largest when the classes are equally frequent; with two classes the maxima are \(1/2\) and \(\ln 2 = 0.693\). A split is judged by the weighted average of the impurity of its two children, each weighted by its share of the cases, and the split that lowers this average most is chosen. The term \(\hat{p}\ln\hat{p}\) is taken as zero when \(\hat{p} = 0\), because the product tends to zero as \(\hat{p}\) does, although \(\ln 0\) itself is not defined.

The variance of an average of correlated predictions

\[ \mathrm{Var}\!\left(\frac{1}{B}\sum_{b=1}^{B}\hat{f}_b(x)\right) = \rho\,\sigma^2 + \frac{1 - \rho}{B}\,\sigma^2 \] If each of \(B\) predictions \(\hat{f}_b(x)\) has variance \(\sigma^2\) and every pair of them has correlation \(\rho\), the variance of their average falls towards \(\rho\sigma^2\) as \(B\) grows and never below it. With \(\rho = 0\) it falls to zero like \(\sigma^2/B\); with \(\rho = 1\) averaging achieves nothing.

The boosting update

\[ \hat{f}(x) \;\leftarrow\; \hat{f}(x) + \nu\,\hat{h}_b(x) , \qquad \hat{f}(x) = \bar{y} + \nu\sum_{b=1}^{B}\hat{h}_b(x) \] Each new tree \(\hat{h}_b\) is fitted to the residuals of the model so far, and the learning rate \(\nu\) (the Greek letter nu, between 0 and 1) decides how much of it is added. A small \(\nu\) makes each step cautious and requires more trees \(B\); the two are tuned together, and \(B\) is chosen by cross-validation or on a validation set, because unlike a forest a boosted model does overfit when \(B\) grows too large.

Multiple Testing and the False Discovery Rate

The Bonferroni correction

\[ \mathrm{FWER} \le m_0\,\alpha_{\mathrm{each}} \le m\,\alpha_{\mathrm{each}} , \qquad\text{so test each hypothesis at}\qquad \alpha_{\mathrm{each}} = \frac{\alpha}{m} . \] Here \(\alpha_{\mathrm{each}}\) is the level at which each single test is run. With \(m\) tests, of which \(m_0\) concern true null hypotheses, running each test at the level \(\alpha/m\) keeps the family-wise error rate at or below \(\alpha\). Equivalently, a p-value counts as significant when \(m\) times the p-value is below \(\alpha\). The bound holds for any dependence between the tests.

The false discovery rate

\[ \mathrm{FDR} = \mathrm{E}\!\left[\frac{V}{R}\right] , \qquad\text{with } \frac{V}{R} = 0 \text{ when } R = 0 , \] where \(V\) is the number of false discoveries and \(R\) the number of discoveries in Table 1. The ratio \(V/R\) is the false discovery proportion of one study; the false discovery rate is its expectation over repetitions of the whole study. The family-wise error rate is \(\mathrm{P}(V \ge 1)\).

The Benjamini-Hochberg procedure

\[ \text{find the largest } i \text{ with } p_{(i)} \le \frac{i}{m}\,q , \qquad\text{and reject } H_{(1)}, \ldots, H_{(i)} . \] The \(m\) p-values are sorted from the smallest to the largest, \(p_{(1)} \le \ldots \le p_{(m)}\), and \(q\) is the false discovery rate to be controlled, usually 0.05 or 0.10. Every hypothesis up to the largest \(i\) that satisfies the inequality is rejected, including any whose own p-value exceeds its threshold \(iq/m\).

Expected false positives and the fraction of true nulls

\[ \mathrm{E}[V] = m_0\,\alpha_{\mathrm{each}} , \qquad \hat{\pi}_0 = \frac{\text{number of p-values above } \lambda}{(1 - \lambda)\,m} \quad\text{with } \lambda = 0.5 . \] With \(m_0\) true null hypotheses, each tested at the level \(\alpha_{\mathrm{each}}\), the expected number of false positives is \(m_0\alpha_{\mathrm{each}}\). The fraction of true nulls, \(\pi_0 = m_0/m\), is estimated from the right part of the p-value histogram, where almost only null p-values fall. A fraction \(1 - \lambda\) of the null p-values is expected above \(\lambda\), so the count above \(\lambda\) divided by \(1 - \lambda\) estimates \(m_0\), and dividing by \(m\) gives the fraction; the estimate is slightly too high, because a few false nulls have large p-values too.

Genome-wide significance

\[ \frac{0.05}{10^{6}} = 5 \times 10^{-8} \] The Bonferroni threshold for a family-wise error rate of 5 per cent over about one million independent common variants, the effective number of tests in a genome-wide association study of people of European ancestry. A variant with a p-value below it is called genome-wide significant, and a locus, the geneticists’ word for a place in the genome, is accepted once it reaches this level and is replicated in an independent sample.