Milk being poured into a cup of Darjeeling tea, the experiment of Fisher's lady tasting tea
Unit 07 · All masterclasses · 75 min

Hypothesis Tests and p-Values

Null hypotheses, test statistics, the p-value, errors of two kinds, power, and the chi-squared test
Image: cyclonebill, Wikimedia Commons, CC BY-SA 2.0

A difference in a sample can be a real effect or the ordinary wobble of chance, and a hypothesis test is the controlled way to tell the two apart. This unit builds the test on Fisher's lady tasting tea, defines the p-value exactly, carries out the z, t, paired, and two-sample tests, sets out the two kinds of error and the power of a test, plans the size of a trial, tests two proportions on Snow's water companies and tables of counts on Mendel's peas and Doll and Hill's smokers, and ends with what a p-value is not.

Preview: the beginning of the unit

Ten patients slept on average 1.58 hours longer under one drug than under another, and 5,474 round peas stood against 1,850 wrinkled ones where a theory predicted three to one. Both numbers could be the trace of a real effect, and both could be the ordinary variation of a sample that Units 5 and 6 described. A hypothesis test is the procedure statisticians use to decide between these two readings in a controlled way, with a known rate of mistakes. This unit builds the procedure on a cup of tea, applies it to means, proportions, and tables of counts, and ends with what its number, the p-value, can and cannot say.

1 The logic of a test

1.1 Null hypothesis, test statistic, and p-value

Unit 3 counted the ways of choosing 4 cups out of 8 and found 70. The count was part of an experiment that Ronald Fisher (1890-1962) described in The Design of Experiments (1935), in which a lady claims that she can taste whether the milk or the tea was poured into the cup first. Fisher’s design presents her with 8 cups, 4 of each kind in random order, tells her so, and asks her to name the 4 cups with milk first. The same experiment now serves to define the parts of a test.

The null hypothesis, written \(H_0\), is the statement that nothing is going on, made precise enough to compute with. Here it says that the lady cannot tell the difference, so that her choice of 4 cups is, as far as the outcome is concerned, a random pick out of the 70 possible choices. The alternative hypothesis \(H_1\) is the statement the experiment is looking for, which is that she can tell. The test statistic is the number computed from the data that measures how far the result is from what the null hypothesis leads one to expect, and here it is the number of correctly named cups, which can be 0, 1, 2, 3, or 4. The last ingredient is a rule for turning the observed value of the statistic into a judgement.

The procedure is known as Fisher’s exact test, exact because the probability is computed by counting rather than from an approximating curve. The quantity at its heart deserves its own definition. The p-value is the probability, computed as if the null hypothesis were true, of obtaining a test statistic at least as extreme as the one actually observed: \begin{equation} \text{p-value} = \mathrm{P}(\text{a result at least as extreme as the observed one} \mid H_0 \text{ true}) . \label{eq:pvalue} \end{equation} Every word of this sentence matters, and Section 5 returns to the words that are most often dropped. Two of its parts need attention at once. The condition in Equation \eqref{eq:pvalue} is that \(H_0\) is true; the p-value is a statement about the data given the hypothesis, in the same way that \(\mathrm{P}(B \mid A)\) in Unit 3 is a statement about \(B\) given \(A\), and it is not the probability of the hypothesis given the data. The phrase “at least as extreme” includes the observed result and everything further from \(H_0\), because a single exact value, such as exactly 3 correct cups, is not what makes a result surprising; what makes it surprising is that it lies so far out in the tail.

The two-sided p-value of an observed test statistic \(z = 2.17\) under a null distribution that is standard normal, computed from the normal density …
Figure 1. The two-sided p-value of an observed test statistic \(z = 2.17\) under a null distribution that is standard normal, computed from the normal density of Unit 4. Each shaded tail has area \(1 - \Phi(2.17) = 0.015\), so the p-value is \(0.030\). The left tail is included because, before the data were seen, a difference in either direction would have counted against \(H_0\).

For a test statistic with a continuous distribution the p-value is an area under the null distribution, as in Figure 1. When the alternative hypothesis allows a difference in either direction, both tails count and the test is two-sided; Section 2 says when a one-sided test is justified.

The full unit is part of the program

Want to see a complete unit first? Unit 5, Sampling and the Central Limit Theorem, is free to read.