Chapter 5 · Testing of Hypotheses

5.1 Most Powerful, UMP and Likelihood Ratio Tests

How to decide between two hypotheses with controlled error rates: size, power and $p$-values; the Neyman–Pearson lemma and most powerful tests; monotone likelihood ratio and uniformly most powerful one-sided tests; why two-sided UMP tests usually fail to exist; and likelihood ratio tests, with Wilks' theorem and the $z$, $t$, $\chi^2$ and $F$ tests as special cases.

A pharmaceutical regulator must decide whether a new tablet line meets the declared mean weight of $500$ mg. An engineer must decide whether a new alloy lasts longer than the old one. A geneticist must decide whether offspring counts agree with Mendel's $3:1$ ratio. In each case the data are random, so any rule that turns data into a decision will sometimes be wrong. It can reject a claim that is true, or accept one that is false. The theory of hypothesis testing does not try to avoid errors altogether, which is impossible. It fixes an upper bound $\alpha$ on the probability of the more serious error and then, among all rules that respect that bound, looks for the one that is least likely to make the other error. The question "which rule is best?" has a sharp answer when we compare two simple hypotheses: the Neyman–Pearson lemma says we should reject when the likelihood ratio is large. Much of this section consists of following that idea as far as it goes.

It goes a long way. For one-sided questions in families with a monotone likelihood ratio (every one-parameter exponential family, the uniform family, and many others), the Neyman–Pearson test does not depend on the particular alternative, and so it is uniformly most powerful. For two-sided questions it usually breaks down, and we show exactly why; one remedy is to restrict attention to unbiased tests. For problems with nuisance parameters we use the generalised likelihood ratio test, which reproduces the familiar $z$-, $t$-, $\chi^2$- and $F$-tests and, by Wilks' theorem, has an approximately $\chi^2$ null distribution in large samples. CSIR NET examines this material heavily. Part B questions ask for a size, a power, a critical value or a $p$-value. Part C questions ask whether a UMP test exists, whether a test is unbiased, whether a family has a monotone likelihood ratio, and what the null distribution of a likelihood ratio statistic is. We use the notation of Chapter 4: $X=(X_1,\dots,X_n)$ has a distribution $P_\theta$ with density or pmf $f(x;\theta)$, $\theta\in\Theta$, and $L(\theta)=f(x;\theta)$ is the likelihood. Upper $\alpha$ points are written $z_\alpha$, $t_{\nu,\alpha}$, $\chi^2_{\nu,\alpha}$ and $F_{\nu_1,\nu_2,\alpha}$ as in §4.3, and all quoted quantiles were computed with scipy.stats.

Hypotheses, Errors, Size and Power

📘 Definition 5.1 — Hypotheses, tests and critical regions

Let $\Theta$ be split into two disjoint nonempty sets, $\Theta=\Theta_0\cup\Theta_1$.

  1. The null hypothesis is $H_0:\theta\in\Theta_0$ and the alternative hypothesis is $H_1:\theta\in\Theta_1$. A hypothesis is simple if its set contains a single point, and composite otherwise.
  2. A (randomised) test is a measurable function $\phi$ of the data with values in $[0,1]$. When $x$ is observed we reject $H_0$ with probability $\phi(x)$, using an independent auxiliary randomisation (for example a uniform random number) if $0\lt\phi(x)\lt1$.
  3. A test with values in $\{0,1\}$ is non-randomised. It has the form $\phi=\mathbf 1_C$, and the set $C=\{x:\phi(x)=1\}$ is its critical region (rejection region). The complement of $C$ is the acceptance region.

Most tests are written as "reject $H_0$ when $T(x)\gt c$" for a test statistic $T$ and a critical value $c$.

Randomised tests are rarely used in practice, but they are indispensable in the theory. With discrete data the probability of a critical region can take only countably many values, and randomisation fills the gaps, so that every size $\alpha$ can be attained exactly. They also make the set of tests convex, which is what makes the optimisation problems below solvable.

📘 Definition 5.2 — Errors, power function, size and level
  1. Rejecting $H_0$ when it is true is a type I error. Accepting $H_0$ when it is false is a type II error.
  2. The power function of a test $\phi$ is $$\beta_\phi(\theta)=E_\theta[\phi(X)]=P_\theta(\text{reject }H_0),\qquad\theta\in\Theta.$$ For $\theta\in\Theta_0$ it is the probability of a type I error; for $\theta\in\Theta_1$ it is the power at $\theta$, and $1-\beta_\phi(\theta)$ is the probability of a type II error.
  3. The size of $\phi$ is $\sup_{\theta\in\Theta_0}\beta_\phi(\theta)$. The test has level $\alpha$ if its size is at most $\alpha$. The number $\alpha$ is called the significance level.

The roles of the two hypotheses are not symmetric. The error we control is the type I error, so $H_0$ should be the claim whose false rejection is more costly: "the drug is no better than placebo", "the batch meets specification", "the coin is fair". A test that fails to reject $H_0$ has not shown $H_0$ to be true. It has only found the data compatible with it, and when the power is low that is weak evidence.

✏️ Example 1 — Size, power and a randomised test for a binomial proportion
A new treatment is given to $n=20$ patients. The standard treatment succeeds with probability $0.5$. Let $X$ be the number of successes, so $X\sim\operatorname{Bin}(20,p)$, and test $H_0:p=0.5$ against $H_1:p\gt0.5$. (a) Find the size of the test that rejects when $X\ge15$, and its power at $p=0.6$, $0.7$ and $0.8$. (b) What is the largest size not exceeding $0.05$ that a non-randomised test of the form $X\ge c$ can have? (c) Construct a randomised test of size exactly $0.05$ and find its power at $p=0.7$. (d) Find the $p$-value of the observation $x=14$.
(a) Size The size is $P_{0.5}(X\ge15)=\sum_{k=15}^{20}\binom{20}{k}2^{-20}=0.0207$.
(a) Power The power function is $\beta(p)=P_p(X\ge15)$. It equals $0.1256$ at $p=0.6$, $0.4164$ at $p=0.7$ and $0.8042$ at $p=0.8$. With $20$ patients a real improvement from $0.5$ to $0.6$ is detected only one time in eight.
(b) $P_{0.5}(X\ge14)=0.0577\gt0.05$ and $P_{0.5}(X\ge15)=0.0207$. Among tests "reject when $X\ge c$" the largest size not above $0.05$ is $0.0207$. The jump from $0.0207$ to $0.0577$ is the probability $P_{0.5}(X=14)=0.0370$ of a single point.
(c) Reject when $X\ge15$, and when $X=14$ reject with probability $\gamma$. The size is $0.0207+\gamma\cdot0.0370=0.05$, so $$\gamma=\frac{0.05-0.020695}{0.036964}=0.793.$$ The power at $p=0.7$ is $P_{0.7}(X\ge15)+\gamma P_{0.7}(X=14)=0.568$, compared with $0.416$ for the non-randomised test.
(d) Larger $X$ is stronger evidence against $H_0$, so the $p$-value of $x=14$ is $P_{0.5}(X\ge14)=0.0577$. At the $5\%$ level $H_0$ is not rejected (Definition 5.3 below).
✏️ Example 2 — The power function of a $z$-test and the sample size
Tablet weights are $N(\mu,\sigma^2)$ with $\sigma=4$ mg known. We test $H_0:\mu=500$ against $H_1:\mu\gt500$ at level $\alpha=0.05$ using $n=25$ tablets and the test that rejects when $\bar X\gt c$. (a) Find $c$ and the power function. (b) Find the power at $\mu=501$ and $\mu=502$. (c) How many tablets are needed for power $0.90$ at $\mu=502$? (d) Find the $p$-value if $\bar x=501.6$.
(a) Under $\mu$, $\bar X\sim N(\mu,\sigma^2/n)$ with $\sigma/\sqrt n=0.8$. The size is $P_{500}(\bar X\gt c)=1-\Phi\left(\frac{c-500}{0.8}\right)=0.05$, so $c=500+z_{0.05}\cdot0.8=500+1.6449\times0.8=501.316$ mg. The power function is $$\beta(\mu)=1-\Phi\left(\frac{501.316-\mu}{0.8}\right)=\Phi\left(\frac{\sqrt n(\mu-500)}{\sigma}-z_{0.05}\right).$$
(b) $\beta(501)=\Phi(1.25-1.6449)=\Phi(-0.395)=0.346$ and $\beta(502)=\Phi(2.5-1.6449)=\Phi(0.855)=0.804$.
(c) We need $\frac{\sqrt n\cdot2}{4}-z_{0.05}\ge z_{0.10}$, that is $$n\ge\left(\frac{(z_{0.05}+z_{0.10})\sigma}{\delta}\right)^2=\left(\frac{(1.6449+1.2816)\times4}{2}\right)^2=34.26,$$ so $n=35$ tablets (power $0.905$; with $n=34$ it is $0.898$).
(d) $z=\frac{501.6-500}{0.8}=2.0$ and the $p$-value is $P_{500}(\bar X\ge501.6)=1-\Phi(2.0)=0.0228$: reject at $5\%$, not at $1\%$.
Lesson The sample size formula $n=\left((z_\alpha+z_\beta)\sigma/\delta\right)^2$, with $1-\beta$ the target power at distance $\delta$, is used to plan almost every clinical trial.
📘 Definition 5.3 — The $p$-value

Suppose large values of a statistic $T$ are evidence against $H_0$, and for each $\alpha$ the test rejects when $T\ge c_\alpha$. The $p$-value of the observed data $x$ is $$p(x)=\sup_{\theta\in\Theta_0}P_\theta\left(T(X)\ge T(x)\right).$$

It is the smallest significance level at which the observed data lead to rejection: $H_0$ is rejected at level $\alpha$ exactly when $p(x)\le\alpha$. In words, it is the probability, computed under $H_0$, of a result at least as extreme as the one observed.

📐 Theorem 5.1 — $p$-values are stochastically at least uniform under $H_0$

With $p(x)$ as in Definition 5.3:

  1. for every $\theta\in\Theta_0$ and every $u\in[0,1]$, $P_\theta\left(p(X)\le u\right)\le u$. So the test "reject when $p(X)\le\alpha$" has level $\alpha$;
  2. if $H_0=\{\theta_0\}$ is simple and $T$ has a continuous distribution under $\theta_0$, then $p(X)\sim U(0,1)$ under $H_0$.

Proof. Fix $\theta\in\Theta_0$ and put $G(t)=P_\theta(T\ge t)$, a non-increasing function of $t$. Since $p(x)\ge G(T(x))$, it suffices to show $P_\theta\left(G(T)\le u\right)\le u$. The set $A=\{t:G(t)\le u\}$ is an up-set: if $t\in A$ and $t'\gt t$ then $G(t')\le G(t)\le u$. So $A$ is empty, or a half-line $[t^*,\infty)$, or a half-line $(t^*,\infty)$. In the first case the probability is $0$. In the second, $P_\theta(T\in A)=P_\theta(T\ge t^*)=G(t^*)\le u$. In the third, $P_\theta(T\in A)=P_\theta(T\gt t^*)=\lim_{t\downarrow t^*}G(t)\le u$, since $G(t)\le u$ for every $t\gt t^*$. This proves (1). For (2), $p(X)=P_{\theta_0}(T\ge T(X))=1-F(T(X))$ where $F$ is the continuous cdf of $T$, and $F(T)\sim U(0,1)$ by the probability integral transform (§1.3), so $1-F(T)\sim U(0,1)$ as well. ∎

For discrete statistics the inequality in (1) is usually strict. In Example 1 the possible $p$-values are $P_{0.5}(X\ge k)$, and $P(p(X)\le0.05)=0.0207$. This is one reason why exact tests for discrete data are conservative.

📐 Theorem 5.2 — Duality between tests and confidence sets

(a) Suppose that for each $\theta_0\in\Theta$ there is a non-randomised level-$\alpha$ test of $H_0:\theta=\theta_0$ with acceptance region $A(\theta_0)$. Then $C(x)=\{\theta:x\in A(\theta)\}$ is a confidence set of level $1-\alpha$.

(b) Conversely, if $C(X)$ is a confidence set of level $1-\alpha$, then for each $\theta_0$ the test that accepts $H_0:\theta=\theta_0$ exactly when $\theta_0\in C(x)$ has level $\alpha$.

Proof. In both directions, $\theta\in C(x)\iff x\in A(\theta)$. Hence, for every $\theta$, $$P_\theta\left(\theta\in C(X)\right)=P_\theta\left(X\in A(\theta)\right)\ge1-\alpha,$$ because $1-P_\theta(X\in A(\theta))$ is the size of the test of $H_0:\theta$; and the same identity read backwards gives (b). ∎

So the $t$-interval of §4.3 is the set of $\mu_0$ not rejected by the two-sided $t$-test, and the Wilson interval for a proportion is the set of $p_0$ not rejected by the score test of §5.2. "A $95\%$ interval excludes $\theta_0$" and "the $5\%$ test rejects $\theta_0$" are the same statement.

The Neyman–Pearson Lemma and Most Powerful Tests

We begin with the simplest problem: one distribution against one other. Write $f_0=f(\cdot;\theta_0)$ and $f_1=f(\cdot;\theta_1)$, densities with respect to Lebesgue measure or counting measure (a pmf). Integrals $\int\phi f_0$ then mean sums in the discrete case.

📘 Definition 5.4 — Most powerful test
For the simple hypotheses $H_0:\theta=\theta_0$ and $H_1:\theta=\theta_1$, a test $\phi$ of level $\alpha$ is most powerful (MP) at level $\alpha$ if $E_{\theta_1}\phi\ge E_{\theta_1}\phi'$ for every test $\phi'$ with $E_{\theta_0}\phi'\le\alpha$.

The idea of the lemma is an economic one. Each point $x$ "costs" $f_0(x)$ of the type I error budget $\alpha$ and "buys" $f_1(x)$ of power. To get the most power for a fixed budget, buy the points with the best ratio $f_1(x)/f_0(x)$ first.

📐 Theorem 5.3 — The Neyman–Pearson lemma

Let $f_0,f_1$ be densities (or pmfs) and $0\le\alpha\le1$.

  1. Sufficiency. If a test $\phi$ satisfies, for some constant $k\ge0$, $$\phi(x)=\begin{cases}1,&f_1(x)\gt kf_0(x),\\0,&f_1(x)\lt kf_0(x),\end{cases}\qquad(\ast)$$ and $E_{\theta_0}\phi=\alpha$, then $\phi$ is most powerful at level $\alpha$.
  2. Existence. For every $\alpha\in(0,1]$ there are constants $k\ge0$ and $\gamma\in[0,1]$ such that the test $(\ast)$ with $\phi(x)=\gamma$ on the boundary $\{f_1=kf_0\}$ has size exactly $\alpha$. (For $\alpha=0$, the test $\phi=\mathbf 1\{f_1\gt0=f_0\}$ is MP.)
  3. Necessity. If $\phi'$ is MP at level $\alpha$ and $k$ is as in (2), then $\phi'$ satisfies $(\ast)$ for almost every $x$ in $\{f_1\ne kf_0\}$ (with respect to $f_0+f_1$). Moreover $E_{\theta_0}\phi'=\alpha$, unless some test of size less than $\alpha$ has power $1$.

Proof. (1) Let $\phi'$ be any test with $E_{\theta_0}\phi'\le\alpha$. We claim that $$\left(\phi(x)-\phi'(x)\right)\left(f_1(x)-kf_0(x)\right)\ge0\quad\text{for every }x.$$ Where $f_1\gt kf_0$, $\phi=1\ge\phi'$; where $f_1\lt kf_0$, $\phi=0\le\phi'$; where $f_1=kf_0$ the product is $0$. Integrating, $$\int(\phi-\phi')f_1\ \ge\ k\int(\phi-\phi')f_0=k\left(\alpha-E_{\theta_0}\phi'\right)\ge0.$$ The left side is $E_{\theta_1}\phi-E_{\theta_1}\phi'$, so $\phi$ is at least as powerful as $\phi'$.

(2) Let $R=f_1(X)/f_0(X)$, defined under $P_{\theta_0}$ (where $f_0\gt0$ with probability one), and let $G(k)=P_{\theta_0}(R\gt k)$. $G$ is non-increasing and right-continuous, $G(k)\to0$ as $k\to\infty$, and $G(0^-)=1$. Take $k=\inf\{t\ge0:G(t)\le\alpha\}$. Then $G(k)\le\alpha\le G(k^-)=P_{\theta_0}(R\ge k)$, and the jump $G(k^-)-G(k)=P_{\theta_0}(R=k)$. If this jump is zero put $\gamma=0$; otherwise put $$\gamma=\frac{\alpha-P_{\theta_0}(R\gt k)}{P_{\theta_0}(R=k)}\in[0,1].$$ Then $E_{\theta_0}\phi=P_{\theta_0}(R\gt k)+\gamma P_{\theta_0}(R=k)=\alpha$. (Points with $f_0=0\lt f_1$ have $R=\infty$ and are always in the critical region; they cost nothing.)

(3) Let $\phi$ be the test of (2) and $\phi'$ another MP test at level $\alpha$. Both have the same power, so the integral in (1) satisfies $$0\le\int(\phi-\phi')(f_1-kf_0)=E_{\theta_1}\phi-E_{\theta_1}\phi'-k\left(\alpha-E_{\theta_0}\phi'\right)=-k\left(\alpha-E_{\theta_0}\phi'\right)\le0.$$ Hence the integral of the non-negative function $(\phi-\phi')(f_1-kf_0)$ is zero, so this function vanishes almost everywhere, and on $\{f_1\ne kf_0\}$ this forces $\phi'=\phi$. Also $k(\alpha-E_{\theta_0}\phi')=0$. If $k\gt0$ this gives $E_{\theta_0}\phi'=\alpha$. If $k=0$, then $\phi=1$ wherever $f_1\gt0$, so $\phi$ has power $1$ with size $\alpha$, and $\phi'$, being equally powerful, also has power $1$; this is the exceptional case. ∎

📐 Theorem 5.4 — Two consequences of the lemma
  1. If $\phi$ is MP at level $\alpha$ for $f_0$ against $f_1$, its power $\beta=E_{\theta_1}\phi$ satisfies $\beta\ge\alpha$. If $0\lt\alpha\lt1$ and $f_0\ne f_1$ (on a set of positive probability), then $\beta\gt\alpha$.
  2. If $T$ is sufficient for $\{f_0,f_1\}$, the MP test can be taken to be a function of $T$ alone.

Proof. (1) The trivial test $\phi'\equiv\alpha$ has size $\alpha$ and power $\alpha$, so $\beta\ge\alpha$. Suppose $\beta=\alpha$ with $0\lt\alpha\lt1$. Then $\phi'\equiv\alpha$ is also MP, and by Theorem 5.3(3) it agrees with the NP test on $\{f_1\ne kf_0\}$. But the NP test takes only the values $0$ and $1$ there, and $\phi'$ takes the value $\alpha\notin\{0,1\}$. So $\{f_1\ne kf_0\}$ has probability zero, that is, $f_1=kf_0$ almost everywhere; integrating gives $k=1$ and $f_1=f_0$. (2) By the factorisation theorem (§4.2), $f(x;\theta)=g(T(x);\theta)h(x)$, so $f_1(x)/f_0(x)=g(T(x);\theta_1)/g(T(x);\theta_0)$ wherever $h(x)\gt0$. The NP test $(\ast)$, with a boundary value $\gamma$ that is constant, is therefore a function of $T(x)$. ∎

✏️ Example 3 — An MP test for exponential lifetimes
Ten components with iid exponential lifetimes of mean $\theta$ hours are tested to failure. The supplier claims $\theta=1000$; a rival claims $\theta=500$. (a) Find the MP test of $H_0:\theta=1000$ against $H_1:\theta=500$ at level $0.05$. (b) Find its power. (c) Is the same test MP against every alternative $\theta_1\lt1000$?
Ratio With $T=\sum x_i$, $f(x;\theta)=\theta^{-10}e^{-T/\theta}$ for $x_i\gt0$, so $$\frac{f_1(x)}{f_0(x)}=\left(\frac{1000}{500}\right)^{10}\exp\left\{-T\left(\frac1{500}-\frac1{1000}\right)\right\}=2^{10}e^{-T/1000}.$$ This is a decreasing function of $T$, so $\{f_1\gt kf_0\}=\{T\lt c\}$ for some $c$.
(a) Critical value Under $\theta$, $2T/\theta\sim\chi^2_{20}$ (§4.1). We need $P_{1000}(T\lt c)=P(\chi^2_{20}\lt2c/1000)=0.05$, so $2c/1000=\chi^2_{20,0.95}=10.851$ (the lower $5\%$ point) and $c=5425.4$ hours. Reject the supplier's claim if the total lifetime is below $5425$ hours, that is, if the mean lifetime is below $542.5$ hours.
(b) Power Under $\theta=500$, $2T/500\sim\chi^2_{20}$, so the power is $P(\chi^2_{20}\lt2\times5425.4/500)=P(\chi^2_{20}\lt21.70)=0.643$.
(c) For any $\theta_1\lt1000$ the ratio is $(1000/\theta_1)^{10}\exp\{-T(1/\theta_1-1/1000)\}$, again decreasing in $T$. So the MP test is "reject when $T\lt5425.4$" for every such alternative. The critical region does not depend on $\theta_1$. This is the phenomenon behind uniformly most powerful tests (next subsection).
✏️ Example 4 — Prove or disprove: an MP critical region is always a tail
A single observation $X$ has the Cauchy density $f(x;\theta)=\dfrac{1}{\pi\left(1+(x-\theta)^2\right)}$. For $H_0:\theta=0$ against $H_1:\theta=1$, find the MP test with $k=2$, its size and its power. Is "reject when $X$ is large" a most powerful test?
Ratio $r(x)=\dfrac{f_1(x)}{f_0(x)}=\dfrac{1+x^2}{1+(x-1)^2}$. As $x\to\pm\infty$, $r(x)\to1$. Setting $r'(x)=0$ gives $x^2-x-1=0$, so $r$ has its maximum $\frac{3+\sqrt5}{2}=2.618$ at $x=\frac{1+\sqrt5}2$ and its minimum $\frac{3-\sqrt5}2=0.382$ at $x=\frac{1-\sqrt5}2$.
Region for $k=2$ $r(x)\gt2\iff1+x^2\gt2\left(1+(x-1)^2\right)\iff x^2-4x+3\lt0\iff1\lt x\lt3$. By the Neyman–Pearson lemma, the test that rejects $H_0$ exactly when $1\lt X\lt3$ is MP at its own size.
Size and power Size: $P_0(1\lt X\lt3)=\frac1\pi(\arctan3-\arctan1)=0.1476$. Power: $P_1(1\lt X\lt3)=\frac1\pi(\arctan2-\arctan0)=0.3524$.
Disprove The MP region is a bounded interval. A very large observation, say $x=100$, is not evidence for $\theta=1$ over $\theta=0$: both densities are almost equal there, since $r(100)=1.02$. So a right-tail test $X\gt c$ is not MP. By Theorem 5.3(3) any MP test agrees with $\mathbf 1\{1\lt x\lt3\}$ except on a null set, and the tail test of the same size, $X\gt2$ (indeed $P_0(X\gt2)=\frac1\pi\arctan\frac12=0.1476$), differs from it on a set of positive probability. Its power is only $P_1(X\gt2)=0.25$, against $0.3524$. The tail test is the natural guess, but it is wrong whenever the likelihood ratio is not monotone.
Interactive 1 — Size, Power and the Neyman–Pearson Cut-off

$X_1,\dots,X_n$ iid $N(\mu,1)$; test $H_0:\mu=0$ against $H_1:\mu=\mu_1$. The curves are the densities of $\bar X$ under $H_0$ and under $H_1$. The test rejects when $\bar X\gt c$ with $c=z/\sqrt n$. The red area is the size $\alpha$ and the blue area is the type II error probability $\beta$. Moving the cut-off trades one error for the other. Increasing $n$ separates the curves, and only then do both errors shrink together. Press Play to sweep the cut-off.

90.60
cut-off c—
size α—
type II error β—
power 1 − β—
likelihood ratio k at c—

Monotone Likelihood Ratio and Uniformly Most Powerful Tests

📘 Definition 5.5 — Uniformly most powerful test
A test $\phi$ of level $\alpha$ for $H_0:\theta\in\Theta_0$ against $H_1:\theta\in\Theta_1$ is uniformly most powerful (UMP) at level $\alpha$ if $\beta_\phi(\theta)\ge\beta_{\phi'}(\theta)$ for every $\theta\in\Theta_1$ and every test $\phi'$ of level $\alpha$.

A UMP test is MP against each simple alternative $\theta_1\in\Theta_1$ at once. Example 3 suggests when this can happen: the NP critical region must be the same for all $\theta_1\in\Theta_1$. The following property guarantees this for one-sided problems.

📘 Definition 5.6 — Monotone likelihood ratio
Let $\Theta\subseteq\mathbb{R}$ and let $T(x)$ be a real statistic. The family $\{f(\cdot;\theta)\}$ has a monotone likelihood ratio (MLR) in $T$ if the distributions are distinct for distinct $\theta$ and, for every $\theta\lt\theta'$, the ratio $f(x;\theta')/f(x;\theta)$ is a non-decreasing function of $T(x)$ on the set where at least one of the two densities is positive (with $c/0=\infty$ for $c\gt0$).
📐 Theorem 5.5 — Families with a monotone likelihood ratio

(a) A one-parameter exponential family $f(x;\theta)=h(x)\exp\{c(\theta)T(x)-A(\theta)\}$ with $c$ strictly increasing has MLR in $T$. If $c$ is strictly decreasing it has MLR in $-T$.

(b) Consequently, for iid samples, the following have MLR in the stated statistic: $\operatorname{Bin}(m,p)$, $\operatorname{Poi}(\lambda)$, $\operatorname{Geo}(p)$ (in $-\sum X_i$), $N(\mu,\sigma_0^2)$ in $\sum X_i$, $N(\mu_0,\sigma^2)$ in $\sum(X_i-\mu_0)^2$, $\operatorname{Exp}$ with mean $\theta$ in $\sum X_i$, and $\operatorname{Gamma}$ with known shape in $\sum X_i$ (as a function of the mean). The non-exponential families $U(0,\theta)$ and the shifted exponential $e^{-(x-\theta)}\mathbf 1\{x\gt\theta\}$ have MLR in $X_{(n)}$ and $X_{(1)}$ respectively. The Cauchy location family does not have MLR in $X$ (Example 4).

Proof. (a) For $\theta\lt\theta'$, $$\frac{f(x;\theta')}{f(x;\theta)}=e^{(c(\theta')-c(\theta))T(x)}\,e^{A(\theta)-A(\theta')}$$ with $c(\theta')-c(\theta)\gt0$, an increasing function of $T(x)$; both densities are positive on the same set $\{h\gt0\}$. (b) The iid sample from an exponential family is again an exponential family with $T=\sum T(X_i)$ and the same $c$. For example, $\operatorname{Poi}(\lambda)$ has $c(\lambda)=\log\lambda$, and $\operatorname{Geo}(p)$, with pmf $p(1-p)^{x}$, has $c(p)=\log(1-p)$, which is decreasing. For $U(0,\theta)$ and $\theta\lt\theta'$, $$\frac{f(x;\theta')}{f(x;\theta)}=\frac{\theta^n}{\theta'^n}\cdot\frac{\mathbf 1\{x_{(n)}\lt\theta'\}}{\mathbf 1\{x_{(n)}\lt\theta\}}=\begin{cases}(\theta/\theta')^n,&x_{(n)}\lt\theta,\\\infty,&\theta\le x_{(n)}\lt\theta',\end{cases}$$ non-decreasing in $x_{(n)}$ (for $x_i\gt0$). The shifted exponential is similar, with ratio $e^{n(\theta'-\theta)}$ for $x_{(1)}\gt\theta'$ and $0$ for $\theta\lt x_{(1)}\le\theta'$. ∎

📐 Theorem 5.6 — The Karlin–Rubin theorem (UMP one-sided tests)

Let the family have MLR in $T$, and let $0\lt\alpha\lt1$. For testing $H_0:\theta\le\theta_0$ against $H_1:\theta\gt\theta_0$, consider $$\phi(x)=\begin{cases}1,&T(x)\gt c,\\\gamma,&T(x)=c,\\0,&T(x)\lt c,\end{cases}\qquad\text{with }E_{\theta_0}\phi=\alpha.$$

  1. Such $c$ and $\gamma\in[0,1]$ exist, and $\phi$ is UMP at level $\alpha$.
  2. The power function $\beta_\phi(\theta)$ is non-decreasing in $\theta$, and strictly increasing on the set where $0\lt\beta_\phi(\theta)\lt1$.
  3. For $\theta\lt\theta_0$, $\phi$ minimises $E_\theta\phi'$ among all tests $\phi'$ with $E_{\theta_0}\phi'=\alpha$.

For $H_0:\theta\ge\theta_0$ against $H_1:\theta\lt\theta_0$ the UMP test rejects for small $T$, with all inequalities reversed.

Proof. Existence of $c,\gamma$. This is the argument of Theorem 5.3(2) applied to the cdf of $T$ under $\theta_0$: take $c$ with $P_{\theta_0}(T\gt c)\le\alpha\le P_{\theta_0}(T\ge c)$ and $\gamma=\left(\alpha-P_{\theta_0}(T\gt c)\right)/P_{\theta_0}(T=c)$ when $P_{\theta_0}(T=c)\gt0$.

$\phi$ is an NP test for every pair $\theta_1\lt\theta_2$. Write the ratio as $f(x;\theta_2)/f(x;\theta_1)=g(T(x))$ with $g$ non-decreasing, and put $k=g(c)$. If $g(T(x))\gt k$ then $T(x)\gt c$ (otherwise $g(T(x))\le g(c)$), so $\phi(x)=1$. If $g(T(x))\lt k$ then $T(x)\lt c$, so $\phi(x)=0$. Thus $\phi$ has the form $(\ast)$ for $f_0=f(\cdot;\theta_1)$, $f_1=f(\cdot;\theta_2)$, and by Theorem 5.3(1) it is MP at level $\beta_\phi(\theta_1)$ for $\theta_1$ against $\theta_2$. (If $g(c)=\infty$, then $\phi$ rejects only where $f(x;\theta_1)=0$, so $\beta_\phi(\theta_1)=0$ and the inequality needed below holds trivially.)

Monotone power (2). For $\theta_1\lt\theta_2$, $\phi$ is MP at level $\beta_\phi(\theta_1)$, so by Theorem 5.4(1) $\beta_\phi(\theta_2)\ge\beta_\phi(\theta_1)$, with strict inequality when $0\lt\beta_\phi(\theta_1)\lt1$ (the two distributions differ by the definition of MLR).

UMP (1). By (2), $\sup_{\theta\le\theta_0}\beta_\phi(\theta)=\beta_\phi(\theta_0)=\alpha$, so $\phi$ has level $\alpha$. Let $\phi'$ be any level-$\alpha$ test of $H_0:\theta\le\theta_0$. Then $E_{\theta_0}\phi'\le\alpha$. Fix $\theta_1\gt\theta_0$. Since $\phi$ is MP at level $\alpha$ for $\theta_0$ against $\theta_1$, $\beta_\phi(\theta_1)\ge\beta_{\phi'}(\theta_1)$. As $\theta_1\gt\theta_0$ was arbitrary, $\phi$ is UMP.

(3). For $\theta\lt\theta_0$ apply the lemma to $1-\phi$ with the roles of the hypotheses exchanged: $1-\phi$ rejects "$\theta_0$" in favour of "$\theta$" when $T\lt c$, which is the NP form for $f_0=f(\cdot;\theta_0)$, $f_1=f(\cdot;\theta)$ because the ratio $f(x;\theta)/f(x;\theta_0)$ is non-increasing in $T$. So $1-\phi$ maximises $E_\theta(1-\phi')$ among tests with $E_{\theta_0}(1-\phi')=1-\alpha$. ∎

✏️ Example 5 — A randomised UMP test for a Poisson rate
The numbers of faults in five rolls of cloth are iid $\operatorname{Poi}(\lambda)$. The standard is $\lambda\le2$ faults per roll. (a) Find the UMP test of $H_0:\lambda\le2$ against $H_1:\lambda\gt2$ of size exactly $0.05$. (b) Find its power at $\lambda=2.5$, $3$ and $4$. (c) What does the best non-randomised test of level $0.05$ look like, and what is its size?
MLR The Poisson family is an exponential family with $c(\lambda)=\log\lambda$ increasing and $T=\sum X_i\sim\operatorname{Poi}(5\lambda)$. By Theorem 5.6 the UMP test rejects for large $T$, with randomisation at the boundary.
(a) Under $\lambda=2$, $T\sim\operatorname{Poi}(10)$: $P(T\ge16)=0.0487$, $P(T\ge15)=0.0835$ and $P(T=15)=0.0347$. So $c=15$ and $$\gamma=\frac{0.05-0.04874}{0.03472}=0.0363.$$ Reject if $T\ge16$; if $T=15$, reject with probability $0.0363$; otherwise accept.
(b) $\beta(\lambda)=P_\lambda(T\ge16)+0.0363\,P_\lambda(T=15)$ with $T\sim\operatorname{Poi}(5\lambda)$: $\beta(2.5)=0.197$, $\beta(3)=0.436$, $\beta(4)=0.845$. By Theorem 5.6(2) these increase with $\lambda$, and $\beta(\lambda)\lt0.05$ for every $\lambda\lt2$: the composite null is respected.
(c) Without randomisation the best we can do at level $0.05$ is "reject if $T\ge16$", of size $0.0487$. Here $\gamma$ is small and the difference is negligible. In Example 1 it was not.
📐 Theorem 5.7 — There is no UMP two-sided test for a normal mean
Let $X_1,\dots,X_n$ be iid $N(\mu,\sigma^2)$ with $\sigma$ known, and let $0\lt\alpha\lt1$. There is no UMP level-$\alpha$ test of $H_0:\mu=\mu_0$ against $H_1:\mu\ne\mu_0$.

Proof. Suppose $\phi$ were UMP. Fix $\mu_1\gt\mu_0$. The NP test of $\mu_0$ against $\mu_1$ at level $\alpha$ is $\phi_+=\mathbf 1\{\bar x\gt\mu_0+z_\alpha\sigma/\sqrt n\}$, and the likelihood ratio $\exp\{n(\mu_1-\mu_0)\bar x/\sigma^2-\text{const}\}$ has a continuous distribution, so the boundary $\{f_1=kf_0\}$ has probability zero. No test of size less than $\alpha$ has power $1$ (the normal densities are positive everywhere). Since $\phi$ is UMP, it is MP for this pair, and by Theorem 5.3(3) $\phi=\phi_+$ almost everywhere. Repeating the argument with $\mu_1\lt\mu_0$ gives $\phi=\phi_-=\mathbf 1\{\bar x\lt\mu_0-z_\alpha\sigma/\sqrt n\}$ almost everywhere. But $\phi_+$ and $\phi_-$ differ on the set $\{\bar x\gt\mu_0+z_\alpha\sigma/\sqrt n\}$, which has positive probability. This contradiction proves the theorem. ∎

The same argument works in any family where the MP tests against alternatives on the two sides of $\theta_0$ are essentially unique and different. It does not mean that two-sided UMP tests never exist, as the next example shows.

✏️ Example 6 — Prove or disprove: a two-sided UMP test never exists
Let $X_1,\dots,X_n$ be iid $U(0,\theta)$ and $0\lt\alpha\lt1$. Show that $$\phi^*(x)=\begin{cases}1,&x_{(n)}\gt\theta_0\ \text{ or }\ x_{(n)}\le\theta_0\,\alpha^{1/n},\\0,&\text{otherwise}\end{cases}$$ is a UMP level-$\alpha$ test of $H_0:\theta=\theta_0$ against $H_1:\theta\ne\theta_0$. Find its power at $\theta=1.5$ and $\theta=0.8$ when $\theta_0=1$, $n=5$, $\alpha=0.05$.
Size Under $\theta_0$, $P(X_{(n)}\le t)=(t/\theta_0)^n$ for $0\le t\le\theta_0$, and $X_{(n)}\gt\theta_0$ is impossible. So $E_{\theta_0}\phi^*=\left(\alpha^{1/n}\right)^n=\alpha$.
$\theta\gt\theta_0$ $\beta(\theta)=P_\theta(X_{(n)}\gt\theta_0)+P_\theta(X_{(n)}\le\theta_0\alpha^{1/n})=1-\left(\tfrac{\theta_0}{\theta}\right)^n+\alpha\left(\tfrac{\theta_0}{\theta}\right)^n=1-(1-\alpha)\left(\tfrac{\theta_0}{\theta}\right)^n$. Any level-$\alpha$ test $\phi'$ satisfies, for $\theta\gt\theta_0$, $$E_\theta\phi'\le P_\theta(X_{(n)}\gt\theta_0)+\left(\tfrac{\theta_0}{\theta}\right)^nE_{\theta_0}\phi'\le1-\left(\tfrac{\theta_0}{\theta}\right)^n+\alpha\left(\tfrac{\theta_0}{\theta}\right)^n,$$ because on $\{x_{(n)}\le\theta_0\}$ the density under $\theta$ is $(\theta_0/\theta)^n$ times the density under $\theta_0$. So $\phi^*$ is most powerful at every $\theta\gt\theta_0$.
$\theta\lt\theta_0$ If $\theta\le\theta_0\alpha^{1/n}$, then $X_{(n)}\lt\theta\le\theta_0\alpha^{1/n}$ and $\phi^*$ has power $1$. Otherwise take $k=(\theta_0/\theta)^n$. On $\{x_{(n)}\lt\theta\}$ we have $f_\theta=kf_{\theta_0}$, the boundary set where $(\ast)$ allows any value. On $\{\theta\le x_{(n)}\lt\theta_0\}$ we have $f_\theta=0\lt kf_{\theta_0}$, where $(\ast)$ requires $\phi=0$; and $\phi^*$ does vanish there, because $\theta_0\alpha^{1/n}\lt\theta$. So $\phi^*$ has the form $(\ast)$ and size $\alpha$, and it is MP by Theorem 5.3(1). Its power is $P_\theta(X_{(n)}\le\theta_0\alpha^{1/n})=\alpha(\theta_0/\theta)^n$.
Numbers $\alpha^{1/5}=0.5493$. $\beta(1.5)=1-0.95\,(1/1.5)^5=0.8749$ and $\beta(0.8)=0.05\,(1/0.8)^5=0.1526$.
Conclusion The statement is false: here a UMP two-sided test exists. The reason is that the likelihood ratio takes only the values $0$, a constant and $\infty$, so the MP tests are far from unique, and one of them works on both sides at once. The shifted exponential family behaves the same way (Exercise III.8).
Interactive 2 — Power Functions: One-Sided, Two-Sided and the Missing UMP Test

Three size-$\alpha$ tests of $H_0:\theta=\theta_0$ and their power functions: the test that is UMP against right-hand alternatives, the one that is UMP against left-hand alternatives, and a two-sided test. For the normal mean no curve lies above the others everywhere, so no UMP test exists (Theorem 5.7). Each one-sided test also falls below the dashed line $\alpha$ on its wrong side, so it is biased. Switch to the uniform model: there the purple curve follows the upper envelope of the other two, and a UMP two-sided test exists (Example 6). Play moves the marker $\theta$.

9
θ—
right-sided test—
left-sided test—
two-sided test—
📘 Definition 5.7 — Unbiased tests
A test $\phi$ of $H_0:\theta\in\Theta_0$ against $H_1:\theta\in\Theta_1$ is unbiased at level $\alpha$ if $\beta_\phi(\theta)\le\alpha$ for all $\theta\in\Theta_0$ and $\beta_\phi(\theta)\ge\alpha$ for all $\theta\in\Theta_1$. An unbiased level-$\alpha$ test that is at least as powerful as every other unbiased level-$\alpha$ test at every $\theta\in\Theta_1$ is uniformly most powerful unbiased (UMPU).

An unbiased test is never more likely to reject when $H_0$ is true than when it is false. This is a mild requirement, and it rules out exactly the one-sided tests that caused the trouble in Theorem 5.7.

📐 Theorem 5.8 — Unbiasedness and UMPU tests
  1. A UMP level-$\alpha$ test is unbiased.
  2. In the normal model of Theorem 5.7, the equal-tailed test $\phi=\mathbf 1\left\{\sqrt n|\bar x-\mu_0|/\sigma\gt z_{\alpha/2}\right\}$ is unbiased, with power function $$\beta(\mu)=\Phi(-z_{\alpha/2}-\delta)+\Phi(-z_{\alpha/2}+\delta),\qquad\delta=\frac{\sqrt n(\mu-\mu_0)}{\sigma},$$ which is symmetric in $\delta$ and strictly increasing in $|\delta|$. The one-sided test $\phi_+$ is not unbiased for $H_1:\mu\ne\mu_0$.
  3. (Statement.) In a one-parameter exponential family with natural statistic $T$, the test that rejects when $T\lt c_1$ or $T\gt c_2$ (randomising at $c_1,c_2$), with $c_1,c_2$ chosen so that $E_{\theta_0}\phi=\alpha$ and $E_{\theta_0}[T\phi]=\alpha E_{\theta_0}[T]$, is UMPU for $H_0:\theta=\theta_0$ against $H_1:\theta\ne\theta_0$. For the normal mean these conditions give the equal-tailed $z$-test; for the normal variance they give unequal tails. The one- and two-sided $t$-tests are UMPU in the presence of the unknown $\sigma$.

Proof. (1) Let $\phi$ be UMP at level $\alpha$. The test $\phi'\equiv\alpha$ has level $\alpha$, so $\beta_\phi(\theta)\ge\beta_{\phi'}(\theta)=\alpha$ for every $\theta\in\Theta_1$. (2) Under $\mu$, $Z=\sqrt n(\bar X-\mu_0)/\sigma\sim N(\delta,1)$, so $\beta=P(Z\gt z_{\alpha/2})+P(Z\lt-z_{\alpha/2})$, that is, $$\beta=\Phi(\delta-z_{\alpha/2})+\Phi(-\delta-z_{\alpha/2}),$$ which is symmetric in $\delta$. Writing $\varphi$ for the standard normal density, $$\frac{d\beta}{d\delta}=\varphi(\delta-z_{\alpha/2})-\varphi(\delta+z_{\alpha/2}).$$ For $\delta\gt0$ we have $|\delta-z_{\alpha/2}|\lt\delta+z_{\alpha/2}$, so the derivative is positive. Hence $\beta$ has its minimum $\alpha$ at $\delta=0$ and $\beta(\mu)\gt\alpha$ for $\mu\ne\mu_0$. For $\phi_+$, $\beta_+(\mu)=\Phi(\delta-z_\alpha)\lt\Phi(-z_\alpha)=\alpha$ when $\delta\lt0$. (3) The proof uses a generalised form of the Neyman–Pearson lemma with two side conditions; we only quote the result. ∎

Likelihood Ratio Tests and Wilks' Theorem

Most practical problems are composite on both sides and contain nuisance parameters, such as the unknown variance when the mean is tested. The Neyman–Pearson ratio $f_1/f_0$ is then not defined, because neither hypothesis specifies a single distribution. The natural generalisation compares the best explanation of the data under $H_0$ with the best explanation overall.

📘 Definition 5.8 — The (generalised) likelihood ratio test

The likelihood ratio statistic for $H_0:\theta\in\Theta_0$ against $H_1:\theta\in\Theta\setminus\Theta_0$ is $$\Lambda(x)=\frac{\sup_{\theta\in\Theta_0}L(\theta)}{\sup_{\theta\in\Theta}L(\theta)}=\frac{L(\hat\theta_0)}{L(\hat\theta)},$$ where $\hat\theta_0$ is the MLE under $H_0$ and $\hat\theta$ the unrestricted MLE (when they exist). Then $0\le\Lambda\le1$. The likelihood ratio test (LRT) rejects $H_0$ when $\Lambda(x)\le\lambda_\alpha$, equivalently when $-2\log\Lambda(x)\ge c_\alpha$, with the constant chosen so that $\sup_{\theta\in\Theta_0}P_\theta\left(\Lambda(X)\le\lambda_\alpha\right)=\alpha$ (or $\le\alpha$).

Because $L(\theta)=g(T(x);\theta)h(x)$ for a sufficient statistic $T$ (§4.2), $\Lambda$ is a function of $T$.

Two remarks. First, the denominator uses all of $\Theta$, not $\Theta_1$; this changes nothing when $\Theta_0$ is "small" (a point or a lower-dimensional set) and the likelihood is continuous, and it makes $\Lambda\le1$. Second, when $H_0$ and $H_1$ are both simple, $\Lambda=\min\left(1,f_0/f_1\right)$, so the LRT is the Neyman–Pearson test. For one-sided problems in MLR families the LRT usually coincides with the UMP test (Exercise IV.1 gives the two-sided analogue for the exponential model).

📐 Theorem 5.9 — The normal-theory tests are likelihood ratio tests

Let $X_1,\dots,X_n$ be iid $N(\mu,\sigma^2)$ with both parameters unknown, $S^2=\frac1{n-1}\sum(X_i-\bar X)^2$, and $\hat\sigma^2=\frac{n-1}{n}S^2$.

  1. The $t$-test. For $H_0:\mu=\mu_0$ against $\mu\ne\mu_0$, $$\Lambda=\left(1+\frac{T^2}{n-1}\right)^{-n/2},\qquad T=\frac{\sqrt n(\bar X-\mu_0)}{S}\sim t_{n-1}\ \text{under }H_0,$$ so the LRT rejects when $|T|\gt t_{n-1,\alpha/2}$.
  2. The $\chi^2$-test for a variance. For $H_0:\sigma^2=\sigma_0^2$ against $\sigma^2\ne\sigma_0^2$, with $W=\hat\sigma^2/\sigma_0^2$, $$\Lambda=W^{n/2}e^{-n(W-1)/2},$$ so the LRT rejects when $(n-1)S^2/\sigma_0^2=nW$ lies outside an interval $[c_1,c_2]$, where $c_1,c_2$ satisfy $P(c_1\le\chi^2_{n-1}\le c_2)=1-\alpha$ and $\frac{c_1}{n}-\log c_1=\frac{c_2}{n}-\log c_2$. The equal-tailed test with $c_1=\chi^2_{n-1,1-\alpha/2}$, $c_2=\chi^2_{n-1,\alpha/2}$ is an approximation to it.
  3. The pooled two-sample $t$-test. For independent samples $X_1,\dots,X_m\sim N(\mu_1,\sigma^2)$ and $Y_1,\dots,Y_n\sim N(\mu_2,\sigma^2)$ and $H_0:\mu_1=\mu_2$, $$\Lambda=\left(1+\frac{T^2}{m+n-2}\right)^{-(m+n)/2},\qquad T=\frac{\bar X-\bar Y}{S_p\sqrt{\frac1m+\frac1n}}\sim t_{m+n-2}\ \text{under }H_0,$$ where $S_p^2=\frac{(m-1)S_1^2+(n-1)S_2^2}{m+n-2}$.
  4. The $F$-test. For independent samples $N(\mu_1,\sigma_1^2)$ and $N(\mu_2,\sigma_2^2)$ and $H_0:\sigma_1^2=\sigma_2^2$, $\Lambda$ is a function of $F=S_1^2/S_2^2$ that is small exactly when $F$ is far from a value near $1$ on either side. Under $H_0$, $F\sim F_{m-1,n-1}$, and the usual test rejects when $F\gt F_{m-1,n-1,\alpha/2}$ or $F\lt F_{m-1,n-1,1-\alpha/2}$.

Proof. The normal log-likelihood is $\ell(\mu,\sigma^2)=-\frac n2\log(2\pi\sigma^2)-\frac{1}{2\sigma^2}\sum(x_i-\mu)^2$. For fixed $\mu$ it is maximised at $\sigma^2=\frac1n\sum(x_i-\mu)^2$, giving the profile value $-\frac n2\log\left(\frac{2\pi e}{n}\sum(x_i-\mu)^2\right)$. So, for any restriction on $\mu$ alone, $$\Lambda=\left(\frac{\min_{\Theta}\sum(x_i-\mu)^2}{\min_{\Theta_0}\sum(x_i-\mu)^2}\right)^{n/2}.$$

(1) The unrestricted minimum is $\sum(x_i-\bar x)^2=n\hat\sigma^2$, and under $H_0$ it is $\sum(x_i-\mu_0)^2=n\hat\sigma^2+n(\bar x-\mu_0)^2$. Hence $\Lambda^{-2/n}=1+\frac{(\bar x-\mu_0)^2}{\hat\sigma^2}=1+\frac{n(\bar x-\mu_0)^2}{(n-1)s^2}=1+\frac{t^2}{n-1}$. This is an increasing function of $|t|$, so $\Lambda\le\lambda_\alpha\iff|t|\ge c$. Under $H_0$, $T\sim t_{n-1}$ for every $\sigma$ (§4.1), so the size is $\alpha$ for $c=t_{n-1,\alpha/2}$, whatever the nuisance parameter.

(2) Unrestricted, $\hat\mu=\bar x$ and $\sigma^2=\hat\sigma^2$, and $L=(2\pi\hat\sigma^2)^{-n/2}e^{-n/2}$. Under $H_0$, $\hat\mu=\bar x$ and $L=(2\pi\sigma_0^2)^{-n/2}e^{-n\hat\sigma^2/(2\sigma_0^2)}$. The ratio is $W^{n/2}e^{-n(W-1)/2}$ with $W=\hat\sigma^2/\sigma_0^2$. Now $\log\Lambda=\frac n2(\log W-W+1)$ is increasing for $W\lt1$ and decreasing for $W\gt1$, with maximum $0$ at $W=1$. So $\Lambda\le\lambda$ iff $W\le w_1$ or $W\ge w_2$, where $w_1\lt1\lt w_2$ have $\log w_1-w_1=\log w_2-w_2$. Writing $c_i=nw_i$ gives the stated condition. Under $H_0$, $nW=(n-1)S^2/\sigma_0^2\sim\chi^2_{n-1}$.

(3) The same computation with the pooled log-likelihood: the unrestricted minimum of the sum of squares is $Q=\sum(x_i-\bar x)^2+\sum(y_j-\bar y)^2=(m+n-2)s_p^2$, and under $H_0$ (common mean, estimated by the grand mean) it is $Q+\frac{mn}{m+n}(\bar x-\bar y)^2$. Since $\frac{mn}{m+n}=\left(\frac1m+\frac1n\right)^{-1}$, $\Lambda^{-2/(m+n)}=1+\frac{(\bar x-\bar y)^2}{(1/m+1/n)Q}=1+\frac{t^2}{m+n-2}$. The law of $T$ under $H_0$ is §4.1.

(4) With $\hat\sigma_1^2=\frac{m-1}mS_1^2$, $\hat\sigma_2^2=\frac{n-1}nS_2^2$, and, under $H_0$, $\hat\sigma^2=\frac{m\hat\sigma_1^2+n\hat\sigma_2^2}{m+n}$, one finds $$\Lambda=\frac{(\hat\sigma_1^2)^{m/2}(\hat\sigma_2^2)^{n/2}}{(\hat\sigma^2)^{(m+n)/2}}=\frac{(m+n)^{(m+n)/2}}{m^{m/2}n^{n/2}}\cdot\frac{v^{m/2}}{(1+v)^{(m+n)/2}},\qquad v=\frac{m\hat\sigma_1^2}{n\hat\sigma_2^2}.$$ The function $v^{m/2}(1+v)^{-(m+n)/2}$ increases up to $v=m/n$ and then decreases, so $\Lambda$ is small exactly when $v$, and hence $F=\frac{(m-1)n}{m(n-1)}\,v$, is in either tail. ∎

📐 Theorem 5.10 — Wilks' theorem

Let $X_1,\dots,X_n$ be iid with density $f(x;\theta)$, $\theta\in\Theta\subseteq\mathbb{R}^p$ open, satisfying the regularity conditions of the asymptotic theory of the MLE (§4.2): common support not depending on $\theta$, three continuous derivatives in $\theta$ with integrable bounds, and a positive definite Fisher information matrix. Let $H_0$ specify $r$ of the coordinates, $\Theta_0=\{\theta:\theta_1=\theta_{0,1},\dots,\theta_r=\theta_{0,r}\}$ with $1\le r\le p$, and let the true $\theta$ be an interior point of $\Theta_0$ (in the remaining $p-r$ coordinates). Then, under $H_0$, $$-2\log\Lambda_n\xrightarrow{d}\chi^2_r,\qquad r=\dim\Theta-\dim\Theta_0.$$ Consequently the test that rejects when $-2\log\Lambda_n\gt\chi^2_{r,\alpha}$ has asymptotic size $\alpha$.

Proof for $p=r=1$. Let $\ell_n(\theta)=\sum\log f(X_i;\theta)$ and $H_0:\theta=\theta_0$. By §4.2 there is a consistent root $\hat\theta_n$ of $\ell_n'(\theta)=0$ with $\sqrt n(\hat\theta_n-\theta_0)\xrightarrow dN(0,1/I(\theta_0))$. Expand $\ell_n(\theta_0)$ about $\hat\theta_n$ by Taylor's theorem: $$\ell_n(\theta_0)=\ell_n(\hat\theta_n)+\ell_n'(\hat\theta_n)(\theta_0-\hat\theta_n)+\tfrac12\ell_n''(\theta_n^*)(\theta_0-\hat\theta_n)^2$$ with $\theta_n^*$ between $\theta_0$ and $\hat\theta_n$. The middle term vanishes, so $$-2\log\Lambda_n=2\left[\ell_n(\hat\theta_n)-\ell_n(\theta_0)\right]=\left(-\frac1n\ell_n''(\theta_n^*)\right)\cdot\left(\sqrt n(\hat\theta_n-\theta_0)\right)^2.$$ Since $\theta_n^*\to\theta_0$ in probability, the bound on the third derivative and the weak law give $-\frac1n\ell_n''(\theta_n^*)\xrightarrow{P}I(\theta_0)$, exactly as in the proof of asymptotic normality in §4.2. And $\sqrt{nI(\theta_0)}(\hat\theta_n-\theta_0)\xrightarrow dZ\sim N(0,1)$. By Slutsky's theorem and the continuous mapping theorem (§2.3), $-2\log\Lambda_n\xrightarrow dZ^2\sim\chi^2_1$.

Sketch for general $p,r$. Let $\theta^*$ be the true value and $Q(u)=n\,u^{\mathsf T}I(\theta^*)u$. The same expansion, applied to both suprema, gives $$-2\log\Lambda_n\approx Q(\hat\theta-\theta^*)-Q(\hat\theta_0-\theta^*).$$ In the coordinates in which $\sqrt n\,I(\theta^*)^{1/2}(\hat\theta-\theta^*)$ is approximately $N_p(0,I_p)$, the restricted MLE is approximately the orthogonal projection onto a $(p-r)$-dimensional subspace, so the difference is the squared length of the projection onto an $r$-dimensional subspace. The squared length of the projection of a standard normal vector onto an $r$-dimensional subspace is $\chi^2_r$ (§7.1). ∎

Wilks' theorem is the reason likelihood ratio tests are used everywhere: one formula, one reference distribution, and the degrees of freedom are found by counting free parameters. The chi-square tests of §5.2 are its most famous instance. The hypotheses matter, though. The theorem fails when the support depends on $\theta$ (Exercise IV.6) and when the true value lies on the boundary of $\Theta$ (⚠️ box).

✏️ Example 7 — The tablet data: exact $t$-test, likelihood ratio and Wilks
$n=25$ tablets have $\bar x=502.3$ mg and $s=4.1$ mg; the weights are assumed $N(\mu,\sigma^2)$ with both parameters unknown. (a) Test $H_0:\mu=500$ against $\mu\ne500$ at level $5\%$ and find the $p$-value. (b) Compute $-2\log\Lambda$ and the $p$-value given by Wilks' approximation. (c) Test $H_0:\sigma=4$ against $\sigma\ne4$ at level $5\%$ with the equal-tailed $\chi^2$-test, and find the exact LRT critical values.
(a) $t=\frac{\sqrt{25}(502.3-500)}{4.1}=\frac{2.3}{0.82}=2.805$. Since $t_{24,0.025}=2.064\lt2.805$, reject $H_0$. The $p$-value is $2P(t_{24}\gt2.805)=0.0098$.
(b) By Theorem 5.9(1), $-2\log\Lambda=n\log\left(1+\frac{t^2}{n-1}\right)=25\log\left(1+\frac{7.868}{24}\right)=7.088$. Here $\dim\Theta=2$ and $\dim\Theta_0=1$, so Wilks gives $\chi^2_1$: the approximate $p$-value is $P(\chi^2_1\gt7.088)=0.0078$, close to the exact $0.0098$. The exact test is preferable, because the relation $-2\log\Lambda=n\log(1+t^2/(n-1))$ gives the exact null law through $t_{n-1}$.
(c) Equal tails $(n-1)s^2/\sigma_0^2=24\times16.81/16=25.215$. The equal-tailed acceptance region is $[\chi^2_{24,0.975},\chi^2_{24,0.025}]=[12.401,\ 39.364]$, which contains $25.215$: do not reject. The $p$-value is $2\min\{P(\chi^2_{24}\le25.215),P(\chi^2_{24}\ge25.215)\}=0.788$.
(c) Exact LRT Solving $P(c_1\le\chi^2_{24}\le c_2)=0.95$ and $\frac{c_1}{25}-\log c_1=\frac{c_2}{25}-\log c_2$ numerically gives $c_1=13.236$ and $c_2=42.257$, with tail probabilities $0.0379$ and $0.0121$. These are not equal tails. The decision here is the same.
🧭 Problem-solving strategy — constructing and analysing a test
  1. Set up. Write the model, $\Theta_0$ and $\Theta_1$. Are the hypotheses simple, one-sided or two-sided? Are there nuisance parameters?
  2. Simple against simple: compute $f_1/f_0$, simplify it to a monotone function of a statistic $T$ (if possible), and find $c$ (and $\gamma$ for discrete $T$) from the null law of $T$.
  3. One-sided, one parameter: check MLR (exponential family with monotone $c(\theta)$, or $U(0,\theta)$-type families). If it holds, the Karlin–Rubin test is UMP, and its size is computed at the boundary $\theta_0$.
  4. Two-sided: expect no UMP test (Theorem 5.7); use the equal-tailed or UMPU test. Exception: families where the likelihood ratio is constant on the support, such as $U(0,\theta)$.
  5. Nuisance parameters or several parameters: compute $\Lambda$, simplify it to a known pivot ($t$, $\chi^2$, $F$) if possible; otherwise use $-2\log\Lambda\approx\chi^2_r$ with $r$ = number of restrictions.
  6. Report: the critical region, the size, the power at relevant alternatives, and the $p$-value.
⚠️ Counterexamples and pitfalls
  1. "Accept $H_0$" is not "$H_0$ is true". With $n=20$ in Example 1 the power at $p=0.6$ is $0.13$, so failing to reject says very little.
  2. A $p$-value is not $P(H_0\mid\text{data})$. It is a probability computed under $H_0$; the posterior probability of $H_0$ needs a prior (§4.3) and can be far larger than $p$.
  3. Size versus level. The size of a composite null is a supremum over $\Theta_0$, not the value at one point. With discrete data non-randomised tests usually cannot have size exactly $\alpha$: in Example 1 the sizes available near $0.05$ are $0.0207$ and $0.0577$.
  4. MP regions need not be tails. For the Cauchy location family the MP region for $\theta=0$ against $\theta=1$ is the bounded interval $(1,3)$ (Example 4); a monotone likelihood ratio is what makes tails optimal.
  5. No MLR outside the right families. Cauchy location has no MLR; an exponential family has MLR only in the direction in which $c(\theta)$ is monotone (geometric: in $-\sum X_i$); the curved family $N(\theta,\theta^2)$ has a likelihood ratio depending on both $\sum X_i$ and $\sum X_i^2$, so Theorem 5.5 does not apply to it.
  6. "Two-sided UMP tests never exist" is false. They fail for the normal mean (Theorem 5.7), but exist for $U(0,\theta)$ (Example 6) and for the shifted exponential (Exercise III.8).
  7. One-sided tests are biased against two-sided alternatives. $\phi_+$ has power below $\alpha$ for every $\mu\lt\mu_0$ (Theorem 5.8(2)).
  8. MP tests need not be unique. When $f_1/f_0$ has atoms (shifted exponential, uniform), many tests of the same size are MP (Exercise II.6).
  9. Wilks fails when the support depends on $\theta$. For $U(0,\theta)$ and $H_0:\theta=\theta_0$, $-2\log\Lambda=-2n\log(X_{(n)}/\theta_0)\sim\chi^2_2$ exactly, not $\chi^2_1$ (Exercise IV.6).
  10. Wilks fails on the boundary. For $N(\mu,1)$ with $\Theta=[0,\infty)$ and $H_0:\mu=0$, $-2\log\Lambda=n\bar X^2\mathbf 1\{\bar X\gt0\}$, whose law is the mixture $\frac12\delta_0+\frac12\chi^2_1$, not $\chi^2_1$.
  11. The exact LRT for a variance is not equal-tailed. For $n=25$ its tails are $0.0379$ and $0.0121$ (Example 7).
  12. Statistical is not practical significance. With $n=10^6$ tablets a mean of $500.02$ mg is "highly significant" against $500$ but irrelevant. Report an effect size or an interval as well.

Exercises

I
Errors, size, power and $p$-values.
  • 1 $X\sim\operatorname{Bin}(10,p)$. The test of $H_0:p=0.5$ against $H_1:p=0.8$ rejects when $X\ge8$. Find its size, its power and the probability of a type II error.
  • 2 $X_1,\dots,X_{16}$ are iid $N(\mu,9)$. The test of $H_0:\mu=10$ against $H_1:\mu\gt10$ rejects when $\bar X\gt11.5$. Find its size and its power at $\mu=12$.
  • 3 A single observation $X$ comes from $U(0,\theta)$. The test of $H_0:\theta=1$ rejects when $X\gt0.9$. Find the power function $\beta(\theta)$ for all $\theta\gt0$, the size, and the power at $\theta=2$.
  • 4 For the one-sided $z$-test with observed statistic $z$, show that the $p$-value is $1-\Phi(z)$ and that it is $U(0,1)$ under $H_0$. Find the one- and two-sided $p$-values for $z=1.8$.
  • 5 With $\sigma=10$ known, how large must $n$ be for the level-$0.01$ test of $H_0:\mu=50$ against $H_1:\mu\gt50$ to have power $0.95$ at $\mu=55$?
  • 6 Show that if $\phi_1,\phi_2$ have level $\alpha$ then so does $\lambda\phi_1+(1-\lambda)\phi_2$ for $\lambda\in[0,1]$. Show by an example that $\max(\phi_1,\phi_2)$ need not have level $\alpha$.
  • 7 A $95\%$ confidence interval for a proportion is $[0.31,\ 0.47]$. What does the dual test decide about $H_0:p=0.5$ and $H_0:p=0.4$ at level $5\%$? Show that the two-sided $z$-test accepts $\mu_0$ exactly when $\mu_0\in\bar x\pm z_{\alpha/2}\sigma/\sqrt n$.
  • 8 A single $X\sim\operatorname{Poi}(\lambda)$ is observed, and $H_0:\lambda=1$ is tested against $H_1:\lambda\gt1$. Find the non-randomised test of the form $X\ge c$ with the largest size not exceeding $0.05$, and the randomised test of size exactly $0.05$.
1 Size $P_{0.5}(X\ge8)=\frac{45+10+1}{1024}=0.0547$. Power $P_{0.8}(X\ge8)=0.6778$, so the type II error probability is $0.3222$.
2 $\sigma/\sqrt n=3/4$. Size $P(Z\gt1.5/0.75)=1-\Phi(2)=0.0228$. Power at $12$: $P\left(Z\gt\frac{11.5-12}{0.75}\right)=\Phi(0.667)=0.7475$.
3 $\beta(\theta)=P_\theta(X\gt0.9)=0$ for $\theta\le0.9$ and $1-0.9/\theta$ for $\theta\gt0.9$. Size $\beta(1)=0.1$; power $\beta(2)=0.55$.
4 Larger $Z$ is more extreme, so $p=P_0(Z\ge z)=1-\Phi(z)$. Under $H_0$, $P(1-\Phi(Z)\le u)=P(Z\ge z_u)=u$, which is Theorem 5.1(2). For $z=1.8$: one-sided $0.0359$, two-sided $2(1-\Phi(1.8))=0.0719$.
5 $n\ge\left((z_{0.01}+z_{0.05})\sigma/\delta\right)^2=\left((2.3263+1.6449)\times10/5\right)^2=63.08$, so $n=64$.
6 $E_\theta[\lambda\phi_1+(1-\lambda)\phi_2]=\lambda\beta_1(\theta)+(1-\lambda)\beta_2(\theta)\le\alpha$ on $\Theta_0$, and the combination has values in $[0,1]$. For $X\sim N(\theta,1)$ and $H_0:\theta=0$, $\phi_1=\mathbf 1\{X\gt z_\alpha\}$ and $\phi_2=\mathbf 1\{X\lt-z_\alpha\}$ have size $\alpha$, but $\max(\phi_1,\phi_2)$ has size $2\alpha$.
7 By Theorem 5.2, reject $p=0.5$ (outside the interval) and do not reject $p=0.4$ (inside). For the $z$-test, $|\bar x-\mu_0|\le z_{\alpha/2}\sigma/\sqrt n\iff\mu_0\in[\bar x-z_{\alpha/2}\sigma/\sqrt n,\ \bar x+z_{\alpha/2}\sigma/\sqrt n]$.
8 $P_1(X\ge3)=0.0803$ and $P_1(X\ge4)=0.0190$, so the best non-randomised test is $X\ge4$, of size $0.0190$. With $P_1(X=3)=0.0613$, the randomised test rejects when $X\ge4$ and, when $X=3$, with probability $\gamma=\frac{0.05-0.0190}{0.0613}=0.506$.
II
The Neyman–Pearson lemma.
  • 1 $X_1,\dots,X_n$ iid $N(0,\sigma^2)$. Find the MP test of $H_0:\sigma^2=1$ against $H_1:\sigma^2=4$ at level $\alpha$. For $n=10$, $\alpha=0.05$, find the critical value and the power.
  • 2 $X_1,\dots,X_{15}$ iid $\operatorname{Ber}(p)$. Find the MP test of size exactly $0.05$ for $H_0:p=0.2$ against $H_1:p=0.4$, and its power.
  • 3 A single $X$ has density $f_0(x)=1$ on $(0,1)$ under $H_0$ and $f_1(x)=2x$ on $(0,1)$ under $H_1$. Find the MP test of size $\alpha$ and its power. Evaluate it at $\alpha=0.05$.
  • 4 A single $X$ is $N(0,1)$ under $H_0$ and standard Cauchy under $H_1$. Show that the MP test of size $0.05$ rejects when $|X|\gt1.96$, and find its power.
  • 5 Let $\phi$ be an NP test of the form $(\ast)$ with $k\gt0$, of size $\alpha$ and power $\beta$ for $f_0$ against $f_1$. Show that $1-\phi$ is MP at level $1-\beta$ for testing $f_1$ against $f_0$.
  • 6 $X_1,\dots,X_n$ iid with density $e^{-(x-\theta)}$, $x\gt\theta$. For $H_0:\theta=0$ against $H_1:\theta=1$, show that every MP test of size $\alpha\ge e^{-n}$ has power $1$, and that the MP test is not unique. What happens if $\alpha\lt e^{-n}$?
  • 7 $X_1,\dots,X_5$ iid $\operatorname{Poi}(\lambda)$. Find the MP test of size exactly $0.05$ for $H_0:\lambda=2$ against $H_1:\lambda=1$, and its power.
  • 8 $X_1,\dots,X_n$ iid $N(\mu,1)$. Show that the MP test of $\mu=0$ against $\mu=\mu_1\gt0$ has power $\Phi(\sqrt n\mu_1-z_\alpha)$. Find the smallest $n$ for which both error probabilities are at most $0.05$ when $\mu_1=0.5$.
1 $\frac{f_1}{f_0}=2^{-n}\exp\left\{\left(\tfrac12-\tfrac18\right)\sum x_i^2\right\}$ is increasing in $\sum x_i^2$, so reject when $\sum x_i^2\gt c$. Under $H_0$, $\sum X_i^2\sim\chi^2_n$, so $c=\chi^2_{n,\alpha}$. For $n=10$: $c=18.307$; under $H_1$, $\sum X_i^2/4\sim\chi^2_{10}$, so the power is $P(\chi^2_{10}\gt4.577)=0.918$.
2 The ratio $\left(\frac{0.4}{0.2}\right)^T\left(\frac{0.6}{0.8}\right)^{15-T}$ is increasing in $T=\sum X_i\sim\operatorname{Bin}(15,p)$. Under $p=0.2$: $P(T\ge7)=0.0181$ and $P(T=6)=0.0430$. Reject when $T\ge7$, and when $T=6$ with probability $\gamma=\frac{0.05-0.01806}{0.04299}=0.743$. Power: $P_{0.4}(T\ge7)+0.743\,P_{0.4}(T=6)=0.544$.
3 $f_1/f_0=2x$ is increasing, so reject when $X\gt c$ with $1-c=\alpha$, that is $c=1-\alpha$. Power $\int_{1-\alpha}^12x\,dx=1-(1-\alpha)^2=\alpha(2-\alpha)$; at $\alpha=0.05$ it is $0.0975$, barely more than the size, because the two densities are similar.
4 $r(x)=\frac{f_1}{f_0}=\frac{\sqrt{2\pi}}{\pi}\cdot\frac{e^{x^2/2}}{1+x^2}$ depends on $u=x^2$ through $g(u)=e^{u/2}/(1+u)$, and $g'(u)\propto\frac{1+u}{2}-1$, so $g$ decreases on $[0,1]$ and increases on $[1,\infty)$. Put $k=r(1.96)=1.125$. On $|x|\le1.96$ we have $r(x)\le\max\{r(0),r(1.96)\}=\max\{0.798,1.125\}$, with equality only at $|x|=1.96$, so $\{r\gt k\}=\{|x|\gt1.96\}$, which has size $0.05$. Power $P(|C|\gt1.96)=1-\frac2\pi\arctan1.96=0.300$. (For a large size, such as $0.5$, the MP region would also contain a neighbourhood of $0$.)
5 $1-\phi=1$ where $f_1\lt kf_0$, i.e. where $f_0\gt\frac1kf_1$, and $1-\phi=0$ where $f_0\lt\frac1kf_1$. This is the form $(\ast)$ for the pair $(f_1,f_0)$ with constant $1/k$, and its size under $f_1$ is $1-\beta$. By Theorem 5.3(1) it is MP at that level.
6 $\frac{f_1}{f_0}=e^n$ on $\{x_{(1)}\gt1\}$, $0$ on $\{0\lt x_{(1)}\le1\}$; $P_0(X_{(1)}\gt1)=e^{-n}$ and $P_1(X_{(1)}\gt1)=1$. If $\alpha\ge e^{-n}$, any test with $\phi=1$ on $\{x_{(1)}\gt1\}$ and $E_0\phi=\alpha$ has power $1$, hence is MP: for instance, reject also when $x_{(1)}\lt c$ with $1-e^{-nc}=\alpha-e^{-n}$, or reject on $\{x_{(1)}\le1\}$ with a constant probability. These are different tests. If $\alpha\lt e^{-n}$, $k=e^n$ and the MP tests are those supported in $\{x_{(1)}\gt1\}$ with size $\alpha$, all with power $\alpha e^n$; for example, reject when $x_{(1)}\gt-\frac1n\log\alpha$. Again not unique.
7 The ratio $2^{-T}e^{5}$ is decreasing in $T=\sum X_i$, so reject for small $T$. Under $\lambda=2$, $T\sim\operatorname{Poi}(10)$: $P(T\le4)=0.0293$ and $P(T=5)=0.0378$. Reject when $T\le4$, and when $T=5$ with probability $\gamma=\frac{0.05-0.0293}{0.0378}=0.548$. Power under $T\sim\operatorname{Poi}(5)$: $0.4405+0.548\times0.1755=0.537$.
8 The MP test rejects when $\bar X\gt z_\alpha/\sqrt n$; under $\mu_1$, $\sqrt n(\bar X-\mu_1)\sim N(0,1)$, so the power is $P(Z\gt z_\alpha-\sqrt n\mu_1)=\Phi(\sqrt n\mu_1-z_\alpha)$. We need $\sqrt n\times0.5-1.6449\ge1.6449$, i.e. $n\ge(3.2897/0.5)^2=43.29$: $n=44$.
III
Monotone likelihood ratio, UMP and unbiased tests.
  • 1 $X_1,\dots,X_n$ iid with density $\theta x^{\theta-1}$ on $(0,1)$, $\theta\gt0$. Find the UMP level-$\alpha$ test of $H_0:\theta\le1$ against $H_1:\theta\gt1$ in terms of a $\chi^2$ quantile, and write it out for $n=5$, $\alpha=0.05$.
  • 2 Show that iid geometric observations with pmf $p(1-p)^x$, $x=0,1,2,\dots$, have MLR in $-\sum X_i$. Describe the UMP test of $H_0:p\le p_0$ against $H_1:p\gt p_0$.
  • 3 Prove that the Cauchy location family (one observation) does not have MLR in $X$.
  • 4 $X_1,\dots,X_4$ iid $U(0,\theta)$. Find the UMP level-$0.1$ test of $H_0:\theta\le2$ against $H_1:\theta\gt2$ and its power at $\theta=2.5$.
  • 5 Eight lifetimes are iid exponential with mean $\theta$. Find the UMP level-$0.05$ test of $H_0:\theta\le100$ against $H_1:\theta\gt100$ and its power at $\theta=200$.
  • 6 For $N(\mu,\sigma^2)$, $\sigma$ known, compute the power of the right-sided UMP $5\%$ test at $\mu=\mu_0-\sigma/\sqrt n$. Explain why it is useless for $H_1:\mu\ne\mu_0$.
  • 7 Let $X_1,\dots,X_n$ be iid $N(\mu,1)$ and $\phi$ an unbiased level-$\alpha$ test of $H_0:\mu=\mu_0$ against $H_1:\mu\ne\mu_0$. Show that $\beta_\phi(\mu_0)=\alpha$, that $\beta_\phi'(\mu_0)=0$, and that the latter is equivalent to $E_{\mu_0}[(\bar X-\mu_0)\phi(X)]=0$.
  • 8 $X_1,\dots,X_n$ iid with density $e^{-(x-\theta)}$, $x\gt\theta$. Show that the test rejecting when $X_{(1)}\le\theta_0$ or $X_{(1)}\gt\theta_0-\frac1n\log\alpha$ is UMP at level $\alpha$ for $H_0:\theta=\theta_0$ against $H_1:\theta\ne\theta_0$.
1 $f(x;\theta)=\theta^n\exp\{(\theta-1)\sum\log x_i\}$: an exponential family with $c(\theta)=\theta-1$ increasing and $T=\sum\log X_i$. The UMP test rejects when $T\gt c$, i.e. when $-2\sum\log X_i\lt k$. Under $\theta=1$, $-\log X_i\sim\operatorname{Exp}(1)$, so $-2\sum\log X_i\sim\chi^2_{2n}$ and $k=\chi^2_{2n,1-\alpha}$ (the lower $\alpha$ point). For $n=5$, $\alpha=0.05$: reject when $-2\sum\log x_i\lt3.940$.
2 $\prod p(1-p)^{x_i}=p^n\exp\{\log(1-p)\sum x_i\}$ with $c(p)=\log(1-p)$ decreasing; so the family has MLR in $-\sum X_i$ (Theorem 5.5(a)). The UMP test rejects for large $-\sum X_i$, i.e. for small $\sum X_i$, with randomisation at the boundary, the size being computed at $p_0$ from the negative binomial law of $\sum X_i$.
3 For $\theta=0\lt\theta'=1$ the ratio $r(x)=\frac{1+x^2}{1+(x-1)^2}$ satisfies $r(0)=\frac12$, $r(2)=\frac52$ and $r(x)\to1$ as $x\to\infty$ (Example 4). So it increases and then decreases: it is not monotone in $x$.
4 MLR in $X_{(4)}$; reject when $X_{(4)}\gt c$ with $P_2(X_{(4)}\gt c)=1-(c/2)^4=0.1$: $c=2(0.9)^{1/4}=1.948$. Power at $2.5$: $1-(1.948/2.5)^4=1-0.9\times(2/2.5)^4=0.631$.
5 The likelihood $\theta^{-8}e^{-T/\theta}$ has $c(\theta)=-1/\theta$ increasing, so reject for large $T=\sum X_i$. Under $\theta=100$, $2T/100\sim\chi^2_{16}$, so $c=100\,\chi^2_{16,0.05}/2=100\times26.296/2=1314.8$. Power at $200$: $P(\chi^2_{16}\gt2\times1314.8/200)=P(\chi^2_{16}\gt13.148)=0.662$.
6 $\beta_+(\mu)=\Phi(\delta-z_{0.05})$ with $\delta=\sqrt n(\mu-\mu_0)/\sigma=-1$: $\Phi(-2.645)=0.0041$, far below $0.05$. For alternatives below $\mu_0$ the test almost never rejects. It is biased (Theorem 5.8(2)).
7 $\beta_\phi(\mu)=\int\phi(x)\prod\varphi(x_i-\mu)\,dx$ is continuous, indeed differentiable (exponential family; differentiation under the integral sign), with $\beta_\phi'(\mu)=E_\mu[\phi(X)\,n(\bar X-\mu)]$. Unbiasedness gives $\beta_\phi(\mu)\ge\alpha$ for $\mu\ne\mu_0$ and $\beta_\phi(\mu_0)\le\alpha$; by continuity $\beta_\phi(\mu_0)=\alpha$, so $\mu_0$ is a minimum of $\beta_\phi$ and $\beta_\phi'(\mu_0)=0$, which is $nE_{\mu_0}[(\bar X-\mu_0)\phi]=0$. This is the second side condition of Theorem 5.8(3).
8 Put $t=-\frac1n\log\alpha$. Size: under $\theta_0$, $P(X_{(1)}\le\theta_0)=0$ and $P(X_{(1)}\gt\theta_0+t)=e^{-nt}=\alpha$. For $\theta\gt\theta_0$: $f_\theta/f_{\theta_0}=e^{n(\theta-\theta_0)}$ on $\{x_{(1)}\gt\theta\}$ and $0$ on $\{\theta_0\lt x_{(1)}\le\theta\}$, so a size-$\alpha$ test is MP iff it rejects only in $\{x_{(1)}\gt\theta\}$ (when $\theta\le\theta_0+t$; then the power is $\alpha e^{n(\theta-\theta_0)}$), or rejects all of it (when $\theta\gt\theta_0+t$; power $1$). Our test does this. For $\theta\lt\theta_0$: on $\{x_{(1)}\gt\theta_0\}$, $f_\theta=e^{-n(\theta_0-\theta)}f_{\theta_0}$, so any level-$\alpha$ $\phi'$ has $E_\theta\phi'\le P_\theta(X_{(1)}\le\theta_0)+e^{-n(\theta_0-\theta)}\alpha=1-(1-\alpha)e^{-n(\theta_0-\theta)}$, and our test attains this bound. So it is UMP on both sides, as in Example 6.
IV
Likelihood ratio tests and Wilks' theorem.
  • 1 $X_1,\dots,X_n$ iid $\operatorname{Exp}(\lambda)$ (rate $\lambda$). Show that the LRT statistic for $H_0:\lambda=\lambda_0$ against $\lambda\ne\lambda_0$ is $\Lambda=(\lambda_0\bar x)^ne^{n(1-\lambda_0\bar x)}$, and describe the critical region in terms of $2\lambda_0\sum X_i\sim\chi^2_{2n}$.
  • 2 Ten measurements give $\bar x=12.6$ and $s=1.5$. Test $H_0:\mu=12$ against $\mu\ne12$ at $5\%$ under normality, and give the $p$-value.
  • 3 Two independent normal samples with a common variance: $m=8$, $\bar x=24.1$, $s_1=3.2$; $n=10$, $\bar y=20.9$, $s_2=2.8$. Test $\mu_1=\mu_2$ at $5\%$.
  • 4 For the data of part 3, test $\sigma_1^2=\sigma_2^2$ at $5\%$ with the $F$-test. Is the pooled test of part 3 justified?
  • 5 $X_1\sim\operatorname{Bin}(100,p_1)$ and $X_2\sim\operatorname{Bin}(100,p_2)$ are independent, with $x_1=30$ and $x_2=45$. Compute $-2\log\Lambda$ for $H_0:p_1=p_2$ and test at $5\%$ using Wilks' theorem.
  • 6 $X_1,\dots,X_n$ iid $U(0,\theta)$ and $H_0:\theta=\theta_0$. Show that $-2\log\Lambda=-2n\log(X_{(n)}/\theta_0)$ when $X_{(n)}\le\theta_0$, and that under $H_0$ it has exactly the $\chi^2_2$ law for every $n$. Why does Wilks' theorem not apply?
  • 7 $20$ iid Poisson counts have $\bar x=2.6$. Test $H_0:\lambda=2$ with the LRT and Wilks' approximation, and compare with the score test $Z=(\bar x-\lambda_0)/\sqrt{\lambda_0/n}$.
  • 8 In Theorem 5.9(2) show that the equal-tailed critical values do not satisfy the LRT condition $\frac{c_1}n-\log c_1=\frac{c_2}n-\log c_2$, using $n=25$, $\alpha=0.05$.
1 $L(\lambda)=\lambda^ne^{-\lambda n\bar x}$ is maximised at $\hat\lambda=1/\bar x$ with $L(\hat\lambda)=\bar x^{-n}e^{-n}$. So $\Lambda=\lambda_0^ne^{-\lambda_0n\bar x}\bar x^ne^n=(\lambda_0\bar x)^ne^{n(1-\lambda_0\bar x)}$. With $w=\lambda_0\bar x$, $\log\Lambda=n(\log w+1-w)$ has its maximum $0$ at $w=1$, so the LRT rejects when $w\notin[w_1,w_2]$ with $w_1-\log w_1=w_2-\log w_2$. Since $2\lambda_0\sum X_i=2nw\sim\chi^2_{2n}$ under $H_0$, the region is $2\lambda_0\sum x_i\notin[2nw_1,2nw_2]$ with $P(2nw_1\le\chi^2_{2n}\le2nw_2)=1-\alpha$.
2 $t=\frac{0.6}{1.5/\sqrt{10}}=1.265\lt t_{9,0.025}=2.262$: do not reject. $p=2P(t_9\gt1.265)=0.238$.
3 $s_p^2=\frac{7\times10.24+9\times7.84}{16}=8.89$, $s_p=2.982$, and $t=\frac{3.2}{2.982\sqrt{1/8+1/10}}=2.263\gt t_{16,0.025}=2.120$: reject at $5\%$ ($p=0.038$).
4 $F=10.24/7.84=1.306$. The acceptance region is $[1/F_{9,7,0.025},\ F_{7,9,0.025}]=[1/4.823,\ 4.197]=[0.207,\ 4.197]$, which contains $1.306$ ($p=0.69$). There is no evidence against equal variances, so pooling is reasonable.
5 Under $H_0$, $\hat p=75/200=0.375$, and the expected counts are $37.5$ and $62.5$ in each sample. $-2\log\Lambda=2\sum O\log(O/E)=2\left[30\log\tfrac{30}{37.5}+70\log\tfrac{70}{62.5}+45\log\tfrac{45}{37.5}+55\log\tfrac{55}{62.5}\right]=4.825\gt3.841$: reject ($p\approx0.028$). Two free parameters against one, so $r=1$. This is the $G^2$ statistic of §5.2; Pearson's $X^2=4.80$ is nearly identical.
6 $L(\theta)=\theta^{-n}\mathbf 1\{x_{(n)}\le\theta\}$, maximised at $\hat\theta=x_{(n)}$. So $\Lambda=(x_{(n)}/\theta_0)^n$ if $x_{(n)}\le\theta_0$ (and $0$ otherwise). Under $H_0$, $U=(X_{(n)}/\theta_0)^n$ has $P(U\le u)=P(X_{(n)}\le\theta_0u^{1/n})=u$, so $U\sim U(0,1)$, and $-2\log U\sim\operatorname{Exp}$ with mean $2$, which is $\chi^2_2$. Wilks needs a support that does not depend on $\theta$ and a regular log-likelihood with a quadratic expansion; here $\hat\theta$ converges at rate $1/n$, not $1/\sqrt n$.
7 $-2\log\Lambda=2n\left[\bar x\log(\bar x/\lambda_0)-(\bar x-\lambda_0)\right]=40\,[2.6\log1.3-0.6]=3.286\lt3.841$: do not reject ($p=0.070$). Score: $z=\frac{0.6}{\sqrt{2/20}}=1.897$, $z^2=3.60$, $p=0.058$. The two tests are asymptotically equivalent and agree here.
8 $c_1=\chi^2_{24,0.975}=12.401$ and $c_2=\chi^2_{24,0.025}=39.364$. The condition is equivalent to $w_1-\log w_1=w_2-\log w_2$ with $w_i=c_i/25$, and here $w_1-\log w_1=0.4960+0.7011=1.1971$ while $w_2-\log w_2=1.5746-0.4540=1.1206$. They differ, so the equal-tailed test is not the LRT. The exact LRT values are $13.236$ and $42.257$ (Example 7).
🎯 Exam corner CSIR Part BCSIR Part CGATE ST

E1. CSIR Part B A single observation $X$ has density $\theta x^{\theta-1}$, $0\lt x\lt1$. The test of $H_0:\theta=1$ against $H_1:\theta=2$ rejects $H_0$ when $X\gt0.8$. Its size and power are (A) $0.2$ and $0.36$; (B) $0.2$ and $0.64$; (C) $0.8$ and $0.36$; (D) $0.36$ and $0.2$.

Idea Size $=P_{\theta_0}(\text{reject})$, power $=P_{\theta_1}(\text{reject})$; the cdf is $x^\theta$.
Compute Size $=P_1(X\gt0.8)=0.2$. Power $=P_2(X\gt0.8)=1-0.8^2=0.36$. Option (B) uses $0.8^2$ instead of $1-0.8^2$; (C) and (D) confuse the roles.
Answer (A).

E2. CSIR Part B GATE ST Let $X_1,\dots,X_9$ be iid $N(\mu,1)$. The UMP level-$0.05$ test of $H_0:\mu\le0$ against $H_1:\mu\gt0$ has power at $\mu=1$ approximately (A) $0.050$; (B) $0.851$; (C) $0.912$; (D) $0.999$.

Idea Karlin–Rubin: reject when $\bar X\gt z_{0.05}/\sqrt9=0.548$; the power is $\Phi(\sqrt n\mu-z_\alpha)$.
Compute $\Phi(3-1.645)=\Phi(1.355)=0.912$. Option (B) is $\Phi(3-1.96)$, the right tail of the two-sided test; (D) is $\Phi(3)$, which ignores the critical value.
Answer (C).

E3. CSIR Part B For a random sample, which of the following families does not have a monotone likelihood ratio in the indicated statistic? (A) $\operatorname{Poi}(\lambda)$, in $\sum X_i$; (B) $U(0,\theta)$, in $X_{(n)}$; (C) the Cauchy location family with $n=1$, in $X_1$; (D) $N(0,\sigma^2)$, in $\sum X_i^2$.

Idea Exponential families with monotone $c(\theta)$ and the uniform family have MLR (Theorem 5.5).
Options (A) $c(\lambda)=\log\lambda$ increasing. (B) the ratio is $(\theta/\theta')^n$ or $\infty$, non-decreasing in $x_{(n)}$. (C) the ratio $\frac{1+x^2}{1+(x-1)^2}$ rises to $2.618$ and falls back to $1$. (D) $c(\sigma^2)=-\frac1{2\sigma^2}$ increasing.
Answer (C).

E4. CSIR Part C Let $X_1,\dots,X_n$ be iid $N(\mu,\sigma^2)$ with $\sigma$ known and $0\lt\alpha\lt\frac12$. Which of the following are true? (One or more options may be correct.) (A) There is a UMP level-$\alpha$ test of $H_0:\mu\le0$ against $H_1:\mu\gt0$. (B) There is a UMP level-$\alpha$ test of $H_0:\mu=0$ against $H_1:\mu\ne0$. (C) The equal-tailed two-sided $z$-test is unbiased. (D) The power function of the test in (A) is a convex function of $\mu$ on $\mathbb{R}$.

(A) True MLR in $\bar X$ and Karlin–Rubin (Theorem 5.6).
(B) False Theorem 5.7: the MP tests against $\mu_1\gt0$ and against $\mu_1\lt0$ are essentially unique and different.
(C) True Theorem 5.8(2): its power $\Phi(-z_{\alpha/2}-\delta)+\Phi(-z_{\alpha/2}+\delta)$ is minimal, equal to $\alpha$, at $\delta=0$.
(D) False The power is $\Phi(\sqrt n\mu/\sigma-z_\alpha)$, which is convex where the argument is negative and concave where it is positive (it increases to $1$).
Answer (A) and (C).

E5. CSIR Part C Let $X_1,\dots,X_n$ be iid $U(0,\theta)$, $0\lt\alpha\lt1$, and consider $H_0:\theta=1$. Which of the following are true? (A) The test that rejects when $X_{(n)}\gt1$ or $X_{(n)}\le\alpha^{1/n}$ has size $\alpha$. (B) This test is UMP at level $\alpha$ for $H_0:\theta=1$ against $H_1:\theta\ne1$. (C) Under $H_0$, the likelihood ratio statistic $-2\log\Lambda$ for $\theta=1$ against $\theta\ne1$ converges in distribution to $\chi^2_1$. (D) The power of the test in (A) at $\theta=2$ is $1-(1-\alpha)2^{-n}$.

(A) True $P_1(X_{(n)}\gt1)=0$ and $P_1(X_{(n)}\le\alpha^{1/n})=\alpha$.
(B) True Example 6.
(C) False $-2\log\Lambda=-2n\log X_{(n)}$ is exactly $\chi^2_2$ for every $n$ (Exercise IV.6); the support depends on $\theta$, so Wilks does not apply.
(D) True $1-(\theta_0/\theta)^n+\alpha(\theta_0/\theta)^n$ with $\theta=2$, $\theta_0=1$.
Answer (A), (B) and (D).

E6. CSIR Part C GATE ST Let $X_1,\dots,X_n$ ($n\ge3$) be iid $N(\mu,\sigma^2)$ with both parameters unknown, let $T=\sqrt n\bar X/S$, and let $\Lambda$ be the likelihood ratio statistic for $H_0:\mu=0$ against $H_1:\mu\ne0$. Which of the following are true? (A) $\Lambda=\left(1+T^2/(n-1)\right)^{-n/2}$. (B) Under $H_0$, $-2\log\Lambda\xrightarrow d\chi^2_1$ as $n\to\infty$. (C) Under $H_0$, $-2\log\Lambda$ has exactly the $\chi^2_1$ distribution for every $n$. (D) The likelihood ratio test of $H_0:\sigma^2=1$ against $\sigma^2\ne1$ ($\mu$ unknown) is the equal-tailed $\chi^2$-test.

(A) True Theorem 5.9(1).
(B) True Wilks with $r=2-1=1$; directly, $n\log(1+T^2/(n-1))\approx T^2\xrightarrow dZ^2$.
(C) False For $n=5$, $P(-2\log\Lambda\gt3.841)=P\left(|t_4|\gt2.150\right)=0.098$, not $0.05$.
(D) False The LRT needs $\frac{c_1}n-\log c_1=\frac{c_2}n-\log c_2$, which gives unequal tails (Theorem 5.9(2), Example 7).
Answer (A) and (B).
🧑‍🏫 For the teacher

Suggested plan — 3 lectures × 60 min.

  1. The language of testing. Definitions 5.1–5.3 with Example 1 on the board (build the binomial table and show why randomisation is needed). Project Interactive 1 with $n=9$, $\mu_1=0.6$: move the cut-off and let the class call out how $\alpha$ and $\beta$ change; then raise $n$. Prove Theorem 5.1 and state Theorem 5.2, linking back to the $t$-interval of §4.3. Example 2 (sample size) to close.
  2. Neyman–Pearson. Motivate with the "budget" picture (buy the points with the best $f_1/f_0$), then prove Theorem 5.3 in full and Theorem 5.4. Examples 3 and 4; for Example 4 ask the class first which region they would use, and let the computation overturn the guess.
  3. UMP, unbiased and LR tests. MLR and Karlin–Rubin (Theorems 5.5, 5.6) with Example 5. Project Interactive 2 in the normal mode to motivate Theorem 5.7, then switch to the uniform mode and do Example 6. Theorem 5.8 briefly; Theorem 5.9(1) and Wilks for $p=1$; Example 7. Finish with E5 and E6 live.

Discussion questions. (1) Why do we fix the type I error and minimise the type II error, rather than minimising their sum? When would minimising $\alpha+\beta$ be more sensible, and which test does that produce? (2) A study reports $p=0.04$. List three different statements that people make about it and decide which are correct. (3) Interactive 2 shows a two-sided UMP test for $U(0,\theta)$. What feature of the likelihood ratio makes this possible, and why does the normal family lack it?

Misconceptions to watch for. Treating "not rejected" as "proved"; reading the $p$-value as the probability that $H_0$ is true; computing the size of a composite null at an arbitrary point instead of the supremum; assuming the MP region is always a tail; believing that Wilks' $\chi^2$ holds for every model; using the $\chi^2$ quantiles with the wrong tail (upper versus lower points).

Suggested homework. I(1, 3, 5, 8), II(1, 2, 4, 6), III(1, 4, 5, 8), IV(2, 3, 5, 6); Exam corner E2, E4, E5.

🔬 Research–Teaching NexusResearch-tutored · optional, about 60 minutes

Research question. If a field of research runs thousands of tests at level $5\%$, what fraction of its "significant" findings are false, and how does that fraction depend on the power of the studies?

Why it matters. Drug-discovery screens, genome-wide association studies and large programmes in psychology and economics test very many hypotheses, most of them null. Replication projects in several countries have found that a substantial share of published significant results do not hold up. Low power is one of the main reasons, and the Neyman–Pearson quantities of this section explain how.

What to do.

  1. Suppose a proportion $\pi$ of the hypotheses tested are false nulls (real effects), each test has level $\alpha$, and power $1-\beta$ against the real effects. Show that the expected fraction of rejections that are false is $$\mathrm{FDP}=\frac{\alpha(1-\pi)}{\alpha(1-\pi)+(1-\beta)\pi}.$$
  2. Simulate $10\,000$ one-sample $z$-tests (seeded generator): a fraction $\pi=0.1$ have true effect $\delta$, the rest have $\delta=0$. Choose $n$ to give power $0.8$, and then $0.2$. Compare the simulated false-discovery proportion with the formula.
  3. Among the significant results, compute the average estimated effect size and compare it with the true $\delta$. Explain the exaggeration you see when the power is low (the "winner's curse").
  4. Apply the Benjamini–Hochberg procedure (reject the $k$ smallest $p$-values, where $k$ is the largest index with $p_{(k)}\le k\,q/m$) at $q=0.1$, and report its realised false-discovery proportion.

Tools: Python with numpy (or a spreadsheet for step 1); the $p$-values come from $1-\Phi(z)$.

Hand in: A one-page note with the table of FDP (formula and simulation) for both power levels, the winner's-curse comparison, and the result of the Benjamini–Hochberg step.

Going further. Read about the Bayesian interpretation of the false-discovery rate, and about why the Benjamini–Hochberg procedure controls it when the $p$-values are independent (it uses the super-uniformity of Theorem 5.1).

Instructor note: with $\pi=0.1$ and $\alpha=0.05$, the formula gives $\mathrm{FDP}=0.045/(0.045+0.08)=0.36$ at power $0.8$ and $0.045/(0.045+0.02)=0.69$ at power $0.2$: at low power most "discoveries" are false. In the simulation the average significant estimate is close to $\delta$ at power $0.8$ but roughly doubled at power $0.2$.

🧪 Self-check quiz

0/5 correct

Q1. The size of a test $\phi$ of a composite null hypothesis $H_0:\theta\in\Theta_0$ is

Q2. For simple $H_0:f_0$ against simple $H_1:f_1$, the most powerful test of size $\alpha$ rejects when

Q3. For $X\sim\operatorname{Bin}(20,\tfrac12)$, $P(X\ge15)=0.0207$ and $P(X=14)=0.0370$. The size-$0.05$ test that rejects for $X\ge15$ and randomises at $X=14$ rejects there with probability about

Q4. For iid $N(\mu,1)$ data, there is no UMP test of $\mu=0$ against $\mu\ne0$ because

Q5. Three normal populations have unknown means and a common unknown variance. For $H_0:\mu_1=\mu_2=\mu_3$, Wilks' theorem gives $-2\log\Lambda\approx\chi^2_r$ with $r$ equal to