5.1 Most Powerful, UMP and Likelihood Ratio Tests
How to decide between two hypotheses with controlled error rates: size, power and $p$-values; the Neyman–Pearson lemma and most powerful tests; monotone likelihood ratio and uniformly most powerful one-sided tests; why two-sided UMP tests usually fail to exist; and likelihood ratio tests, with Wilks' theorem and the $z$, $t$, $\chi^2$ and $F$ tests as special cases.
A pharmaceutical regulator must decide whether a new tablet line meets the declared mean weight of $500$ mg. An engineer must decide whether a new alloy lasts longer than the old one. A geneticist must decide whether offspring counts agree with Mendel's $3:1$ ratio. In each case the data are random, so any rule that turns data into a decision will sometimes be wrong. It can reject a claim that is true, or accept one that is false. The theory of hypothesis testing does not try to avoid errors altogether, which is impossible. It fixes an upper bound $\alpha$ on the probability of the more serious error and then, among all rules that respect that bound, looks for the one that is least likely to make the other error. The question "which rule is best?" has a sharp answer when we compare two simple hypotheses: the Neyman–Pearson lemma says we should reject when the likelihood ratio is large. Much of this section consists of following that idea as far as it goes.
It goes a long way. For one-sided questions in families with a monotone likelihood ratio (every one-parameter exponential family, the uniform family, and many others), the Neyman–Pearson test does not depend on the particular alternative, and so it is uniformly most powerful. For two-sided questions it usually breaks down, and we show exactly why; one remedy is to restrict attention to unbiased tests. For problems with nuisance parameters we use the generalised likelihood ratio test, which reproduces the familiar $z$-, $t$-, $\chi^2$- and $F$-tests and, by Wilks' theorem, has an approximately $\chi^2$ null distribution in large samples. CSIR NET examines this material heavily. Part B questions ask for a size, a power, a critical value or a $p$-value. Part C questions ask whether a UMP test exists, whether a test is unbiased, whether a family has a monotone likelihood ratio, and what the null distribution of a likelihood ratio statistic is. We use the notation of Chapter 4: $X=(X_1,\dots,X_n)$ has a distribution $P_\theta$ with density or pmf $f(x;\theta)$, $\theta\in\Theta$, and $L(\theta)=f(x;\theta)$ is the likelihood. Upper $\alpha$ points are written $z_\alpha$, $t_{\nu,\alpha}$, $\chi^2_{\nu,\alpha}$ and $F_{\nu_1,\nu_2,\alpha}$ as in §4.3, and all quoted quantiles were computed with scipy.stats.
Hypotheses, Errors, Size and Power
Let $\Theta$ be split into two disjoint nonempty sets, $\Theta=\Theta_0\cup\Theta_1$.
- The null hypothesis is $H_0:\theta\in\Theta_0$ and the alternative hypothesis is $H_1:\theta\in\Theta_1$. A hypothesis is simple if its set contains a single point, and composite otherwise.
- A (randomised) test is a measurable function $\phi$ of the data with values in $[0,1]$. When $x$ is observed we reject $H_0$ with probability $\phi(x)$, using an independent auxiliary randomisation (for example a uniform random number) if $0\lt\phi(x)\lt1$.
- A test with values in $\{0,1\}$ is non-randomised. It has the form $\phi=\mathbf 1_C$, and the set $C=\{x:\phi(x)=1\}$ is its critical region (rejection region). The complement of $C$ is the acceptance region.
Most tests are written as "reject $H_0$ when $T(x)\gt c$" for a test statistic $T$ and a critical value $c$.
Randomised tests are rarely used in practice, but they are indispensable in the theory. With discrete data the probability of a critical region can take only countably many values, and randomisation fills the gaps, so that every size $\alpha$ can be attained exactly. They also make the set of tests convex, which is what makes the optimisation problems below solvable.
- Rejecting $H_0$ when it is true is a type I error. Accepting $H_0$ when it is false is a type II error.
- The power function of a test $\phi$ is $$\beta_\phi(\theta)=E_\theta[\phi(X)]=P_\theta(\text{reject }H_0),\qquad\theta\in\Theta.$$ For $\theta\in\Theta_0$ it is the probability of a type I error; for $\theta\in\Theta_1$ it is the power at $\theta$, and $1-\beta_\phi(\theta)$ is the probability of a type II error.
- The size of $\phi$ is $\sup_{\theta\in\Theta_0}\beta_\phi(\theta)$. The test has level $\alpha$ if its size is at most $\alpha$. The number $\alpha$ is called the significance level.
The roles of the two hypotheses are not symmetric. The error we control is the type I error, so $H_0$ should be the claim whose false rejection is more costly: "the drug is no better than placebo", "the batch meets specification", "the coin is fair". A test that fails to reject $H_0$ has not shown $H_0$ to be true. It has only found the data compatible with it, and when the power is low that is weak evidence.
Suppose large values of a statistic $T$ are evidence against $H_0$, and for each $\alpha$ the test rejects when $T\ge c_\alpha$. The $p$-value of the observed data $x$ is $$p(x)=\sup_{\theta\in\Theta_0}P_\theta\left(T(X)\ge T(x)\right).$$
It is the smallest significance level at which the observed data lead to rejection: $H_0$ is rejected at level $\alpha$ exactly when $p(x)\le\alpha$. In words, it is the probability, computed under $H_0$, of a result at least as extreme as the one observed.
With $p(x)$ as in Definition 5.3:
- for every $\theta\in\Theta_0$ and every $u\in[0,1]$, $P_\theta\left(p(X)\le u\right)\le u$. So the test "reject when $p(X)\le\alpha$" has level $\alpha$;
- if $H_0=\{\theta_0\}$ is simple and $T$ has a continuous distribution under $\theta_0$, then $p(X)\sim U(0,1)$ under $H_0$.
Proof. Fix $\theta\in\Theta_0$ and put $G(t)=P_\theta(T\ge t)$, a non-increasing function of $t$. Since $p(x)\ge G(T(x))$, it suffices to show $P_\theta\left(G(T)\le u\right)\le u$. The set $A=\{t:G(t)\le u\}$ is an up-set: if $t\in A$ and $t'\gt t$ then $G(t')\le G(t)\le u$. So $A$ is empty, or a half-line $[t^*,\infty)$, or a half-line $(t^*,\infty)$. In the first case the probability is $0$. In the second, $P_\theta(T\in A)=P_\theta(T\ge t^*)=G(t^*)\le u$. In the third, $P_\theta(T\in A)=P_\theta(T\gt t^*)=\lim_{t\downarrow t^*}G(t)\le u$, since $G(t)\le u$ for every $t\gt t^*$. This proves (1). For (2), $p(X)=P_{\theta_0}(T\ge T(X))=1-F(T(X))$ where $F$ is the continuous cdf of $T$, and $F(T)\sim U(0,1)$ by the probability integral transform (§1.3), so $1-F(T)\sim U(0,1)$ as well. ∎
For discrete statistics the inequality in (1) is usually strict. In Example 1 the possible $p$-values are $P_{0.5}(X\ge k)$, and $P(p(X)\le0.05)=0.0207$. This is one reason why exact tests for discrete data are conservative.
(a) Suppose that for each $\theta_0\in\Theta$ there is a non-randomised level-$\alpha$ test of $H_0:\theta=\theta_0$ with acceptance region $A(\theta_0)$. Then $C(x)=\{\theta:x\in A(\theta)\}$ is a confidence set of level $1-\alpha$.
(b) Conversely, if $C(X)$ is a confidence set of level $1-\alpha$, then for each $\theta_0$ the test that accepts $H_0:\theta=\theta_0$ exactly when $\theta_0\in C(x)$ has level $\alpha$.
Proof. In both directions, $\theta\in C(x)\iff x\in A(\theta)$. Hence, for every $\theta$, $$P_\theta\left(\theta\in C(X)\right)=P_\theta\left(X\in A(\theta)\right)\ge1-\alpha,$$ because $1-P_\theta(X\in A(\theta))$ is the size of the test of $H_0:\theta$; and the same identity read backwards gives (b). ∎
So the $t$-interval of §4.3 is the set of $\mu_0$ not rejected by the two-sided $t$-test, and the Wilson interval for a proportion is the set of $p_0$ not rejected by the score test of §5.2. "A $95\%$ interval excludes $\theta_0$" and "the $5\%$ test rejects $\theta_0$" are the same statement.
The Neyman–Pearson Lemma and Most Powerful Tests
We begin with the simplest problem: one distribution against one other. Write $f_0=f(\cdot;\theta_0)$ and $f_1=f(\cdot;\theta_1)$, densities with respect to Lebesgue measure or counting measure (a pmf). Integrals $\int\phi f_0$ then mean sums in the discrete case.
The idea of the lemma is an economic one. Each point $x$ "costs" $f_0(x)$ of the type I error budget $\alpha$ and "buys" $f_1(x)$ of power. To get the most power for a fixed budget, buy the points with the best ratio $f_1(x)/f_0(x)$ first.
Let $f_0,f_1$ be densities (or pmfs) and $0\le\alpha\le1$.
- Sufficiency. If a test $\phi$ satisfies, for some constant $k\ge0$, $$\phi(x)=\begin{cases}1,&f_1(x)\gt kf_0(x),\\0,&f_1(x)\lt kf_0(x),\end{cases}\qquad(\ast)$$ and $E_{\theta_0}\phi=\alpha$, then $\phi$ is most powerful at level $\alpha$.
- Existence. For every $\alpha\in(0,1]$ there are constants $k\ge0$ and $\gamma\in[0,1]$ such that the test $(\ast)$ with $\phi(x)=\gamma$ on the boundary $\{f_1=kf_0\}$ has size exactly $\alpha$. (For $\alpha=0$, the test $\phi=\mathbf 1\{f_1\gt0=f_0\}$ is MP.)
- Necessity. If $\phi'$ is MP at level $\alpha$ and $k$ is as in (2), then $\phi'$ satisfies $(\ast)$ for almost every $x$ in $\{f_1\ne kf_0\}$ (with respect to $f_0+f_1$). Moreover $E_{\theta_0}\phi'=\alpha$, unless some test of size less than $\alpha$ has power $1$.
Proof. (1) Let $\phi'$ be any test with $E_{\theta_0}\phi'\le\alpha$. We claim that $$\left(\phi(x)-\phi'(x)\right)\left(f_1(x)-kf_0(x)\right)\ge0\quad\text{for every }x.$$ Where $f_1\gt kf_0$, $\phi=1\ge\phi'$; where $f_1\lt kf_0$, $\phi=0\le\phi'$; where $f_1=kf_0$ the product is $0$. Integrating, $$\int(\phi-\phi')f_1\ \ge\ k\int(\phi-\phi')f_0=k\left(\alpha-E_{\theta_0}\phi'\right)\ge0.$$ The left side is $E_{\theta_1}\phi-E_{\theta_1}\phi'$, so $\phi$ is at least as powerful as $\phi'$.
(2) Let $R=f_1(X)/f_0(X)$, defined under $P_{\theta_0}$ (where $f_0\gt0$ with probability one), and let $G(k)=P_{\theta_0}(R\gt k)$. $G$ is non-increasing and right-continuous, $G(k)\to0$ as $k\to\infty$, and $G(0^-)=1$. Take $k=\inf\{t\ge0:G(t)\le\alpha\}$. Then $G(k)\le\alpha\le G(k^-)=P_{\theta_0}(R\ge k)$, and the jump $G(k^-)-G(k)=P_{\theta_0}(R=k)$. If this jump is zero put $\gamma=0$; otherwise put $$\gamma=\frac{\alpha-P_{\theta_0}(R\gt k)}{P_{\theta_0}(R=k)}\in[0,1].$$ Then $E_{\theta_0}\phi=P_{\theta_0}(R\gt k)+\gamma P_{\theta_0}(R=k)=\alpha$. (Points with $f_0=0\lt f_1$ have $R=\infty$ and are always in the critical region; they cost nothing.)
(3) Let $\phi$ be the test of (2) and $\phi'$ another MP test at level $\alpha$. Both have the same power, so the integral in (1) satisfies $$0\le\int(\phi-\phi')(f_1-kf_0)=E_{\theta_1}\phi-E_{\theta_1}\phi'-k\left(\alpha-E_{\theta_0}\phi'\right)=-k\left(\alpha-E_{\theta_0}\phi'\right)\le0.$$ Hence the integral of the non-negative function $(\phi-\phi')(f_1-kf_0)$ is zero, so this function vanishes almost everywhere, and on $\{f_1\ne kf_0\}$ this forces $\phi'=\phi$. Also $k(\alpha-E_{\theta_0}\phi')=0$. If $k\gt0$ this gives $E_{\theta_0}\phi'=\alpha$. If $k=0$, then $\phi=1$ wherever $f_1\gt0$, so $\phi$ has power $1$ with size $\alpha$, and $\phi'$, being equally powerful, also has power $1$; this is the exceptional case. ∎
- If $\phi$ is MP at level $\alpha$ for $f_0$ against $f_1$, its power $\beta=E_{\theta_1}\phi$ satisfies $\beta\ge\alpha$. If $0\lt\alpha\lt1$ and $f_0\ne f_1$ (on a set of positive probability), then $\beta\gt\alpha$.
- If $T$ is sufficient for $\{f_0,f_1\}$, the MP test can be taken to be a function of $T$ alone.
Proof. (1) The trivial test $\phi'\equiv\alpha$ has size $\alpha$ and power $\alpha$, so $\beta\ge\alpha$. Suppose $\beta=\alpha$ with $0\lt\alpha\lt1$. Then $\phi'\equiv\alpha$ is also MP, and by Theorem 5.3(3) it agrees with the NP test on $\{f_1\ne kf_0\}$. But the NP test takes only the values $0$ and $1$ there, and $\phi'$ takes the value $\alpha\notin\{0,1\}$. So $\{f_1\ne kf_0\}$ has probability zero, that is, $f_1=kf_0$ almost everywhere; integrating gives $k=1$ and $f_1=f_0$. (2) By the factorisation theorem (§4.2), $f(x;\theta)=g(T(x);\theta)h(x)$, so $f_1(x)/f_0(x)=g(T(x);\theta_1)/g(T(x);\theta_0)$ wherever $h(x)\gt0$. The NP test $(\ast)$, with a boundary value $\gamma$ that is constant, is therefore a function of $T(x)$. ∎
Monotone Likelihood Ratio and Uniformly Most Powerful Tests
A UMP test is MP against each simple alternative $\theta_1\in\Theta_1$ at once. Example 3 suggests when this can happen: the NP critical region must be the same for all $\theta_1\in\Theta_1$. The following property guarantees this for one-sided problems.
(a) A one-parameter exponential family $f(x;\theta)=h(x)\exp\{c(\theta)T(x)-A(\theta)\}$ with $c$ strictly increasing has MLR in $T$. If $c$ is strictly decreasing it has MLR in $-T$.
(b) Consequently, for iid samples, the following have MLR in the stated statistic: $\operatorname{Bin}(m,p)$, $\operatorname{Poi}(\lambda)$, $\operatorname{Geo}(p)$ (in $-\sum X_i$), $N(\mu,\sigma_0^2)$ in $\sum X_i$, $N(\mu_0,\sigma^2)$ in $\sum(X_i-\mu_0)^2$, $\operatorname{Exp}$ with mean $\theta$ in $\sum X_i$, and $\operatorname{Gamma}$ with known shape in $\sum X_i$ (as a function of the mean). The non-exponential families $U(0,\theta)$ and the shifted exponential $e^{-(x-\theta)}\mathbf 1\{x\gt\theta\}$ have MLR in $X_{(n)}$ and $X_{(1)}$ respectively. The Cauchy location family does not have MLR in $X$ (Example 4).
Proof. (a) For $\theta\lt\theta'$, $$\frac{f(x;\theta')}{f(x;\theta)}=e^{(c(\theta')-c(\theta))T(x)}\,e^{A(\theta)-A(\theta')}$$ with $c(\theta')-c(\theta)\gt0$, an increasing function of $T(x)$; both densities are positive on the same set $\{h\gt0\}$. (b) The iid sample from an exponential family is again an exponential family with $T=\sum T(X_i)$ and the same $c$. For example, $\operatorname{Poi}(\lambda)$ has $c(\lambda)=\log\lambda$, and $\operatorname{Geo}(p)$, with pmf $p(1-p)^{x}$, has $c(p)=\log(1-p)$, which is decreasing. For $U(0,\theta)$ and $\theta\lt\theta'$, $$\frac{f(x;\theta')}{f(x;\theta)}=\frac{\theta^n}{\theta'^n}\cdot\frac{\mathbf 1\{x_{(n)}\lt\theta'\}}{\mathbf 1\{x_{(n)}\lt\theta\}}=\begin{cases}(\theta/\theta')^n,&x_{(n)}\lt\theta,\\\infty,&\theta\le x_{(n)}\lt\theta',\end{cases}$$ non-decreasing in $x_{(n)}$ (for $x_i\gt0$). The shifted exponential is similar, with ratio $e^{n(\theta'-\theta)}$ for $x_{(1)}\gt\theta'$ and $0$ for $\theta\lt x_{(1)}\le\theta'$. ∎
Let the family have MLR in $T$, and let $0\lt\alpha\lt1$. For testing $H_0:\theta\le\theta_0$ against $H_1:\theta\gt\theta_0$, consider $$\phi(x)=\begin{cases}1,&T(x)\gt c,\\\gamma,&T(x)=c,\\0,&T(x)\lt c,\end{cases}\qquad\text{with }E_{\theta_0}\phi=\alpha.$$
- Such $c$ and $\gamma\in[0,1]$ exist, and $\phi$ is UMP at level $\alpha$.
- The power function $\beta_\phi(\theta)$ is non-decreasing in $\theta$, and strictly increasing on the set where $0\lt\beta_\phi(\theta)\lt1$.
- For $\theta\lt\theta_0$, $\phi$ minimises $E_\theta\phi'$ among all tests $\phi'$ with $E_{\theta_0}\phi'=\alpha$.
For $H_0:\theta\ge\theta_0$ against $H_1:\theta\lt\theta_0$ the UMP test rejects for small $T$, with all inequalities reversed.
Proof. Existence of $c,\gamma$. This is the argument of Theorem 5.3(2) applied to the cdf of $T$ under $\theta_0$: take $c$ with $P_{\theta_0}(T\gt c)\le\alpha\le P_{\theta_0}(T\ge c)$ and $\gamma=\left(\alpha-P_{\theta_0}(T\gt c)\right)/P_{\theta_0}(T=c)$ when $P_{\theta_0}(T=c)\gt0$.
$\phi$ is an NP test for every pair $\theta_1\lt\theta_2$. Write the ratio as $f(x;\theta_2)/f(x;\theta_1)=g(T(x))$ with $g$ non-decreasing, and put $k=g(c)$. If $g(T(x))\gt k$ then $T(x)\gt c$ (otherwise $g(T(x))\le g(c)$), so $\phi(x)=1$. If $g(T(x))\lt k$ then $T(x)\lt c$, so $\phi(x)=0$. Thus $\phi$ has the form $(\ast)$ for $f_0=f(\cdot;\theta_1)$, $f_1=f(\cdot;\theta_2)$, and by Theorem 5.3(1) it is MP at level $\beta_\phi(\theta_1)$ for $\theta_1$ against $\theta_2$. (If $g(c)=\infty$, then $\phi$ rejects only where $f(x;\theta_1)=0$, so $\beta_\phi(\theta_1)=0$ and the inequality needed below holds trivially.)
Monotone power (2). For $\theta_1\lt\theta_2$, $\phi$ is MP at level $\beta_\phi(\theta_1)$, so by Theorem 5.4(1) $\beta_\phi(\theta_2)\ge\beta_\phi(\theta_1)$, with strict inequality when $0\lt\beta_\phi(\theta_1)\lt1$ (the two distributions differ by the definition of MLR).
UMP (1). By (2), $\sup_{\theta\le\theta_0}\beta_\phi(\theta)=\beta_\phi(\theta_0)=\alpha$, so $\phi$ has level $\alpha$. Let $\phi'$ be any level-$\alpha$ test of $H_0:\theta\le\theta_0$. Then $E_{\theta_0}\phi'\le\alpha$. Fix $\theta_1\gt\theta_0$. Since $\phi$ is MP at level $\alpha$ for $\theta_0$ against $\theta_1$, $\beta_\phi(\theta_1)\ge\beta_{\phi'}(\theta_1)$. As $\theta_1\gt\theta_0$ was arbitrary, $\phi$ is UMP.
(3). For $\theta\lt\theta_0$ apply the lemma to $1-\phi$ with the roles of the hypotheses exchanged: $1-\phi$ rejects "$\theta_0$" in favour of "$\theta$" when $T\lt c$, which is the NP form for $f_0=f(\cdot;\theta_0)$, $f_1=f(\cdot;\theta)$ because the ratio $f(x;\theta)/f(x;\theta_0)$ is non-increasing in $T$. So $1-\phi$ maximises $E_\theta(1-\phi')$ among tests with $E_{\theta_0}(1-\phi')=1-\alpha$. ∎
Proof. Suppose $\phi$ were UMP. Fix $\mu_1\gt\mu_0$. The NP test of $\mu_0$ against $\mu_1$ at level $\alpha$ is $\phi_+=\mathbf 1\{\bar x\gt\mu_0+z_\alpha\sigma/\sqrt n\}$, and the likelihood ratio $\exp\{n(\mu_1-\mu_0)\bar x/\sigma^2-\text{const}\}$ has a continuous distribution, so the boundary $\{f_1=kf_0\}$ has probability zero. No test of size less than $\alpha$ has power $1$ (the normal densities are positive everywhere). Since $\phi$ is UMP, it is MP for this pair, and by Theorem 5.3(3) $\phi=\phi_+$ almost everywhere. Repeating the argument with $\mu_1\lt\mu_0$ gives $\phi=\phi_-=\mathbf 1\{\bar x\lt\mu_0-z_\alpha\sigma/\sqrt n\}$ almost everywhere. But $\phi_+$ and $\phi_-$ differ on the set $\{\bar x\gt\mu_0+z_\alpha\sigma/\sqrt n\}$, which has positive probability. This contradiction proves the theorem. ∎
The same argument works in any family where the MP tests against alternatives on the two sides of $\theta_0$ are essentially unique and different. It does not mean that two-sided UMP tests never exist, as the next example shows.
An unbiased test is never more likely to reject when $H_0$ is true than when it is false. This is a mild requirement, and it rules out exactly the one-sided tests that caused the trouble in Theorem 5.7.
- A UMP level-$\alpha$ test is unbiased.
- In the normal model of Theorem 5.7, the equal-tailed test $\phi=\mathbf 1\left\{\sqrt n|\bar x-\mu_0|/\sigma\gt z_{\alpha/2}\right\}$ is unbiased, with power function $$\beta(\mu)=\Phi(-z_{\alpha/2}-\delta)+\Phi(-z_{\alpha/2}+\delta),\qquad\delta=\frac{\sqrt n(\mu-\mu_0)}{\sigma},$$ which is symmetric in $\delta$ and strictly increasing in $|\delta|$. The one-sided test $\phi_+$ is not unbiased for $H_1:\mu\ne\mu_0$.
- (Statement.) In a one-parameter exponential family with natural statistic $T$, the test that rejects when $T\lt c_1$ or $T\gt c_2$ (randomising at $c_1,c_2$), with $c_1,c_2$ chosen so that $E_{\theta_0}\phi=\alpha$ and $E_{\theta_0}[T\phi]=\alpha E_{\theta_0}[T]$, is UMPU for $H_0:\theta=\theta_0$ against $H_1:\theta\ne\theta_0$. For the normal mean these conditions give the equal-tailed $z$-test; for the normal variance they give unequal tails. The one- and two-sided $t$-tests are UMPU in the presence of the unknown $\sigma$.
Proof. (1) Let $\phi$ be UMP at level $\alpha$. The test $\phi'\equiv\alpha$ has level $\alpha$, so $\beta_\phi(\theta)\ge\beta_{\phi'}(\theta)=\alpha$ for every $\theta\in\Theta_1$. (2) Under $\mu$, $Z=\sqrt n(\bar X-\mu_0)/\sigma\sim N(\delta,1)$, so $\beta=P(Z\gt z_{\alpha/2})+P(Z\lt-z_{\alpha/2})$, that is, $$\beta=\Phi(\delta-z_{\alpha/2})+\Phi(-\delta-z_{\alpha/2}),$$ which is symmetric in $\delta$. Writing $\varphi$ for the standard normal density, $$\frac{d\beta}{d\delta}=\varphi(\delta-z_{\alpha/2})-\varphi(\delta+z_{\alpha/2}).$$ For $\delta\gt0$ we have $|\delta-z_{\alpha/2}|\lt\delta+z_{\alpha/2}$, so the derivative is positive. Hence $\beta$ has its minimum $\alpha$ at $\delta=0$ and $\beta(\mu)\gt\alpha$ for $\mu\ne\mu_0$. For $\phi_+$, $\beta_+(\mu)=\Phi(\delta-z_\alpha)\lt\Phi(-z_\alpha)=\alpha$ when $\delta\lt0$. (3) The proof uses a generalised form of the Neyman–Pearson lemma with two side conditions; we only quote the result. ∎
Likelihood Ratio Tests and Wilks' Theorem
Most practical problems are composite on both sides and contain nuisance parameters, such as the unknown variance when the mean is tested. The Neyman–Pearson ratio $f_1/f_0$ is then not defined, because neither hypothesis specifies a single distribution. The natural generalisation compares the best explanation of the data under $H_0$ with the best explanation overall.
The likelihood ratio statistic for $H_0:\theta\in\Theta_0$ against $H_1:\theta\in\Theta\setminus\Theta_0$ is $$\Lambda(x)=\frac{\sup_{\theta\in\Theta_0}L(\theta)}{\sup_{\theta\in\Theta}L(\theta)}=\frac{L(\hat\theta_0)}{L(\hat\theta)},$$ where $\hat\theta_0$ is the MLE under $H_0$ and $\hat\theta$ the unrestricted MLE (when they exist). Then $0\le\Lambda\le1$. The likelihood ratio test (LRT) rejects $H_0$ when $\Lambda(x)\le\lambda_\alpha$, equivalently when $-2\log\Lambda(x)\ge c_\alpha$, with the constant chosen so that $\sup_{\theta\in\Theta_0}P_\theta\left(\Lambda(X)\le\lambda_\alpha\right)=\alpha$ (or $\le\alpha$).
Because $L(\theta)=g(T(x);\theta)h(x)$ for a sufficient statistic $T$ (§4.2), $\Lambda$ is a function of $T$.
Two remarks. First, the denominator uses all of $\Theta$, not $\Theta_1$; this changes nothing when $\Theta_0$ is "small" (a point or a lower-dimensional set) and the likelihood is continuous, and it makes $\Lambda\le1$. Second, when $H_0$ and $H_1$ are both simple, $\Lambda=\min\left(1,f_0/f_1\right)$, so the LRT is the Neyman–Pearson test. For one-sided problems in MLR families the LRT usually coincides with the UMP test (Exercise IV.1 gives the two-sided analogue for the exponential model).
Let $X_1,\dots,X_n$ be iid $N(\mu,\sigma^2)$ with both parameters unknown, $S^2=\frac1{n-1}\sum(X_i-\bar X)^2$, and $\hat\sigma^2=\frac{n-1}{n}S^2$.
- The $t$-test. For $H_0:\mu=\mu_0$ against $\mu\ne\mu_0$, $$\Lambda=\left(1+\frac{T^2}{n-1}\right)^{-n/2},\qquad T=\frac{\sqrt n(\bar X-\mu_0)}{S}\sim t_{n-1}\ \text{under }H_0,$$ so the LRT rejects when $|T|\gt t_{n-1,\alpha/2}$.
- The $\chi^2$-test for a variance. For $H_0:\sigma^2=\sigma_0^2$ against $\sigma^2\ne\sigma_0^2$, with $W=\hat\sigma^2/\sigma_0^2$, $$\Lambda=W^{n/2}e^{-n(W-1)/2},$$ so the LRT rejects when $(n-1)S^2/\sigma_0^2=nW$ lies outside an interval $[c_1,c_2]$, where $c_1,c_2$ satisfy $P(c_1\le\chi^2_{n-1}\le c_2)=1-\alpha$ and $\frac{c_1}{n}-\log c_1=\frac{c_2}{n}-\log c_2$. The equal-tailed test with $c_1=\chi^2_{n-1,1-\alpha/2}$, $c_2=\chi^2_{n-1,\alpha/2}$ is an approximation to it.
- The pooled two-sample $t$-test. For independent samples $X_1,\dots,X_m\sim N(\mu_1,\sigma^2)$ and $Y_1,\dots,Y_n\sim N(\mu_2,\sigma^2)$ and $H_0:\mu_1=\mu_2$, $$\Lambda=\left(1+\frac{T^2}{m+n-2}\right)^{-(m+n)/2},\qquad T=\frac{\bar X-\bar Y}{S_p\sqrt{\frac1m+\frac1n}}\sim t_{m+n-2}\ \text{under }H_0,$$ where $S_p^2=\frac{(m-1)S_1^2+(n-1)S_2^2}{m+n-2}$.
- The $F$-test. For independent samples $N(\mu_1,\sigma_1^2)$ and $N(\mu_2,\sigma_2^2)$ and $H_0:\sigma_1^2=\sigma_2^2$, $\Lambda$ is a function of $F=S_1^2/S_2^2$ that is small exactly when $F$ is far from a value near $1$ on either side. Under $H_0$, $F\sim F_{m-1,n-1}$, and the usual test rejects when $F\gt F_{m-1,n-1,\alpha/2}$ or $F\lt F_{m-1,n-1,1-\alpha/2}$.
Proof. The normal log-likelihood is $\ell(\mu,\sigma^2)=-\frac n2\log(2\pi\sigma^2)-\frac{1}{2\sigma^2}\sum(x_i-\mu)^2$. For fixed $\mu$ it is maximised at $\sigma^2=\frac1n\sum(x_i-\mu)^2$, giving the profile value $-\frac n2\log\left(\frac{2\pi e}{n}\sum(x_i-\mu)^2\right)$. So, for any restriction on $\mu$ alone, $$\Lambda=\left(\frac{\min_{\Theta}\sum(x_i-\mu)^2}{\min_{\Theta_0}\sum(x_i-\mu)^2}\right)^{n/2}.$$
(1) The unrestricted minimum is $\sum(x_i-\bar x)^2=n\hat\sigma^2$, and under $H_0$ it is $\sum(x_i-\mu_0)^2=n\hat\sigma^2+n(\bar x-\mu_0)^2$. Hence $\Lambda^{-2/n}=1+\frac{(\bar x-\mu_0)^2}{\hat\sigma^2}=1+\frac{n(\bar x-\mu_0)^2}{(n-1)s^2}=1+\frac{t^2}{n-1}$. This is an increasing function of $|t|$, so $\Lambda\le\lambda_\alpha\iff|t|\ge c$. Under $H_0$, $T\sim t_{n-1}$ for every $\sigma$ (§4.1), so the size is $\alpha$ for $c=t_{n-1,\alpha/2}$, whatever the nuisance parameter.
(2) Unrestricted, $\hat\mu=\bar x$ and $\sigma^2=\hat\sigma^2$, and $L=(2\pi\hat\sigma^2)^{-n/2}e^{-n/2}$. Under $H_0$, $\hat\mu=\bar x$ and $L=(2\pi\sigma_0^2)^{-n/2}e^{-n\hat\sigma^2/(2\sigma_0^2)}$. The ratio is $W^{n/2}e^{-n(W-1)/2}$ with $W=\hat\sigma^2/\sigma_0^2$. Now $\log\Lambda=\frac n2(\log W-W+1)$ is increasing for $W\lt1$ and decreasing for $W\gt1$, with maximum $0$ at $W=1$. So $\Lambda\le\lambda$ iff $W\le w_1$ or $W\ge w_2$, where $w_1\lt1\lt w_2$ have $\log w_1-w_1=\log w_2-w_2$. Writing $c_i=nw_i$ gives the stated condition. Under $H_0$, $nW=(n-1)S^2/\sigma_0^2\sim\chi^2_{n-1}$.
(3) The same computation with the pooled log-likelihood: the unrestricted minimum of the sum of squares is $Q=\sum(x_i-\bar x)^2+\sum(y_j-\bar y)^2=(m+n-2)s_p^2$, and under $H_0$ (common mean, estimated by the grand mean) it is $Q+\frac{mn}{m+n}(\bar x-\bar y)^2$. Since $\frac{mn}{m+n}=\left(\frac1m+\frac1n\right)^{-1}$, $\Lambda^{-2/(m+n)}=1+\frac{(\bar x-\bar y)^2}{(1/m+1/n)Q}=1+\frac{t^2}{m+n-2}$. The law of $T$ under $H_0$ is §4.1.
(4) With $\hat\sigma_1^2=\frac{m-1}mS_1^2$, $\hat\sigma_2^2=\frac{n-1}nS_2^2$, and, under $H_0$, $\hat\sigma^2=\frac{m\hat\sigma_1^2+n\hat\sigma_2^2}{m+n}$, one finds $$\Lambda=\frac{(\hat\sigma_1^2)^{m/2}(\hat\sigma_2^2)^{n/2}}{(\hat\sigma^2)^{(m+n)/2}}=\frac{(m+n)^{(m+n)/2}}{m^{m/2}n^{n/2}}\cdot\frac{v^{m/2}}{(1+v)^{(m+n)/2}},\qquad v=\frac{m\hat\sigma_1^2}{n\hat\sigma_2^2}.$$ The function $v^{m/2}(1+v)^{-(m+n)/2}$ increases up to $v=m/n$ and then decreases, so $\Lambda$ is small exactly when $v$, and hence $F=\frac{(m-1)n}{m(n-1)}\,v$, is in either tail. ∎
Let $X_1,\dots,X_n$ be iid with density $f(x;\theta)$, $\theta\in\Theta\subseteq\mathbb{R}^p$ open, satisfying the regularity conditions of the asymptotic theory of the MLE (§4.2): common support not depending on $\theta$, three continuous derivatives in $\theta$ with integrable bounds, and a positive definite Fisher information matrix. Let $H_0$ specify $r$ of the coordinates, $\Theta_0=\{\theta:\theta_1=\theta_{0,1},\dots,\theta_r=\theta_{0,r}\}$ with $1\le r\le p$, and let the true $\theta$ be an interior point of $\Theta_0$ (in the remaining $p-r$ coordinates). Then, under $H_0$, $$-2\log\Lambda_n\xrightarrow{d}\chi^2_r,\qquad r=\dim\Theta-\dim\Theta_0.$$ Consequently the test that rejects when $-2\log\Lambda_n\gt\chi^2_{r,\alpha}$ has asymptotic size $\alpha$.
Proof for $p=r=1$. Let $\ell_n(\theta)=\sum\log f(X_i;\theta)$ and $H_0:\theta=\theta_0$. By §4.2 there is a consistent root $\hat\theta_n$ of $\ell_n'(\theta)=0$ with $\sqrt n(\hat\theta_n-\theta_0)\xrightarrow dN(0,1/I(\theta_0))$. Expand $\ell_n(\theta_0)$ about $\hat\theta_n$ by Taylor's theorem: $$\ell_n(\theta_0)=\ell_n(\hat\theta_n)+\ell_n'(\hat\theta_n)(\theta_0-\hat\theta_n)+\tfrac12\ell_n''(\theta_n^*)(\theta_0-\hat\theta_n)^2$$ with $\theta_n^*$ between $\theta_0$ and $\hat\theta_n$. The middle term vanishes, so $$-2\log\Lambda_n=2\left[\ell_n(\hat\theta_n)-\ell_n(\theta_0)\right]=\left(-\frac1n\ell_n''(\theta_n^*)\right)\cdot\left(\sqrt n(\hat\theta_n-\theta_0)\right)^2.$$ Since $\theta_n^*\to\theta_0$ in probability, the bound on the third derivative and the weak law give $-\frac1n\ell_n''(\theta_n^*)\xrightarrow{P}I(\theta_0)$, exactly as in the proof of asymptotic normality in §4.2. And $\sqrt{nI(\theta_0)}(\hat\theta_n-\theta_0)\xrightarrow dZ\sim N(0,1)$. By Slutsky's theorem and the continuous mapping theorem (§2.3), $-2\log\Lambda_n\xrightarrow dZ^2\sim\chi^2_1$.
Sketch for general $p,r$. Let $\theta^*$ be the true value and $Q(u)=n\,u^{\mathsf T}I(\theta^*)u$. The same expansion, applied to both suprema, gives $$-2\log\Lambda_n\approx Q(\hat\theta-\theta^*)-Q(\hat\theta_0-\theta^*).$$ In the coordinates in which $\sqrt n\,I(\theta^*)^{1/2}(\hat\theta-\theta^*)$ is approximately $N_p(0,I_p)$, the restricted MLE is approximately the orthogonal projection onto a $(p-r)$-dimensional subspace, so the difference is the squared length of the projection onto an $r$-dimensional subspace. The squared length of the projection of a standard normal vector onto an $r$-dimensional subspace is $\chi^2_r$ (§7.1). ∎
Wilks' theorem is the reason likelihood ratio tests are used everywhere: one formula, one reference distribution, and the degrees of freedom are found by counting free parameters. The chi-square tests of §5.2 are its most famous instance. The hypotheses matter, though. The theorem fails when the support depends on $\theta$ (Exercise IV.6) and when the true value lies on the boundary of $\Theta$ (⚠️ box).
- Set up. Write the model, $\Theta_0$ and $\Theta_1$. Are the hypotheses simple, one-sided or two-sided? Are there nuisance parameters?
- Simple against simple: compute $f_1/f_0$, simplify it to a monotone function of a statistic $T$ (if possible), and find $c$ (and $\gamma$ for discrete $T$) from the null law of $T$.
- One-sided, one parameter: check MLR (exponential family with monotone $c(\theta)$, or $U(0,\theta)$-type families). If it holds, the Karlin–Rubin test is UMP, and its size is computed at the boundary $\theta_0$.
- Two-sided: expect no UMP test (Theorem 5.7); use the equal-tailed or UMPU test. Exception: families where the likelihood ratio is constant on the support, such as $U(0,\theta)$.
- Nuisance parameters or several parameters: compute $\Lambda$, simplify it to a known pivot ($t$, $\chi^2$, $F$) if possible; otherwise use $-2\log\Lambda\approx\chi^2_r$ with $r$ = number of restrictions.
- Report: the critical region, the size, the power at relevant alternatives, and the $p$-value.
- "Accept $H_0$" is not "$H_0$ is true". With $n=20$ in Example 1 the power at $p=0.6$ is $0.13$, so failing to reject says very little.
- A $p$-value is not $P(H_0\mid\text{data})$. It is a probability computed under $H_0$; the posterior probability of $H_0$ needs a prior (§4.3) and can be far larger than $p$.
- Size versus level. The size of a composite null is a supremum over $\Theta_0$, not the value at one point. With discrete data non-randomised tests usually cannot have size exactly $\alpha$: in Example 1 the sizes available near $0.05$ are $0.0207$ and $0.0577$.
- MP regions need not be tails. For the Cauchy location family the MP region for $\theta=0$ against $\theta=1$ is the bounded interval $(1,3)$ (Example 4); a monotone likelihood ratio is what makes tails optimal.
- No MLR outside the right families. Cauchy location has no MLR; an exponential family has MLR only in the direction in which $c(\theta)$ is monotone (geometric: in $-\sum X_i$); the curved family $N(\theta,\theta^2)$ has a likelihood ratio depending on both $\sum X_i$ and $\sum X_i^2$, so Theorem 5.5 does not apply to it.
- "Two-sided UMP tests never exist" is false. They fail for the normal mean (Theorem 5.7), but exist for $U(0,\theta)$ (Example 6) and for the shifted exponential (Exercise III.8).
- One-sided tests are biased against two-sided alternatives. $\phi_+$ has power below $\alpha$ for every $\mu\lt\mu_0$ (Theorem 5.8(2)).
- MP tests need not be unique. When $f_1/f_0$ has atoms (shifted exponential, uniform), many tests of the same size are MP (Exercise II.6).
- Wilks fails when the support depends on $\theta$. For $U(0,\theta)$ and $H_0:\theta=\theta_0$, $-2\log\Lambda=-2n\log(X_{(n)}/\theta_0)\sim\chi^2_2$ exactly, not $\chi^2_1$ (Exercise IV.6).
- Wilks fails on the boundary. For $N(\mu,1)$ with $\Theta=[0,\infty)$ and $H_0:\mu=0$, $-2\log\Lambda=n\bar X^2\mathbf 1\{\bar X\gt0\}$, whose law is the mixture $\frac12\delta_0+\frac12\chi^2_1$, not $\chi^2_1$.
- The exact LRT for a variance is not equal-tailed. For $n=25$ its tails are $0.0379$ and $0.0121$ (Example 7).
- Statistical is not practical significance. With $n=10^6$ tablets a mean of $500.02$ mg is "highly significant" against $500$ but irrelevant. Report an effect size or an interval as well.
Exercises
- 1 $X\sim\operatorname{Bin}(10,p)$. The test of $H_0:p=0.5$ against $H_1:p=0.8$ rejects when $X\ge8$. Find its size, its power and the probability of a type II error.
- 2 $X_1,\dots,X_{16}$ are iid $N(\mu,9)$. The test of $H_0:\mu=10$ against $H_1:\mu\gt10$ rejects when $\bar X\gt11.5$. Find its size and its power at $\mu=12$.
- 3 A single observation $X$ comes from $U(0,\theta)$. The test of $H_0:\theta=1$ rejects when $X\gt0.9$. Find the power function $\beta(\theta)$ for all $\theta\gt0$, the size, and the power at $\theta=2$.
- 4 For the one-sided $z$-test with observed statistic $z$, show that the $p$-value is $1-\Phi(z)$ and that it is $U(0,1)$ under $H_0$. Find the one- and two-sided $p$-values for $z=1.8$.
- 5 With $\sigma=10$ known, how large must $n$ be for the level-$0.01$ test of $H_0:\mu=50$ against $H_1:\mu\gt50$ to have power $0.95$ at $\mu=55$?
- 6 Show that if $\phi_1,\phi_2$ have level $\alpha$ then so does $\lambda\phi_1+(1-\lambda)\phi_2$ for $\lambda\in[0,1]$. Show by an example that $\max(\phi_1,\phi_2)$ need not have level $\alpha$.
- 7 A $95\%$ confidence interval for a proportion is $[0.31,\ 0.47]$. What does the dual test decide about $H_0:p=0.5$ and $H_0:p=0.4$ at level $5\%$? Show that the two-sided $z$-test accepts $\mu_0$ exactly when $\mu_0\in\bar x\pm z_{\alpha/2}\sigma/\sqrt n$.
- 8 A single $X\sim\operatorname{Poi}(\lambda)$ is observed, and $H_0:\lambda=1$ is tested against $H_1:\lambda\gt1$. Find the non-randomised test of the form $X\ge c$ with the largest size not exceeding $0.05$, and the randomised test of size exactly $0.05$.
- 1 $X_1,\dots,X_n$ iid $N(0,\sigma^2)$. Find the MP test of $H_0:\sigma^2=1$ against $H_1:\sigma^2=4$ at level $\alpha$. For $n=10$, $\alpha=0.05$, find the critical value and the power.
- 2 $X_1,\dots,X_{15}$ iid $\operatorname{Ber}(p)$. Find the MP test of size exactly $0.05$ for $H_0:p=0.2$ against $H_1:p=0.4$, and its power.
- 3 A single $X$ has density $f_0(x)=1$ on $(0,1)$ under $H_0$ and $f_1(x)=2x$ on $(0,1)$ under $H_1$. Find the MP test of size $\alpha$ and its power. Evaluate it at $\alpha=0.05$.
- 4 A single $X$ is $N(0,1)$ under $H_0$ and standard Cauchy under $H_1$. Show that the MP test of size $0.05$ rejects when $|X|\gt1.96$, and find its power.
- 5 Let $\phi$ be an NP test of the form $(\ast)$ with $k\gt0$, of size $\alpha$ and power $\beta$ for $f_0$ against $f_1$. Show that $1-\phi$ is MP at level $1-\beta$ for testing $f_1$ against $f_0$.
- 6 $X_1,\dots,X_n$ iid with density $e^{-(x-\theta)}$, $x\gt\theta$. For $H_0:\theta=0$ against $H_1:\theta=1$, show that every MP test of size $\alpha\ge e^{-n}$ has power $1$, and that the MP test is not unique. What happens if $\alpha\lt e^{-n}$?
- 7 $X_1,\dots,X_5$ iid $\operatorname{Poi}(\lambda)$. Find the MP test of size exactly $0.05$ for $H_0:\lambda=2$ against $H_1:\lambda=1$, and its power.
- 8 $X_1,\dots,X_n$ iid $N(\mu,1)$. Show that the MP test of $\mu=0$ against $\mu=\mu_1\gt0$ has power $\Phi(\sqrt n\mu_1-z_\alpha)$. Find the smallest $n$ for which both error probabilities are at most $0.05$ when $\mu_1=0.5$.
- 1 $X_1,\dots,X_n$ iid with density $\theta x^{\theta-1}$ on $(0,1)$, $\theta\gt0$. Find the UMP level-$\alpha$ test of $H_0:\theta\le1$ against $H_1:\theta\gt1$ in terms of a $\chi^2$ quantile, and write it out for $n=5$, $\alpha=0.05$.
- 2 Show that iid geometric observations with pmf $p(1-p)^x$, $x=0,1,2,\dots$, have MLR in $-\sum X_i$. Describe the UMP test of $H_0:p\le p_0$ against $H_1:p\gt p_0$.
- 3 Prove that the Cauchy location family (one observation) does not have MLR in $X$.
- 4 $X_1,\dots,X_4$ iid $U(0,\theta)$. Find the UMP level-$0.1$ test of $H_0:\theta\le2$ against $H_1:\theta\gt2$ and its power at $\theta=2.5$.
- 5 Eight lifetimes are iid exponential with mean $\theta$. Find the UMP level-$0.05$ test of $H_0:\theta\le100$ against $H_1:\theta\gt100$ and its power at $\theta=200$.
- 6 For $N(\mu,\sigma^2)$, $\sigma$ known, compute the power of the right-sided UMP $5\%$ test at $\mu=\mu_0-\sigma/\sqrt n$. Explain why it is useless for $H_1:\mu\ne\mu_0$.
- 7 Let $X_1,\dots,X_n$ be iid $N(\mu,1)$ and $\phi$ an unbiased level-$\alpha$ test of $H_0:\mu=\mu_0$ against $H_1:\mu\ne\mu_0$. Show that $\beta_\phi(\mu_0)=\alpha$, that $\beta_\phi'(\mu_0)=0$, and that the latter is equivalent to $E_{\mu_0}[(\bar X-\mu_0)\phi(X)]=0$.
- 8 $X_1,\dots,X_n$ iid with density $e^{-(x-\theta)}$, $x\gt\theta$. Show that the test rejecting when $X_{(1)}\le\theta_0$ or $X_{(1)}\gt\theta_0-\frac1n\log\alpha$ is UMP at level $\alpha$ for $H_0:\theta=\theta_0$ against $H_1:\theta\ne\theta_0$.
- 1 $X_1,\dots,X_n$ iid $\operatorname{Exp}(\lambda)$ (rate $\lambda$). Show that the LRT statistic for $H_0:\lambda=\lambda_0$ against $\lambda\ne\lambda_0$ is $\Lambda=(\lambda_0\bar x)^ne^{n(1-\lambda_0\bar x)}$, and describe the critical region in terms of $2\lambda_0\sum X_i\sim\chi^2_{2n}$.
- 2 Ten measurements give $\bar x=12.6$ and $s=1.5$. Test $H_0:\mu=12$ against $\mu\ne12$ at $5\%$ under normality, and give the $p$-value.
- 3 Two independent normal samples with a common variance: $m=8$, $\bar x=24.1$, $s_1=3.2$; $n=10$, $\bar y=20.9$, $s_2=2.8$. Test $\mu_1=\mu_2$ at $5\%$.
- 4 For the data of part 3, test $\sigma_1^2=\sigma_2^2$ at $5\%$ with the $F$-test. Is the pooled test of part 3 justified?
- 5 $X_1\sim\operatorname{Bin}(100,p_1)$ and $X_2\sim\operatorname{Bin}(100,p_2)$ are independent, with $x_1=30$ and $x_2=45$. Compute $-2\log\Lambda$ for $H_0:p_1=p_2$ and test at $5\%$ using Wilks' theorem.
- 6 $X_1,\dots,X_n$ iid $U(0,\theta)$ and $H_0:\theta=\theta_0$. Show that $-2\log\Lambda=-2n\log(X_{(n)}/\theta_0)$ when $X_{(n)}\le\theta_0$, and that under $H_0$ it has exactly the $\chi^2_2$ law for every $n$. Why does Wilks' theorem not apply?
- 7 $20$ iid Poisson counts have $\bar x=2.6$. Test $H_0:\lambda=2$ with the LRT and Wilks' approximation, and compare with the score test $Z=(\bar x-\lambda_0)/\sqrt{\lambda_0/n}$.
- 8 In Theorem 5.9(2) show that the equal-tailed critical values do not satisfy the LRT condition $\frac{c_1}n-\log c_1=\frac{c_2}n-\log c_2$, using $n=25$, $\alpha=0.05$.
E1. CSIR Part B A single observation $X$ has density $\theta x^{\theta-1}$, $0\lt x\lt1$. The test of $H_0:\theta=1$ against $H_1:\theta=2$ rejects $H_0$ when $X\gt0.8$. Its size and power are (A) $0.2$ and $0.36$; (B) $0.2$ and $0.64$; (C) $0.8$ and $0.36$; (D) $0.36$ and $0.2$.
E2. CSIR Part B GATE ST Let $X_1,\dots,X_9$ be iid $N(\mu,1)$. The UMP level-$0.05$ test of $H_0:\mu\le0$ against $H_1:\mu\gt0$ has power at $\mu=1$ approximately (A) $0.050$; (B) $0.851$; (C) $0.912$; (D) $0.999$.
E3. CSIR Part B For a random sample, which of the following families does not have a monotone likelihood ratio in the indicated statistic? (A) $\operatorname{Poi}(\lambda)$, in $\sum X_i$; (B) $U(0,\theta)$, in $X_{(n)}$; (C) the Cauchy location family with $n=1$, in $X_1$; (D) $N(0,\sigma^2)$, in $\sum X_i^2$.
E4. CSIR Part C Let $X_1,\dots,X_n$ be iid $N(\mu,\sigma^2)$ with $\sigma$ known and $0\lt\alpha\lt\frac12$. Which of the following are true? (One or more options may be correct.) (A) There is a UMP level-$\alpha$ test of $H_0:\mu\le0$ against $H_1:\mu\gt0$. (B) There is a UMP level-$\alpha$ test of $H_0:\mu=0$ against $H_1:\mu\ne0$. (C) The equal-tailed two-sided $z$-test is unbiased. (D) The power function of the test in (A) is a convex function of $\mu$ on $\mathbb{R}$.
E5. CSIR Part C Let $X_1,\dots,X_n$ be iid $U(0,\theta)$, $0\lt\alpha\lt1$, and consider $H_0:\theta=1$. Which of the following are true? (A) The test that rejects when $X_{(n)}\gt1$ or $X_{(n)}\le\alpha^{1/n}$ has size $\alpha$. (B) This test is UMP at level $\alpha$ for $H_0:\theta=1$ against $H_1:\theta\ne1$. (C) Under $H_0$, the likelihood ratio statistic $-2\log\Lambda$ for $\theta=1$ against $\theta\ne1$ converges in distribution to $\chi^2_1$. (D) The power of the test in (A) at $\theta=2$ is $1-(1-\alpha)2^{-n}$.
E6. CSIR Part C GATE ST Let $X_1,\dots,X_n$ ($n\ge3$) be iid $N(\mu,\sigma^2)$ with both parameters unknown, let $T=\sqrt n\bar X/S$, and let $\Lambda$ be the likelihood ratio statistic for $H_0:\mu=0$ against $H_1:\mu\ne0$. Which of the following are true? (A) $\Lambda=\left(1+T^2/(n-1)\right)^{-n/2}$. (B) Under $H_0$, $-2\log\Lambda\xrightarrow d\chi^2_1$ as $n\to\infty$. (C) Under $H_0$, $-2\log\Lambda$ has exactly the $\chi^2_1$ distribution for every $n$. (D) The likelihood ratio test of $H_0:\sigma^2=1$ against $\sigma^2\ne1$ ($\mu$ unknown) is the equal-tailed $\chi^2$-test.
Suggested plan — 3 lectures × 60 min.
- The language of testing. Definitions 5.1–5.3 with Example 1 on the board (build the binomial table and show why randomisation is needed). Project Interactive 1 with $n=9$, $\mu_1=0.6$: move the cut-off and let the class call out how $\alpha$ and $\beta$ change; then raise $n$. Prove Theorem 5.1 and state Theorem 5.2, linking back to the $t$-interval of §4.3. Example 2 (sample size) to close.
- Neyman–Pearson. Motivate with the "budget" picture (buy the points with the best $f_1/f_0$), then prove Theorem 5.3 in full and Theorem 5.4. Examples 3 and 4; for Example 4 ask the class first which region they would use, and let the computation overturn the guess.
- UMP, unbiased and LR tests. MLR and Karlin–Rubin (Theorems 5.5, 5.6) with Example 5. Project Interactive 2 in the normal mode to motivate Theorem 5.7, then switch to the uniform mode and do Example 6. Theorem 5.8 briefly; Theorem 5.9(1) and Wilks for $p=1$; Example 7. Finish with E5 and E6 live.
Discussion questions. (1) Why do we fix the type I error and minimise the type II error, rather than minimising their sum? When would minimising $\alpha+\beta$ be more sensible, and which test does that produce? (2) A study reports $p=0.04$. List three different statements that people make about it and decide which are correct. (3) Interactive 2 shows a two-sided UMP test for $U(0,\theta)$. What feature of the likelihood ratio makes this possible, and why does the normal family lack it?
Misconceptions to watch for. Treating "not rejected" as "proved"; reading the $p$-value as the probability that $H_0$ is true; computing the size of a composite null at an arbitrary point instead of the supremum; assuming the MP region is always a tail; believing that Wilks' $\chi^2$ holds for every model; using the $\chi^2$ quantiles with the wrong tail (upper versus lower points).
Suggested homework. I(1, 3, 5, 8), II(1, 2, 4, 6), III(1, 4, 5, 8), IV(2, 3, 5, 6); Exam corner E2, E4, E5.
Research question. If a field of research runs thousands of tests at level $5\%$, what fraction of its "significant" findings are false, and how does that fraction depend on the power of the studies?
Why it matters. Drug-discovery screens, genome-wide association studies and large programmes in psychology and economics test very many hypotheses, most of them null. Replication projects in several countries have found that a substantial share of published significant results do not hold up. Low power is one of the main reasons, and the Neyman–Pearson quantities of this section explain how.
What to do.
- Suppose a proportion $\pi$ of the hypotheses tested are false nulls (real effects), each test has level $\alpha$, and power $1-\beta$ against the real effects. Show that the expected fraction of rejections that are false is $$\mathrm{FDP}=\frac{\alpha(1-\pi)}{\alpha(1-\pi)+(1-\beta)\pi}.$$
- Simulate $10\,000$ one-sample $z$-tests (seeded generator): a fraction $\pi=0.1$ have true effect $\delta$, the rest have $\delta=0$. Choose $n$ to give power $0.8$, and then $0.2$. Compare the simulated false-discovery proportion with the formula.
- Among the significant results, compute the average estimated effect size and compare it with the true $\delta$. Explain the exaggeration you see when the power is low (the "winner's curse").
- Apply the Benjamini–Hochberg procedure (reject the $k$ smallest $p$-values, where $k$ is the largest index with $p_{(k)}\le k\,q/m$) at $q=0.1$, and report its realised false-discovery proportion.
Tools: Python with numpy (or a spreadsheet for step 1); the $p$-values come from $1-\Phi(z)$.
Hand in: A one-page note with the table of FDP (formula and simulation) for both power levels, the winner's-curse comparison, and the result of the Benjamini–Hochberg step.
Going further. Read about the Bayesian interpretation of the false-discovery rate, and about why the Benjamini–Hochberg procedure controls it when the $p$-values are independent (it uses the super-uniformity of Theorem 5.1).
🧪 Self-check quiz
Q1. The size of a test $\phi$ of a composite null hypothesis $H_0:\theta\in\Theta_0$ is
Q2. For simple $H_0:f_0$ against simple $H_1:f_1$, the most powerful test of size $\alpha$ rejects when
Q3. For $X\sim\operatorname{Bin}(20,\tfrac12)$, $P(X\ge15)=0.0207$ and $P(X=14)=0.0370$. The size-$0.05$ test that rejects for $X\ge15$ and randomises at $X=14$ rejects there with probability about
Q4. For iid $N(\mu,1)$ data, there is no UMP test of $\mu=0$ against $\mu\ne0$ because
Q5. Three normal populations have unknown means and a common unknown variance. For $H_0:\mu_1=\mu_2=\mu_3$, Wilks' theorem gives $-2\log\Lambda\approx\chi^2_r$ with $r$ equal to