Chapter 4 · Sampling Distributions and Estimation

4.2 Methods of Estimation and Properties of Estimators

How to turn a sample into an estimate, and how to tell a good estimate from a bad one: moments and maximum likelihood; bias, MSE and consistency; sufficiency and completeness; the Rao–Blackwell and Lehmann–Scheffé route to the best unbiased estimator; and the Cramér–Rao bound that no unbiased estimator can beat.

A model with an unknown parameter, $X_1,\dots,X_n$ iid with density or pmf $f(x;\theta)$, $\theta\in\Theta$, is the starting point of nearly every statistical analysis. The infection rate in a Poisson model of case counts, the mean lifetime in an exponential model of component failures, the proportion of defectives in a binomial model of inspections: in each case the data must be turned into a number $\hat\theta$, an estimate. There are infinitely many ways to do this. For a uniform sample on $(0,\theta)$, $2\bar X$, $X_{(n)}$, $\frac{n+1}{n}X_{(n)}$ and $(n+1)X_{(1)}$ all have some claim to estimate $\theta$. Two questions organise the subject: how do we find reasonable estimators, and how do we compare them?

This section answers both. The method of moments and the method of maximum likelihood produce estimators almost automatically. Bias, mean squared error and consistency are the first yardsticks. Sufficiency identifies which functions of the data carry all the information about $\theta$, and completeness makes the best unbiased estimator unique. The Rao–Blackwell and Lehmann–Scheffé theorems then construct the uniformly minimum variance unbiased estimator (UMVUE). The Fisher information and the Cramér–Rao inequality give a floor below which no unbiased estimator's variance can go, and the MLE reaches that floor asymptotically. CSIR Part B questions regularly ask for an MLE, a UMVUE or a Cramér–Rao bound. Part C questions test the exceptions: the uniform model where the bound "fails", curved families that are not complete, MLEs that are not unique or not unbiased, and unbiased estimators that do not exist.

Estimators: Bias, Mean Squared Error and Consistency

📘 Definition 4.4 — Estimator, bias, MSE and consistency

Let $X_1,\dots,X_n$ be iid from $f(x;\theta)$, $\theta\in\Theta$, and let $\psi(\theta)$ be a real function of the parameter. An estimator of $\psi(\theta)$ is any statistic $T=T_n(X_1,\dots,X_n)$; its value on the data is an estimate. Writing $E_\theta$ for expectations under $\theta$:

  • the bias is $b_T(\theta)=E_\theta[T]-\psi(\theta)$, and $T$ is unbiased if $b_T(\theta)=0$ for every $\theta\in\Theta$; $T_n$ is asymptotically unbiased if $b_{T_n}(\theta)\to0$ for every $\theta$;
  • the mean squared error is $\operatorname{MSE}_\theta(T)=E_\theta\left[(T-\psi(\theta))^2\right]$;
  • $T_n$ is (weakly) consistent if $T_n\xrightarrow{P}\psi(\theta)$ under every $\theta$, strongly consistent if $T_n\to\psi(\theta)$ almost surely, and MSE-consistent if $\operatorname{MSE}_\theta(T_n)\to0$.
📐 Theorem 4.13 — Bias–variance decomposition and consistency criteria
  1. $\operatorname{MSE}_\theta(T)=\operatorname{Var}_\theta(T)+b_T(\theta)^2$.
  2. If $b_{T_n}(\theta)\to0$ and $\operatorname{Var}_\theta(T_n)\to0$ for every $\theta$, then $T_n$ is MSE-consistent and hence consistent.
  3. If $T_n$ is consistent for $\theta$ and $g$ is continuous at $\theta$, then $g(T_n)$ is consistent for $g(\theta)$.
  4. If $E|X_1|^r\lt\infty$, the sample moment $m_r'=\frac1n\sum X_i^r$ is strongly consistent for $\mu_r'$. Hence every estimator of the form $h(m_1',\dots,m_k')$ with $h$ continuous at $(\mu_1',\dots,\mu_k')$ is consistent.

Proof. (1) Expand $(T-\psi)^2=\left((T-E T)+(ET-\psi)\right)^2$; the cross term has mean $0$. (2) By (1), $\operatorname{MSE}\to0$, and Chebyshev–Markov gives $P(|T_n-\psi|\gt\varepsilon)\le\operatorname{MSE}/\varepsilon^2\to0$. (3) is the continuous mapping theorem (§2.3). (4) is Kolmogorov's strong law (§2.3) applied to the iid variables $X_i^r$, followed by (3) in $k$ variables. ∎

Unbiasedness is a property of the average of $T$ over repeated samples. Consistency is a property of the whole sequence $T_n$ as $n\to\infty$. Neither implies the other (Example 2). Mean squared error is the most common single measure of accuracy: it rewards small variance and penalises bias, and the decomposition (1) shows the trade-off between them.

✏️ Example 1 — Trading bias for variance
(a) For a normal sample, let $SS=\sum(X_i-\bar X)^2$ and consider the estimators $cSS$ of $\sigma^2$. Find the $c$ that minimises the MSE, and compare the divisors $n-1$, $n$ and $n+1$ when $n=10$. (b) For a $U(0,\theta)$ sample, compare $2\bar X$, $X_{(n)}$, $\frac{n+1}{n}X_{(n)}$ and $\frac{n+2}{n+1}X_{(n)}$ by MSE.
(a) Set up By the fundamental theorem for normal samples (§4.1), $SS/\sigma^2\sim\chi^2_{n-1}$, so $E[SS]=(n-1)\sigma^2$ and $\operatorname{Var}(SS)=2(n-1)\sigma^4$. Hence $$\operatorname{MSE}(cSS)=\sigma^4\left[2c^2(n-1)+\left(c(n-1)-1\right)^2\right].$$
(a) Minimise Setting the derivative in $c$ to zero gives $4c(n-1)+2(n-1)(c(n-1)-1)=0$, i.e. $c(n+1)=1$. The optimum is $c=\frac{1}{n+1}$, with $\operatorname{MSE}=\frac{2\sigma^4}{n+1}$. For $n=10$: divisor $9$ (unbiased $S^2$) gives $\frac29\sigma^4=0.222\sigma^4$; divisor $10$ (the MLE) gives $\frac{19}{100}\sigma^4=0.190\sigma^4$; divisor $11$ gives $\frac{2}{11}\sigma^4=0.182\sigma^4$. The unbiased estimator has the largest MSE of the three.
(b) With $X_{(n)}=\theta U_{(n)}$ and $U_{(n)}\sim\operatorname{Beta}(n,1)$: $E[X_{(n)}]=\frac{n\theta}{n+1}$ and $E[X_{(n)}^2]=\frac{n\theta^2}{n+2}$. Then $\operatorname{MSE}(cX_{(n)})=\theta^2\left[\frac{c^2n}{n+2}-\frac{2cn}{n+1}+1\right]$, which is minimised at $c=\frac{n+2}{n+1}$ with value $\frac{\theta^2}{(n+1)^2}$.
(b) Table
estimatorbiasMSE$/\theta^2$$n=5$
$2\bar X$$0$$\frac{1}{3n}$$0.0667$
$X_{(n)}$ (MLE)$-\frac{\theta}{n+1}$$\frac{2}{(n+1)(n+2)}$$0.0476$
$\frac{n+1}{n}X_{(n)}$$0$$\frac{1}{n(n+2)}$$0.0286$
$\frac{n+2}{n+1}X_{(n)}$$-\frac{\theta}{(n+1)^2}$$\frac{1}{(n+1)^2}$$0.0278$
The estimators built on $X_{(n)}$ have MSE of order $1/n^2$, while the moment estimator's is of order $1/n$. (All four rows were checked by simulation.)
✏️ Example 2 — Prove or disprove: four facts about unbiasedness
(a) Every parametric function has an unbiased estimator. (b) An unbiased estimator always takes values in the range of $\psi$. (c) Unbiased estimators are consistent, and consistent estimators are asymptotically unbiased. (d) $S$ is unbiased for $\sigma$ because $S^2$ is unbiased for $\sigma^2$.
(a) False Let $X\sim\operatorname{Bin}(n,p)$ and $\psi(p)=1/p$. For any $T$, $E_p[T(X)]=\sum_{k=0}^nT(k)\binom nkp^k(1-p)^{n-k}$ is a polynomial in $p$, hence bounded on $(0,1)$, while $1/p\to\infty$ as $p\to0$. No estimator is unbiased for $1/p$.
(b) False Let $X\sim\operatorname{Poi}(\lambda)$ be a single observation and $\psi(\lambda)=e^{-2\lambda}\in(0,1)$. $E[T(X)]=e^{-2\lambda}$ means $\sum_kT(k)\frac{\lambda^k}{k!}=e^{-\lambda}=\sum_k\frac{(-\lambda)^k}{k!}$ for all $\lambda\gt0$. By uniqueness of power series, $T(k)=(-1)^k$. The only unbiased estimator of a number in $(0,1)$ takes the values $\pm1$, which is absurd.
(c) False, twice $T_n=X_1$ is unbiased for $\mu$ for every $n$ but is not consistent: its law never changes. Conversely, let $T_n=\bar X$ with probability $1-\frac1n$ and $T_n=n^2$ with probability $\frac1n$ (an independent coin). Then $P(T_n\ne\bar X)\to0$, so $T_n\xrightarrow{P}\mu$, but $E[T_n]=\left(1-\tfrac1n\right)\mu+n\to\infty$.
(d) False By Jensen's inequality (strict, since $S$ is not constant), $E[S]=E\left[\sqrt{S^2}\right]\lt\sqrt{E[S^2]}=\sigma$. For normal samples, $E[S]=\sigma\sqrt{\frac{2}{n-1}}\,\frac{\Gamma(n/2)}{\Gamma((n-1)/2)}$, e.g. $0.798\sigma$ for $n=2$ and $0.973\sigma$ for $n=10$. Unbiasedness is not preserved by nonlinear maps. Consistency is (Theorem 4.13(3)).

The Method of Moments and Maximum Likelihood

📘 Definition 4.5 — Method of moments
Suppose $\theta=(\theta_1,\dots,\theta_k)$ and the population moments $\mu_r'(\theta)=E_\theta[X^r]$, $r=1,\dots,k$, exist. The method-of-moments estimator (MME) $\tilde\theta$ solves the $k$ equations $\mu_r'(\tilde\theta)=m_r'$, $r=1,\dots,k$, which equate population moments to sample moments. Equivalently one may use the mean and central moments ($\mu=\bar X$, $\sigma^2=\hat\sigma^2$, …).
📘 Definition 4.6 — Likelihood and maximum likelihood estimator
For observed data $x=(x_1,\dots,x_n)$, the likelihood function is $L(\theta)=L(\theta;x)=\prod_{i=1}^nf(x_i;\theta)$, viewed as a function of $\theta$ with the data fixed, and $\ell(\theta)=\log L(\theta)$ is the log-likelihood. A maximum likelihood estimate is a value $\hat\theta\in\Theta$ (more precisely, in the closure of $\Theta$ when the supremum is approached at the boundary) with $L(\hat\theta)=\sup_{\theta\in\Theta}L(\theta)$. The corresponding statistic is the MLE. When $\ell$ is differentiable and the maximum is interior, $\hat\theta$ solves the likelihood equation $\ell'(\theta)=0$.

The likelihood is not a density in $\theta$, and it does not integrate to $1$. It ranks parameter values by how probable they make the data that were actually observed. The MLE need not exist, need not be unique, need not solve the likelihood equation, and need not be unbiased (Example 4). One property, however, makes it very convenient.

📐 Theorem 4.14 — Invariance of the MLE
Let $\hat\theta$ be an MLE of $\theta$ and let $\tau:\Theta\to\Lambda$ be any function, not necessarily one-to-one. Define the induced likelihood $L^*(\lambda)=\sup\{L(\theta):\theta\in\Theta,\ \tau(\theta)=\lambda\}$. Then $\tau(\hat\theta)$ maximises $L^*$. That is, the MLE of $\tau(\theta)$ is $\tau(\hat\theta)$.

Proof. For every $\lambda\in\Lambda$, $L^*(\lambda)\le\sup_\theta L(\theta)=L(\hat\theta)$. Since $\hat\theta$ itself belongs to $\{\theta:\tau(\theta)=\tau(\hat\theta)\}$, also $L(\hat\theta)\le L^*(\tau(\hat\theta))$. Hence $L^*(\lambda)\le L^*(\tau(\hat\theta))$ for all $\lambda$. ∎

For example, the MLE of $\sigma$ in a normal model is $\sqrt{\hat\sigma^2}$, and the MLE of $P(X\gt1)=e^{-\lambda}$ in an exponential model is $e^{-1/\bar X}$. Unbiasedness has no such property.

🧭 Problem-solving strategy — finding an MLE
  1. Write $L(\theta)$ with all indicator functions. If the support depends on $\theta$ (uniform, shifted exponential, Pareto), the indicators restrict $\theta$ to an interval determined by $X_{(1)}$ or $X_{(n)}$.
  2. Non-regular case: on that interval, decide whether $L$ is monotone. If it is, the MLE is the appropriate endpoint, an order statistic. Do not differentiate across the indicator.
  3. Regular case: solve $\ell'(\theta)=0$. Check that it is a maximum ($\ell''\lt0$, or $\ell\to-\infty$ at the ends of $\Theta$) and compare with the boundary values.
  4. Several parameters: solve the score equations jointly, often by profiling (maximise over one parameter for each fixed value of the other).
  5. Functions of $\theta$: use invariance. Check uniqueness: a flat likelihood (as for $U(\theta,\theta+1)$) gives many MLEs.
✏️ Example 3 — Moments and likelihood in regular models
(a) Find the MLE of $(\mu,\sigma^2)$ for a normal sample. (b) Find the MME of $(\alpha,\lambda)$ for a $\operatorname{Gamma}(\alpha,\lambda)$ sample, and evaluate it when $\bar x=4$ and $\hat\sigma^2=8$. (c) For the Laplace density $\frac12e^{-|x-\theta|}$, find the MLE of $\theta$.
(a) $\ell(\mu,\sigma^2)=-\frac n2\log(2\pi\sigma^2)-\frac{1}{2\sigma^2}\sum(x_i-\mu)^2$. For each fixed $\sigma^2$, the sum is minimised at $\mu=\bar x$ (the identity of §4.1). The profile $-\frac n2\log\sigma^2-\frac{SS}{2\sigma^2}$ has derivative $-\frac{n}{2\sigma^2}+\frac{SS}{2\sigma^4}$, which vanishes only at $\sigma^2=SS/n$, where the profile is maximal (it tends to $-\infty$ at both ends). So $\hat\mu=\bar X$ and $\hat\sigma^2=\frac1n\sum(X_i-\bar X)^2$, which is biased (Example 1). By invariance, $\hat\sigma=\sqrt{\hat\sigma^2}$.
(b) $E[X]=\frac\alpha\lambda$ and $\operatorname{Var}(X)=\frac{\alpha}{\lambda^2}$. Equating to $\bar X$ and $\hat\sigma^2$ gives $\tilde\lambda=\bar X/\hat\sigma^2$ and $\tilde\alpha=\bar X^2/\hat\sigma^2$. With the data, $\tilde\lambda=0.5$ and $\tilde\alpha=2$. (The MLE of $\alpha$ solves $\log\alpha-\frac{\Gamma'(\alpha)}{\Gamma(\alpha)}=\log\bar x-\overline{\log x}$, which has no closed form. Here the MME is a convenient starting value.)
(c) $\ell(\theta)=-n\log2-\sum|x_i-\theta|$, so the MLE minimises $\sum|x_i-\theta|$. This function is convex and piecewise linear, with slope $\#\{x_i\lt\theta\}-\#\{x_i\gt\theta\}$ between data points. The slope changes sign at the median. So $\hat\theta$ is the sample median; for even $n$ every point between the two middle order statistics is an MLE. The likelihood is not differentiable at the data points, so the likelihood equation is useless here.
✏️ Example 4 — Non-regular models: the MLE at the boundary
(a) For a $U(0,\theta)$ sample, find the MLE and the MME of $\theta$, and evaluate both on the data $0.2,\ 0.5,\ 0.3,\ 3.0$. (b) For $f(x;\theta)=e^{-(x-\theta)}$, $x\gt\theta$, find the MLE of $\theta$ and its bias. (c) For a $U(\theta,\theta+1)$ sample, show that the MLE is not unique.
(a) $L(\theta)=\prod\frac1\theta\mathbf 1\{0\lt x_i\le\theta\}=\theta^{-n}\mathbf 1\{\theta\ge x_{(n)}\}$. This is $0$ for $\theta\lt x_{(n)}$ and strictly decreasing for $\theta\ge x_{(n)}$, so $\hat\theta=X_{(n)}$. Note that $\ell'(\theta)=-n/\theta$ never vanishes. The MME solves $\theta/2=\bar X$: $\tilde\theta=2\bar X$. On the data, $\bar x=1.0$, so $\tilde\theta=2.0$, which is smaller than the observation $3.0$: an impossible value. The MLE is $3.0$, and the unbiased version $\frac{n+1}{n}X_{(n)}=3.75$.
(b) $L(\theta)=e^{n\theta-\sum x_i}\mathbf 1\{\theta\lt x_{(1)}\}$ increases in $\theta$ up to $x_{(1)}$. So $\hat\theta=X_{(1)}$, the supremum attained at the closure of the allowed set. $X_{(1)}-\theta\sim\operatorname{Exp}(n)$ (§4.1, Exercise III.2), so $E[\hat\theta]=\theta+\frac1n$: biased, but consistent with error of order $1/n$.
(c) $L(\theta)=\mathbf 1\{\theta\le x_{(1)}\}\mathbf 1\{x_{(n)}\le\theta+1\}=\mathbf 1\{x_{(n)}-1\le\theta\le x_{(1)}\}$. Every $\theta$ in the interval $[X_{(n)}-1,\,X_{(1)}]$ maximises $L$. For instance the midpoint $\frac12(X_{(1)}+X_{(n)}-1)$ is a natural choice, but so is $X_{(1)}$. The MLE is a set, not a number.
Interactive 1 — The Likelihood Sharpens Around the Truth

The blue curve is the relative likelihood $L(\theta)/L(\hat\theta)$ for the first $n$ observations of a seeded data stream. The red line marks the MLE $\hat\theta$, the green dashed line the true $\theta_0$, and the ticks along the axis show the data (for the uniform and Cauchy models). The dashed purple bell is the normal approximation $\exp\left(-\tfrac12 nI(\hat\theta)(\theta-\hat\theta)^2\right)$ of Theorem 4.23. Press Play to add observations. In regular models the likelihood narrows like $1/\sqrt n$ and becomes bell-shaped. For $U(0,\theta)$ it is a cliff at $X_{(n)}$ that moves at rate $1/n$. For Cauchy data and small $n$ it can have several peaks.

10
n—
MLE θ̂—
true θ₀—
SE ≈ 1/√(n I(θ̂))—
local maxima of L—

Sufficiency, Minimal Sufficiency and Completeness

A statistic $T$ is sufficient if, once $T$ is known, the rest of the data carry no further information about $\theta$. In other words, the conditional law of the sample given $T$ does not involve $\theta$, so a statistician who knew only $T$ could regenerate a sample with exactly the same law as the original without knowing $\theta$.

📘 Definition 4.7 — Sufficient statistic
A statistic $T=T(X_1,\dots,X_n)$ is sufficient for $\theta$ (or for the family $\{f(\cdot;\theta):\theta\in\Theta\}$) if the conditional distribution of $(X_1,\dots,X_n)$ given $T=t$ does not depend on $\theta$, for every $t$.
📐 Theorem 4.15 — The Neyman–Fisher factorisation theorem
$T$ is sufficient for $\theta$ if and only if the joint pmf or density factorises as $$f(x_1,\dots,x_n;\theta)=g\left(T(x);\theta\right)\,h(x)$$ for some nonnegative functions $g$ and $h$, where $h$ does not depend on $\theta$. The same holds for a vector statistic $T=(T_1,\dots,T_k)$.

Proof (discrete case). Write $f(x;\theta)=P_\theta(X=x)$. ($\Leftarrow$) Suppose the factorisation holds and fix $t$ with $P_\theta(T=t)\gt0$. For $x$ with $T(x)=t$, $$\begin{aligned}P_\theta(X=x\mid T=t)&=\frac{f(x;\theta)}{\sum_{y:T(y)=t}f(y;\theta)}\\&=\frac{g(t;\theta)h(x)}{g(t;\theta)\sum_{y:T(y)=t}h(y)}\\&=\frac{h(x)}{\sum_{y:T(y)=t}h(y)},\end{aligned}$$ which is free of $\theta$. For $T(x)\ne t$ the conditional probability is $0$. ($\Rightarrow$) If $T$ is sufficient, put $g(t;\theta)=P_\theta(T=t)$ and $h(x)=P(X=x\mid T=T(x))$, which is free of $\theta$ by assumption. Then $$f(x;\theta)=P_\theta(T=T(x))\,P(X=x\mid T=T(x))=g(T(x);\theta)h(x).$$ The continuous case needs measure-theoretic conditioning; the statement is the same, and we use it freely. ∎

Examples follow at once. For $\operatorname{Ber}(p)$, $\prod p^{x_i}(1-p)^{1-x_i}=p^{t}(1-p)^{n-t}$ with $t=\sum x_i$, so $\sum X_i$ is sufficient. For $N(\mu,\sigma^2)$, the density is $(2\pi\sigma^2)^{-n/2}\exp\left(-\frac{\sum x_i^2-2\mu\sum x_i+n\mu^2}{2\sigma^2}\right)$, so $(\sum X_i,\sum X_i^2)$, or equivalently $(\bar X,S^2)$, is sufficient. For $U(0,\theta)$, $\theta^{-n}\mathbf 1\{x_{(n)}\le\theta\}\mathbf 1\{x_{(1)}\gt0\}$ shows that $X_{(n)}$ is sufficient. The whole sample, and the vector of order statistics, are always sufficient; the question is how far the data can be reduced.

📘 Definition 4.8 — Minimal sufficient statistic
A sufficient statistic $T$ is minimal sufficient if it is a function of every other sufficient statistic: whenever $U$ is sufficient, $T=\varphi(U)$ for some function $\varphi$. A minimal sufficient statistic gives the greatest reduction of the data that loses no information about $\theta$.
📐 Theorem 4.16 — The ratio criterion for minimal sufficiency
Suppose that for all sample points $x,y$ in the support, $$\frac{f(x;\theta)}{f(y;\theta)}\ \text{ does not depend on }\theta\quad\Longleftrightarrow\quad T(x)=T(y).$$ Then $T$ is minimal sufficient.

Proof (sketch). Sufficiency. For each value $t$ of $T$ choose one point $x_t$ with $T(x_t)=t$. By the criterion, $h(x)=f(x;\theta)/f(x_{T(x)};\theta)$ does not depend on $\theta$, so $f(x;\theta)=f(x_{T(x)};\theta)\,h(x)=g(T(x);\theta)h(x)$, and the factorisation theorem applies. Minimality. Let $U$ be sufficient, with $f=\tilde g(U(x);\theta)\tilde h(x)$. If $U(x)=U(y)$, then $f(x;\theta)/f(y;\theta)=\tilde h(x)/\tilde h(y)$ is free of $\theta$, so $T(x)=T(y)$ by the criterion. Thus $T$ is constant wherever $U$ is constant, so $T$ is a function of $U$. ∎

📘 Definition 4.9 — Complete statistic and ancillary statistic

A statistic $T$ (or its family of laws) is complete if, for every measurable $g$, $$E_\theta[g(T)]=0\ \text{ for all }\theta\in\Theta\quad\Longrightarrow\quad P_\theta\left(g(T)=0\right)=1\ \text{ for all }\theta\in\Theta.$$ It is boundedly complete if this holds for every bounded $g$. A statistic $V$ is ancillary if its distribution does not depend on $\theta$.

Completeness says that the family of laws of $T$ is rich enough that no nonzero function of $T$ has mean zero for all $\theta$. In particular, there is at most one unbiased estimator of $\psi(\theta)$ that is a function of $T$.

📘 Definition 4.10 — Exponential family
A family of pmfs or densities is a $k$-parameter exponential family if $$f(x;\theta)=h(x)\exp\Big(\sum_{j=1}^k\eta_j(\theta)T_j(x)-A(\theta)\Big),$$ where the support $\{x:h(x)\gt0\}$ does not depend on $\theta$. The $\eta_j(\theta)$ are the natural parameters. The family is of full rank if the set $\{(\eta_1(\theta),\dots,\eta_k(\theta)):\theta\in\Theta\}$ contains a $k$-dimensional open rectangle; otherwise (for example, when $k=2$ but the natural parameters lie on a curve) it is a curved exponential family.
📐 Theorem 4.17 — Sufficiency and completeness in exponential families
For a sample from a $k$-parameter exponential family, the statistic $$\mathbf T=\Big(\sum_{i=1}^nT_1(X_i),\dots,\sum_{i=1}^nT_k(X_i)\Big)$$ is sufficient. If the family has full rank, $\mathbf T$ is complete and minimal sufficient.

Proof (sketch). The joint density is $$\prod_i h(x_i)\cdot\exp\Big(\sum_j\eta_j(\theta)\mathbf T_j(x)-nA(\theta)\Big),$$ so the factorisation theorem gives sufficiency. For completeness, the law of $\mathbf T$ has the form $c(\eta)e^{\eta\cdot t}\,\nu(dt)$ for a measure $\nu$ that does not depend on $\eta$. If $E_\eta[g(\mathbf T)]=0$ for all $\eta$ in an open rectangle, then $\int g^+(t)e^{\eta\cdot t}\nu(dt)=\int g^-(t)e^{\eta\cdot t}\nu(dt)$ there. Two Laplace transforms that agree on an open set are equal as measures (the uniqueness theorem for Laplace transforms, as for mgfs in §1.3). Hence $g^+=g^-$ $\nu$-a.e., i.e. $g(\mathbf T)=0$ a.s. Minimality follows from Theorem 4.16: the ratio $$\frac{f(x;\theta)}{f(y;\theta)}=\frac{\prod h(x_i)}{\prod h(y_i)}\exp\Big(\sum_j\eta_j(\theta)\left(\mathbf T_j(x)-\mathbf T_j(y)\right)\Big)$$ is free of $\theta$ iff $\mathbf T(x)=\mathbf T(y)$, because the full-rank condition rules out a fixed linear relation among the $\eta_j(\theta)$. ∎

The binomial, Poisson, geometric, negative binomial (known $r$), exponential, gamma, beta and normal families are all of full rank in their usual parametrisations. Uniform laws with unknown endpoints are not exponential families, since their support moves, and completeness there must be checked by hand.

📐 Theorem 4.18 — Basu's theorem
If $T$ is a boundedly complete sufficient statistic and $V$ is ancillary, then $T$ and $V$ are independent under every $\theta$.

Proof. Fix an event $\{V\in B\}$. Its probability $p_B=P_\theta(V\in B)$ does not depend on $\theta$ ($V$ is ancillary), and $\varphi(T)=P(V\in B\mid T)$ does not depend on $\theta$ ($T$ is sufficient), so $\varphi$ is a bounded statistic. Now $E_\theta[\varphi(T)-p_B]=0$ for all $\theta$, and bounded completeness gives $\varphi(T)=p_B$ a.s. Thus $P(V\in B\mid T)=P(V\in B)$ for every $B$, which is independence. ∎

Basu's theorem gives a second proof that $\bar X$ and $S^2$ are independent for normal samples. Fix $\sigma^2$. Then $\bar X$ is complete sufficient for $\mu$ (a full-rank one-parameter family), and $S^2$ is ancillary for $\mu$ because it is unchanged by shifts. Since $\sigma^2$ was arbitrary, independence holds for every $(\mu,\sigma^2)$.

✏️ Example 5 — Which statistics are complete?
(a) Show that $T=\sum X_i$ is complete sufficient for a Poisson sample. (b) Show that $X_{(n)}$ is complete sufficient for $U(0,\theta)$. (c) For $N(\theta,\theta^2)$ with $\theta\gt0$, show that $(\sum X_i,\sum X_i^2)$ is minimal sufficient but not complete. (d) For $U(\theta,\theta+1)$, show that $(X_{(1)},X_{(n)})$ is minimal sufficient but not complete.
(a) $T\sim\operatorname{Poi}(n\lambda)$. If $E_\lambda[g(T)]=e^{-n\lambda}\sum_{t\ge0}g(t)\frac{(n\lambda)^t}{t!}=0$ for all $\lambda\gt0$, then the power series $\sum g(t)\frac{n^t}{t!}\lambda^t$ vanishes on $(0,\infty)$, so every coefficient is $0$: $g\equiv0$ on $\{0,1,2,\dots\}$. Sufficiency: $\prod e^{-\lambda}\frac{\lambda^{x_i}}{x_i!}=e^{-n\lambda}\lambda^{t}\cdot\frac{1}{\prod x_i!}$.
(b) $X_{(n)}$ has density $nt^{n-1}/\theta^n$ on $(0,\theta)$. If $\int_0^\theta g(t)\,nt^{n-1}\theta^{-n}dt=0$ for all $\theta\gt0$, then $G(\theta)=\int_0^\theta g(t)t^{n-1}dt=0$ for all $\theta$. By the Lebesgue differentiation theorem, $g(\theta)\theta^{n-1}=G'(\theta)=0$ for almost every $\theta$. So $g(X_{(n)})=0$ a.s. Sufficiency was shown after Theorem 4.15.
(c) The ratio $f(x;\theta)/f(y;\theta)=\exp\left(-\frac{\sum x_i^2-\sum y_i^2}{2\theta^2}+\frac{\sum x_i-\sum y_i}{\theta}\right)$ is free of $\theta$ iff both differences vanish, so $T=(T_1,T_2)=(\sum X_i,\sum X_i^2)$ is minimal sufficient by Theorem 4.16. But $E[T_1^2]=n\theta^2+n^2\theta^2=n(n+1)\theta^2$ and $E[T_2]=2n\theta^2$, so $$g(T)=\frac{T_1^2}{n(n+1)}-\frac{T_2}{2n}$$ has mean $0$ for every $\theta$ without being $0$. The natural parameters $\left(\frac1\theta,-\frac{1}{2\theta^2}\right)$ lie on a parabola, a curved family, so Theorem 4.17 does not apply.
(d) $f(x;\theta)=\mathbf 1\{\theta\lt x_{(1)}\}\mathbf 1\{x_{(n)}\lt\theta+1\}$. For two samples $x,y$, the ratio is free of $\theta$ only if the two indicator sets of $\theta$ coincide, i.e. $(x_{(1)},x_{(n)})=(y_{(1)},y_{(n)})$. So $(X_{(1)},X_{(n)})$ is minimal sufficient. But the range $R=X_{(n)}-X_{(1)}$ is ancillary: it is the range of a $U(0,1)$ sample, $\operatorname{Beta}(n-1,2)$ by §4.1. So $g=R-\frac{n-1}{n+1}$ has mean $0$ for all $\theta$ and is not $0$. A minimal sufficient statistic need not be complete. (Conversely, a complete sufficient statistic is minimal sufficient whenever a minimal sufficient statistic exists; see Exercise III.6.)

Best Unbiased Estimators: Rao–Blackwell and Lehmann–Scheffé

📘 Definition 4.11 — UMVUE
An unbiased estimator $T^*$ of $\psi(\theta)$ is a uniformly minimum variance unbiased estimator (UMVUE) if $\operatorname{Var}_\theta(T^*)\le\operatorname{Var}_\theta(T)$ for every $\theta\in\Theta$ and every unbiased estimator $T$ of $\psi(\theta)$ with finite variance.
📐 Theorem 4.19 — Rao–Blackwell theorem
Let $W$ be an unbiased estimator of $\psi(\theta)$ with finite variance, and let $T$ be sufficient. Then $\varphi(T)=E[W\mid T]$ is a statistic, $E_\theta[\varphi(T)]=\psi(\theta)$, and $$\operatorname{Var}_\theta\left(\varphi(T)\right)\le\operatorname{Var}_\theta(W)\quad\text{for all }\theta,$$ with equality iff $W=\varphi(T)$ almost surely.

Proof. Since $T$ is sufficient, the conditional law of the data given $T$, and hence $E[W\mid T]$, does not involve $\theta$. So $\varphi(T)$ is computable from the data: it is a statistic. By the tower property, $E_\theta[\varphi(T)]=E_\theta[W]=\psi(\theta)$. By the law of total variance (§1.4), $$\operatorname{Var}_\theta(W)=\operatorname{Var}_\theta\left(E[W\mid T]\right)+E_\theta\left[\operatorname{Var}(W\mid T)\right]\ge\operatorname{Var}_\theta(\varphi(T)),$$ with equality iff $\operatorname{Var}(W\mid T)=0$ a.s., i.e. $W$ is a function of $T$. ∎

📐 Theorem 4.20 — Lehmann–Scheffé theorem
Let $T$ be a complete sufficient statistic. If $g(T)$ is an unbiased estimator of $\psi(\theta)$ with finite variance, then $g(T)$ is the unique UMVUE of $\psi(\theta)$ (unique up to almost-sure equality).

Proof. Let $W$ be any unbiased estimator of $\psi(\theta)$ with finite variance, and let $\varphi(T)=E[W\mid T]$. By Rao–Blackwell, $\varphi(T)$ is unbiased and $\operatorname{Var}(\varphi(T))\le\operatorname{Var}(W)$. Both $\varphi(T)$ and $g(T)$ are unbiased functions of $T$, so $E_\theta[\varphi(T)-g(T)]=0$ for all $\theta$, and completeness gives $\varphi(T)=g(T)$ a.s. Hence $\operatorname{Var}_\theta(g(T))\le\operatorname{Var}_\theta(W)$ for every $\theta$. If $W$ is also a UMVUE, equality holds, and the equality case of Rao–Blackwell gives $W=\varphi(T)=g(T)$ a.s. ∎

🧭 Problem-solving strategy — finding a UMVUE
  1. Find a complete sufficient statistic $T$: the natural statistic of a full-rank exponential family, or an order statistic in a uniform or shifted model (checked by hand).
  2. Route A — unbias a function of $T$. Compute $E[T]$, $E[T^2]$, $E[1/T]$, … and combine them to get an unbiased $g(T)$. For example, $E[T(T-1)]$ for Poisson and binomial counts gives $\lambda^2$ and $p^2$.
  3. Route B — Rao–Blackwellise. Start from any simple unbiased estimator, often an indicator such as $\mathbf 1\{X_1=0\}$ or $\mathbf 1\{X_1\le c\}$, and compute $E[\,\cdot\mid T]$ using the conditional law of $X_1$ given $T$.
  4. Either way, Lehmann–Scheffé says the result is the UMVUE. If no complete sufficient statistic exists, a UMVUE may fail to exist.
✏️ Example 6 — Computing UMVUEs
(a) $X_1,\dots,X_n$ iid $\operatorname{Poi}(\lambda)$. Find the UMVUEs of $e^{-\lambda}=P(X=0)$ and of $\lambda e^{-\lambda}=P(X=1)$, and evaluate them for $n=5$, $\sum x_i=8$. (b) $X_1,\dots,X_n$ iid $\operatorname{Ber}(p)$. Find the UMVUE of $p(1-p)$. (c) $X_1,\dots,X_n$ iid $U(0,\theta)$. Find the UMVUE of $\theta$.
(a) Route B $T=\sum X_i$ is complete sufficient (Example 5(a)), and $W=\mathbf 1\{X_1=0\}$ is unbiased for $e^{-\lambda}$. Given $T=t$, $X_1\sim\operatorname{Bin}(t,\frac1n)$ by the conditioning property of Poisson counts (§2.1). So $$E[W\mid T]=\left(1-\frac1n\right)^T.$$ Similarly $E[\mathbf 1\{X_1=1\}\mid T]=\frac Tn\left(1-\frac1n\right)^{T-1}$. With $n=5$, $t=8$: the UMVUEs are $(0.8)^8=0.168$ and $1.6\times0.8^7=0.336$. For comparison, the MLEs by invariance are $e^{-1.6}=0.202$ and $1.6e^{-1.6}=0.323$.
(b) Route A $T=\sum X_i\sim\operatorname{Bin}(n,p)$ is complete sufficient. $E[T(n-T)]=nE[T]-E[T^2]=n^2p-np(1-p)-n^2p^2=n(n-1)p(1-p)$, so $\frac{T(n-T)}{n(n-1)}$ is the UMVUE. Equivalently, it is $\frac{n}{n-1}\hat p(1-\hat p)=S^2$ computed from the $0$/$1$ data.
(c) $X_{(n)}$ is complete sufficient (Example 5(b)) and $E[X_{(n)}]=\frac{n}{n+1}\theta$, so $\frac{n+1}{n}X_{(n)}$ is the UMVUE, with variance $\frac{\theta^2}{n(n+2)}$. The moment estimator $2\bar X$ is also unbiased, and Rao–Blackwellising it gives $E[2\bar X\mid X_{(n)}]=\frac{n+1}{n}X_{(n)}$: given the maximum, the other $n-1$ values are iid $U(0,X_{(n)})$ with mean $X_{(n)}/2$.
Check All three UMVUEs were checked by simulation, with means matching the targets to three decimals. Note that $(1-\frac1n)^T\to e^{-\lambda}$ a.s., since $T\log(1-\frac1n)\approx-T/n=-\bar X\to-\lambda$: the UMVUE and the MLE agree asymptotically.

Fisher Information, the Cramér–Rao Bound and Asymptotics of the MLE

📘 Definition 4.12 — Regular family, score and Fisher information

A one-parameter family $f(x;\theta)$ is regular if (R1) $\Theta$ is an open interval; (R2) the support $\{x:f(x;\theta)\gt0\}$ does not depend on $\theta$; (R3) $\theta\mapsto\log f(x;\theta)$ is twice differentiable; and (R4) $\int f(x;\theta)\,dx$ (or the sum) may be differentiated twice under the integral sign. The score is $s(x;\theta)=\frac{\partial}{\partial\theta}\log f(x;\theta)$, and the Fisher information in one observation is $$I(\theta)=E_\theta\left[s(X;\theta)^2\right].$$

An unbiased estimator whose variance equals the Cramér–Rao bound below for every $\theta$ is called efficient. The efficiency of an unbiased $T$ is the ratio of the bound to $\operatorname{Var}_\theta(T)$, and the relative efficiency of $T_1$ with respect to $T_2$ is $\operatorname{Var}(T_2)/\operatorname{Var}(T_1)$.

📐 Theorem 4.21 — Properties of the Fisher information

In a regular family:

  1. $E_\theta[s(X;\theta)]=0$, so $I(\theta)=\operatorname{Var}_\theta(s(X;\theta))$;
  2. $I(\theta)=-E_\theta\left[\dfrac{\partial^2}{\partial\theta^2}\log f(X;\theta)\right]$;
  3. a sample of $n$ iid observations has score $S_n=\sum_is(X_i;\theta)$ and information $I_n(\theta)=nI(\theta)$;
  4. if $\theta=h(\eta)$ with $h$ differentiable, the information about $\eta$ is $I(h(\eta))\,h'(\eta)^2$.

Proof. (1) Differentiate $\int f\,dx=1$: $0=\int\partial_\theta f\,dx=\int s\,f\,dx=E[s]$. (2) Differentiate again, using $\partial_\theta f=sf$: $0=\int(\partial_\theta s)f\,dx+\int s^2f\,dx$. (3) The score of the sample is the sum of $n$ iid mean-zero scores, so its variance is $nI(\theta)$. (4) By the chain rule the score in $\eta$ is $s\cdot h'(\eta)$. ∎

📐 Theorem 4.22 — The Cramér–Rao inequality

Let the family be regular with $0\lt I(\theta)\lt\infty$, and let $T$ be a statistic with $E_\theta[T]=\psi(\theta)$ and $\operatorname{Var}_\theta(T)\lt\infty$. Suppose $\int T(x)f_n(x;\theta)\,dx$ can be differentiated under the integral sign, where $f_n$ is the joint density. Then $$\operatorname{Var}_\theta(T)\ \ge\ \frac{\psi'(\theta)^2}{nI(\theta)}\qquad\text{for all }\theta.$$ Equality holds for all $\theta$ iff $S_n=a(\theta)\left(T-\psi(\theta)\right)$ a.s. for some function $a$. This happens exactly when the family is a one-parameter exponential family with natural statistic $\sum T_1(X_i)$ and $T$ is an affine function of it.

For a biased estimator with bias $b(\theta)$, the same argument gives $\operatorname{MSE}_\theta(T)\ge\dfrac{(1+b'(\theta))^2}{nI(\theta)}+b(\theta)^2$.

Proof. Differentiating $\psi(\theta)=\int T(x)f_n(x;\theta)\,dx$ under the integral sign gives $\psi'(\theta)=\int T\,S_n\,f_n\,dx=E_\theta[TS_n]$. Since $E_\theta[S_n]=0$, this is $\operatorname{Cov}_\theta(T,S_n)$. By the Cauchy–Schwarz inequality (§2.2), $$\psi'(\theta)^2=\operatorname{Cov}(T,S_n)^2\le\operatorname{Var}(T)\operatorname{Var}(S_n)=\operatorname{Var}(T)\cdot nI(\theta).$$ Equality in Cauchy–Schwarz holds iff $S_n$ and $T-\psi(\theta)$ are linearly dependent, i.e. $S_n=a(\theta)(T-\psi(\theta))$. Integrating this identity in $\theta$ gives $\log f_n=A_1(\theta)T+A_2(\theta)+c(x)$, an exponential family in $T$. ∎

The bound is not always attained. In a regular model, an efficient estimator exists only for $\psi(\theta)=E_\theta[T]$ with $T$ the natural statistic of an exponential family (and affine functions of it). So in the exponential model with rate $\lambda$, the mean $1/\lambda$ has an efficient estimator ($\bar X$), but $\lambda$ itself does not. The UMVUE $(n-1)/\sum X_i$ has variance $\frac{\lambda^2}{n-2}\gt\frac{\lambda^2}{n}$. A UMVUE need not be efficient. What is true is that the MLE attains the bound asymptotically.

📐 Theorem 4.23 — Consistency and asymptotic normality of the MLE

Suppose the family is regular and identifiable ($\theta\ne\theta'$ implies $f(\cdot;\theta)\ne f(\cdot;\theta')$), $0\lt I(\theta)\lt\infty$, and the third derivative satisfies $\left|\frac{\partial^3}{\partial\theta^3}\log f(x;\theta)\right|\le M(x)$ near the true $\theta_0$, with $E_{\theta_0}[M(X)]\lt\infty$. Then, with probability tending to $1$, the likelihood equation has a root $\hat\theta_n$ that is consistent for $\theta_0$, and $$\sqrt n\left(\hat\theta_n-\theta_0\right)\xrightarrow{d}N\left(0,\ \frac{1}{I(\theta_0)}\right).$$ By the delta method, $\sqrt n\left(g(\hat\theta_n)-g(\theta_0)\right)\xrightarrow{d}N\left(0,\frac{g'(\theta_0)^2}{I(\theta_0)}\right)$ for differentiable $g$. The MLE is asymptotically efficient.

Proof (sketch). Consistency. By the strong law, $\frac1n\left[\ell_n(\theta)-\ell_n(\theta_0)\right]\to E_{\theta_0}\log\frac{f(X;\theta)}{f(X;\theta_0)}$. By Jensen's inequality and identifiability, this limit is strictly negative for $\theta\ne\theta_0$, because $E\log Y\lt\log EY=\log1=0$ for the likelihood ratio $Y$. So for small $\delta\gt0$, with probability tending to $1$, $\ell_n(\theta_0\pm\delta)\lt\ell_n(\theta_0)$, and $\ell_n$ has a local maximum, hence a root of $\ell_n'$, in $(\theta_0-\delta,\theta_0+\delta)$.

Normality. Expand the score about $\theta_0$: $$0=\ell_n'(\hat\theta)=\ell_n'(\theta_0)+(\hat\theta-\theta_0)\ell_n''(\theta_0)+\tfrac12(\hat\theta-\theta_0)^2\ell_n'''(\theta^*)$$ for some $\theta^*$ between $\hat\theta$ and $\theta_0$. Rearranging, $$\sqrt n(\hat\theta-\theta_0)=\frac{n^{-1/2}\ell_n'(\theta_0)}{-n^{-1}\ell_n''(\theta_0)-\tfrac12(\hat\theta-\theta_0)\,n^{-1}\ell_n'''(\theta^*)}.$$ The numerator is a normalised sum of iid scores with mean $0$ and variance $I(\theta_0)$, so it converges to $N(0,I(\theta_0))$ by the CLT. In the denominator, $-n^{-1}\ell_n''(\theta_0)\to I(\theta_0)$ by the WLLN and Theorem 4.21(2). The last term is bounded by $|\hat\theta-\theta_0|\cdot\frac1n\sum M(X_i)\xrightarrow{P}0$. Slutsky's theorem gives the limit $N(0,I/I^2)=N(0,1/I)$. ∎

In practice the theorem gives the standard error $1/\sqrt{nI(\hat\theta)}$, or $1/\sqrt{-\ell_n''(\hat\theta)}$ using the observed information. This is the width of the purple bell in Interactive 1, and it is the basis of the large-sample intervals of §4.3 and the likelihood ratio tests of §5.1.

✏️ Example 7 — Information, bounds and the MLE in practice
(a) Exponential lifetimes with mean $\theta$: find $I(\theta)$ and decide whether $\bar X$ is efficient. (b) Normal with both parameters unknown: is $S^2$ efficient for $\sigma^2$? (c) $X_1,\dots,X_{20}$ iid with density $\theta x^{\theta-1}$ on $(0,1)$, with $\sum(-\log x_i)=10$. Find the MLE and the UMVUE of $\theta$, the Cramér–Rao bound, and an approximate $95\%$ interval from Theorem 4.23. (d) Does $\frac{n+1}{n}X_{(n)}$ for $U(0,\theta)$ violate the Cramér–Rao inequality?
(a) $\log f=-\log\theta-x/\theta$, so $s=-\frac1\theta+\frac{x}{\theta^2}=\frac{x-\theta}{\theta^2}$ and $I(\theta)=\frac{\operatorname{Var}X}{\theta^4}=\frac{1}{\theta^2}$. The bound for unbiased estimators of $\theta$ is $\frac{\theta^2}{n}=\operatorname{Var}(\bar X)$, so $\bar X$ is efficient. Indeed $S_n=\frac{n}{\theta^2}(\bar X-\theta)$ is linear in $\bar X$.
(b) The information about $\sigma^2$ in one normal observation is $\frac{1}{2\sigma^4}$. Since the information matrix of $(\mu,\sigma^2)$ is diagonal, the bound for $\sigma^2$ is $\frac{2\sigma^4}{n}$ whether or not $\mu$ is known. But $\operatorname{Var}(S^2)=\frac{2\sigma^4}{n-1}$. So $S^2$ is the UMVUE (Lehmann–Scheffé with the complete statistic $(\bar X,S^2)$) yet not efficient. Its efficiency is $\frac{n-1}{n}\to1$.
(c) $\ell(\theta)=n\log\theta+(\theta-1)\sum\log x_i$, so $\ell'=\frac n\theta-T$ with $T=-\sum\log x_i$, and $\hat\theta=n/T=20/10=2$. Since $-\log X_i\sim\operatorname{Exp}(\theta)$, we have $T\sim\operatorname{Gamma}(n,\theta)$ and $E[1/T]=\frac{\theta}{n-1}$. So the UMVUE is $\frac{n-1}{T}=1.9$, with variance $\frac{\theta^2}{n-2}$. Since $\ell''=-n/\theta^2$, $I(\theta)=1/\theta^2$ and the bound is $\theta^2/n$. The approximate interval is $\hat\theta\pm1.96\,\hat\theta/\sqrt n=2\pm0.877=(1.12,\ 2.88)$. An exact interval from the pivot $2\theta T\sim\chi^2_{2n}$ is in §4.3.
(d) Formally, $\log f=-\log\theta$ on $(0,\theta)$ gives "$I(\theta)=1/\theta^2$" and a "bound" $\theta^2/n$, while $\operatorname{Var}\left(\frac{n+1}{n}X_{(n)}\right)=\frac{\theta^2}{n(n+2)}$ is far smaller. There is no contradiction: (R2) fails because the support depends on $\theta$, and $\frac{d}{d\theta}\int_0^\theta T f\,dx$ cannot be computed by differentiating under the integral sign. Indeed $E[s]=-1/\theta\ne0$. The Cramér–Rao bound simply does not apply.
Interactive 2 — Efficiency: Estimators Against the Cramér–Rao Bound

Each tick simulates many samples of size $n$ (seeded) and adds the value of the chosen estimator to the blue histogram. The dashed purple curve is $N\left(\theta,\ \text{CRLB}\right)$, the narrowest bell an unbiased estimator could have in a regular model. The red line is the true $\theta$. An efficient estimator fills the purple bell exactly. A UMVUE that is not efficient is slightly wider. The uniform maximum is much narrower than the "bound", because the bound does not apply there. The efficiency readout is $\text{CRLB}/\text{MSE}$ from the simulation.

10
samples—
simulated mean, variance—
exact variance—
CRLB = 1/(n I)—
efficiency CRLB/MSE—
⚠️ Counterexamples and pitfalls
  1. The MLE need not be unbiased: $\hat\sigma^2=SS/n$, $X_{(n)}$ for $U(0,\theta)$ and $1/\bar X$ for an exponential rate are all biased. It need not be unique ($U(\theta,\theta+1)$, the Laplace median for even $n$), it need not solve $\ell'=0$ (uniform, shifted exponential), and it need not exist when the supremum is not attained in $\Theta$ (for example, $\operatorname{Bin}(n,p)$ with $\Theta=(0,1)$ and all trials failures).
  2. Moment estimators can be impossible: $2\bar X$ can be smaller than $X_{(n)}$ for $U(0,\theta)$, and moment estimators of variances can be negative.
  3. Unbiased estimators may not exist ($1/p$ for binomial data) or may be absurd ($(-1)^X$ for $e^{-2\lambda}$). Unbiasedness is not preserved by nonlinear maps ($E[S]\lt\sigma$), whereas MLEs are invariant.
  4. Unbiased does not imply consistent ($X_1$), and consistent does not imply asymptotically unbiased (Example 2(c)).
  5. Minimal sufficient does not imply complete: $N(\theta,\theta^2)$ and $U(\theta,\theta+1)$. Complete sufficient does imply minimal sufficient (Exercise III.6).
  6. Curved exponential families are not complete: Theorem 4.17 needs the natural parameters to fill an open set.
  7. A UMVUE need not attain the Cramér–Rao bound: $S^2$ for $\sigma^2$, and $(n-1)/\sum X_i$ for an exponential rate. Efficiency is the exception, confined to exponential families.
  8. The Cramér–Rao bound needs regularity: for $U(0,\theta)$ and the shifted exponential the support depends on $\theta$, and estimators with variance of order $1/n^2$ exist.
  9. Biased estimators can beat the bound and the UMVUE in MSE: $SS/(n+1)$ has smaller MSE than $S^2$; $\frac{n+2}{n+1}X_{(n)}$ beats $\frac{n+1}{n}X_{(n)}$.
  10. Information adds over independent observations ($nI(\theta)$), but it changes under reparametrisation by the factor $h'(\eta)^2$. "The information" always refers to a particular parameter.
  11. A likelihood may have several local maxima (Cauchy location with small $n$). The root of $\ell'=0$ found by an algorithm need not be the global maximum, and Theorem 4.23 concerns a consistent root.

Exercises

I
Bias, MSE and consistency.
  • 1 Let $X_1,\dots,X_n$ have common mean $\mu$ and variance $\sigma^2$ (iid). Show that $\sum a_iX_i$ is unbiased for $\mu$ iff $\sum a_i=1$, and that among these $\bar X$ has the smallest variance.
  • 2 For a normal sample with $n=5$, compute the MSEs of $S^2$ and of $\hat\sigma^2=SS/n$ in units of $\sigma^4$.
  • 3 For $U(0,\theta)$, compare the variances of $2\bar X$ and $\frac{n+1}{n}X_{(n)}$. For which $n$ is the second strictly better?
  • 4 Show that no unbiased estimator of $1/p$ exists for $X\sim\operatorname{Bin}(n,p)$, $0\lt p\lt1$.
  • 5 For a single observation $X\sim\operatorname{Poi}(\lambda)$, show that $(-1)^X$ is the only unbiased estimator of $e^{-2\lambda}$, and find its variance.
  • 6 Show that $X_{(n)}$ is consistent for $\theta$ in $U(0,\theta)$, and that $X_1$ is unbiased but not consistent for the mean of any non-degenerate population.
  • 7 Give an example of a consistent estimator whose MSE tends to $\infty$.
  • 8 Applied. Two laboratories measure the same quantity with independent unbiased estimators $T_1,T_2$ of variances $4$ and $9$. Find the unbiased combination $wT_1+(1-w)T_2$ of least variance and its variance.
1 $E[\sum a_iX_i]=\mu\sum a_i$. The variance is $\sigma^2\sum a_i^2$, and by Cauchy–Schwarz $1=\left(\sum a_i\right)^2\le n\sum a_i^2$, with equality iff all $a_i=\frac1n$.
2 $\operatorname{MSE}(S^2)=\frac{2\sigma^4}{n-1}=0.5\sigma^4$; $\operatorname{MSE}(\hat\sigma^2)=\frac{(2n-1)\sigma^4}{n^2}=\frac{9}{25}\sigma^4=0.36\sigma^4$.
3 $\frac{\theta^2}{3n}$ versus $\frac{\theta^2}{n(n+2)}$: they are equal at $n=1$, and the second is strictly smaller for $n\ge2$ (the ratio is $\frac{n+2}{3}$).
4 $E_p[T]=\sum_kT(k)\binom nkp^k(1-p)^{n-k}$ is a polynomial in $p$, hence bounded on $(0,1)$, while $1/p$ is unbounded there.
5 Uniqueness: see Example 2(b). Variance: $E[((-1)^X)^2]=1$, so $\operatorname{Var}=1-e^{-4\lambda}$, enormous compared with the target $e^{-2\lambda}$.
6 $P(|X_{(n)}-\theta|\gt\varepsilon)=\left(1-\frac\varepsilon\theta\right)^n\to0$. The law of $X_1$ does not depend on $n$, so if $P(|X_1-\mu|\gt\varepsilon)\gt0$ for some $\varepsilon$ (true for a non-degenerate law), it does not tend to $0$.
7 $T_n=\bar X+nB_n$ with $B_n\sim\operatorname{Ber}(1/n)$ independent: $P(T_n\ne\bar X)=\frac1n\to0$, so $T_n\xrightarrow{P}\mu$, but $E[(nB_n)^2]=n\to\infty$.
8 $\operatorname{Var}=4w^2+9(1-w)^2$ is minimised at $w=\frac{9}{13}$, giving $\frac{36}{13}=2.77$. The weights are inversely proportional to the variances, and the combined variance is smaller than either.
II
Method of moments and maximum likelihood.
  • 1 Find the MME and the MLE of the rate $\lambda$ for an $\operatorname{Exp}(\lambda)$ sample.
  • 2 Find the MMEs of the gamma parameters from $\bar x$ and $\hat\sigma^2$ (Example 3(b)), and explain why they are consistent.
  • 3 Verify that the normal MLE of Example 3(a) is a global maximum by showing that the Hessian of $\ell$ at $(\bar x,\hat\sigma^2)$ is negative definite.
  • 4 Find the MLE of $\theta$ for $U(-\theta,\theta)$.
  • 5 Find the MLE of $\theta$ for $U(\theta,2\theta)$, $\theta\gt0$.
  • 6 For the two-parameter exponential density $\frac1\sigma e^{-(x-\mu)/\sigma}$, $x\gt\mu$, find the MLE of $(\mu,\sigma)$.
  • 7 For $\operatorname{Exp}(\lambda)$ data with $\bar x=2$, find the MLE of $P(X\gt1)$.
  • 8 For a Cauchy location sample with $n=2$ and data $x_1=-a$, $x_2=a$, show that the likelihood has two maxima when $a\gt1$ and one when $a\le1$.
  • 9 Applied (genetics). Under Hardy–Weinberg equilibrium the genotypes AA, Aa, aa occur with probabilities $\theta^2$, $2\theta(1-\theta)$, $(1-\theta)^2$. A sample of $100$ people has counts $36$, $48$, $16$. Find the MLE of $\theta$ and its approximate standard error.
1 $E[X]=1/\lambda$ gives $\tilde\lambda=1/\bar X$. And $\ell=n\log\lambda-\lambda\sum x_i$ gives $\hat\lambda=n/\sum X_i=1/\bar X$. The two coincide.
2 $\tilde\alpha=\bar X^2/\hat\sigma^2$ and $\tilde\lambda=\bar X/\hat\sigma^2$. These are continuous functions of $(m_1',m_2')$ wherever $\hat\sigma^2\gt0$, so they are consistent by Theorem 4.13(4).
3 $\partial^2_\mu\ell=-n/\sigma^2$, $\partial_\mu\partial_{\sigma^2}\ell=-\sum(x_i-\mu)/\sigma^4=0$ at $\mu=\bar x$, and $\partial^2_{\sigma^2}\ell=\frac{n}{2\sigma^4}-\frac{SS}{\sigma^6}=-\frac{n}{2\hat\sigma^4}$ at $\sigma^2=\hat\sigma^2$. The Hessian is diagonal with negative entries. With the profile argument, the maximum is global.
4 $L=(2\theta)^{-n}\mathbf 1\{\theta\ge\max|x_i|\}$ is decreasing on the allowed set: $\hat\theta=\max_i|X_i|$.
5 $L=\theta^{-n}\mathbf 1\{\theta\le x_{(1)}\}\mathbf 1\{2\theta\ge x_{(n)}\}$, positive on $[x_{(n)}/2,\ x_{(1)}]$ (nonempty, since all $x_i\in(\theta,2\theta)$) and decreasing there: $\hat\theta=X_{(n)}/2$.
6 For fixed $\sigma$, $L\propto e^{n\mu/\sigma}\mathbf 1\{\mu\le x_{(1)}\}$ increases in $\mu$, so $\hat\mu=X_{(1)}$. Then $\ell=-n\log\sigma-\sum(x_i-x_{(1)})/\sigma$ gives $\hat\sigma=\bar X-X_{(1)}$.
7 $P(X\gt1)=e^{-\lambda}$, and by invariance its MLE is $e^{-1/\bar x}=e^{-0.5}=0.607$.
8 $\ell(\theta)=-\log(1+(\theta+a)^2)-\log(1+(\theta-a)^2)$. Then $\ell'(\theta)=-\frac{2(\theta+a)}{1+(\theta+a)^2}-\frac{2(\theta-a)}{1+(\theta-a)^2}$. Clearing denominators, $\ell'=0$ iff $\theta\left(\theta^2+1-a^2\right)=0$. For $a\le1$ the only root is $\theta=0$, a maximum. For $a\gt1$ there are three roots: $0$ is a local minimum, and $\pm\sqrt{a^2-1}$ are two equal maxima. So the MLE is not unique.
9 $L\propto\theta^{2n_1+n_2}(1-\theta)^{n_2+2n_3}$, so $\hat\theta=\frac{2n_1+n_2}{2n}=\frac{72+48}{200}=0.6$, the allele frequency among the $2n$ genes. The information is $\frac{2n}{\theta(1-\theta)}$, so $\operatorname{SE}\approx\sqrt{\frac{0.6\times0.4}{200}}=0.0346$.
III
Sufficiency, completeness and exponential families.
  • 1 Use the factorisation theorem to find a sufficient statistic for (a) $\operatorname{Gamma}(\alpha,\lambda)$ with both parameters unknown; (b) $U(\theta_1,\theta_2)$; (c) $\operatorname{Geo}(p)$.
  • 2 Show that the Rayleigh family $f(x;\theta)=\frac{x}{\theta}e^{-x^2/(2\theta)}$, $x\gt0$, is a full-rank exponential family, find a complete sufficient statistic, and find the UMVUE of $\theta$.
  • 3 Show that $\sum X_i$ is complete for a $\operatorname{Ber}(p)$ sample.
  • 4 For $f(x;\theta)=e^{-(x-\theta)}$, $x\gt\theta$, show that $X_{(1)}$ is complete sufficient.
  • 5 For $N(\theta,\theta)$ with $\theta\gt0$ (mean equal to variance), find a complete sufficient statistic. Is $\sum X_i$ sufficient?
  • 6 Prove that a complete sufficient statistic is minimal sufficient, assuming a minimal sufficient statistic exists.
  • 7 Use Basu's theorem to show that, for an $\operatorname{Exp}(\lambda)$ sample, $X_1/\sum X_i$ is independent of $\sum X_i$, and find $E[X_1/\sum X_i]$.
  • 8 For a $U(0,\theta)$ sample, use Basu's theorem to show that $X_{(1)}/X_{(n)}$ is independent of $X_{(n)}$.
1 (a) $\frac{\lambda^{n\alpha}}{\Gamma(\alpha)^n}\left(\prod x_i\right)^{\alpha-1}e^{-\lambda\sum x_i}$: the pair $(\prod X_i,\sum X_i)$, or $(\sum\log X_i,\sum X_i)$. (b) $(\theta_2-\theta_1)^{-n}\mathbf 1\{\theta_1\le x_{(1)}\}\mathbf 1\{x_{(n)}\le\theta_2\}$: the pair $(X_{(1)},X_{(n)})$. (c) $p^n(1-p)^{\sum x_i-n}$: $\sum X_i$.
2 $f=x\exp\left(-\frac{1}{2\theta}x^2-\log\theta\right)$, with natural parameter $\eta=-\frac{1}{2\theta}$ ranging over $(-\infty,0)$, an open interval. So $T=\sum X_i^2$ is complete sufficient. $X^2\sim\operatorname{Exp}$ with mean $2\theta$, so $E[T]=2n\theta$ and $T/(2n)$ is the UMVUE.
3 $T\sim\operatorname{Bin}(n,p)$. If $\sum_tg(t)\binom ntp^t(1-p)^{n-t}=0$ for all $p$, divide by $(1-p)^n$ and put $\rho=p/(1-p)\in(0,\infty)$: the polynomial $\sum g(t)\binom nt\rho^t$ vanishes identically, so $g\equiv0$.
4 Sufficiency: $\prod e^{-(x_i-\theta)}\mathbf 1\{x_i\gt\theta\}=e^{n\theta}\mathbf 1\{x_{(1)}\gt\theta\}\cdot e^{-\sum x_i}$. Completeness: $X_{(1)}$ has density $ne^{-n(t-\theta)}$, $t\gt\theta$. If $\int_\theta^\infty g(t)e^{-nt}dt=0$ for all $\theta$, differentiating in $\theta$ gives $g(\theta)e^{-n\theta}=0$ a.e.
5 $f=(2\pi\theta)^{-1/2}\exp\left(-\frac{x^2}{2\theta}+x-\frac\theta2\right)=e^x\exp\left(-\frac{1}{2\theta}x^2-\frac12\log(2\pi\theta)-\frac\theta2\right)$. This is a one-parameter full-rank family in $T(x)=x^2$, so $\sum X_i^2$ is complete sufficient. $\sum X_i$ is not sufficient: the factor $e^{\sum x_i}$ goes into $h$, and $\sum X_i$ is not a function of the minimal sufficient statistic $\sum X_i^2$.
6 Let $T$ be complete sufficient and $U$ minimal sufficient, $U=\varphi(T)$. Put $g(T)=T-E[T\mid U]$. Then $E[g(T)]=0$ for all $\theta$ (for a bounded version, apply this to bounded functions of $T$), so $T=E[T\mid U]$ a.s. Hence $T$ is a function of $U$, and therefore minimal.
7 $\sum X_i$ is complete sufficient (full-rank family). $X_i=E_i/\lambda$ with $E_i$ iid $\operatorname{Exp}(1)$, so $X_1/\sum X_i=E_1/\sum E_i$ is ancillary. Basu gives independence. By exchangeability the $n$ ratios $X_j/\sum X_i$ have equal means and sum to $1$, so each mean is $\frac1n$.
8 $X_{(n)}$ is complete sufficient (Example 5(b)), and $X_{(1)}/X_{(n)}=U_{(1)}/U_{(n)}$ is ancillary (scale-free). Basu gives independence, confirming E6(B) of §4.1.
IV
UMVUE, information and asymptotics.
  • 1 For a $\operatorname{Ber}(p)$ sample, find the UMVUE of $p^2$.
  • 2 For a $\operatorname{Poi}(\lambda)$ sample, find the UMVUE of $\lambda^2$.
  • 3 For an $\operatorname{Exp}(\lambda)$ sample with $n\ge3$, find the UMVUE of $\lambda$ and its variance, and compare with the Cramér–Rao bound.
  • 4 For a $U(0,\theta)$ sample, find the UMVUE of $\theta^2$.
  • 5 For $N(\mu,1)$, find the UMVUEs of $\mu^2$ and of $P(X\le c)=\Phi(c-\mu)$.
  • 6 Compute the Fisher information and the Cramér–Rao bound for unbiased estimation of (a) $\mu$ in $N(\mu,\sigma^2)$ with $\sigma$ known; (b) $p$ in $\operatorname{Ber}(p)$; (c) $\lambda$ in $\operatorname{Poi}(\lambda)$. Which natural estimators attain them?
  • 7 For the Cauchy location family $\frac{1}{\pi(1+(x-\theta)^2)}$, show that $I(\theta)=\frac12$, and find the asymptotic efficiency of the sample median.
  • 8 For $f(x;\theta)=e^{-(x-\theta)}$, $x\gt\theta$, show that $X_{(1)}-\frac1n$ is the UMVUE of $\theta$ with variance $\frac{1}{n^2}$, and explain why this does not contradict the Cramér–Rao inequality.
  • 9 Applied. A city records $120$ road accidents in $50$ days, modelled as iid $\operatorname{Poi}(\lambda)$ per day. Find the MLE and the UMVUE of the probability of an accident-free day, and an approximate standard error for the MLE.
1 $E[T(T-1)]=n(n-1)p^2$ for $T\sim\operatorname{Bin}(n,p)$ (the second factorial moment), so the UMVUE is $\frac{T(T-1)}{n(n-1)}$.
2 $T\sim\operatorname{Poi}(n\lambda)$ has $E[T(T-1)]=(n\lambda)^2$, so the UMVUE is $\frac{T(T-1)}{n^2}$.
3 $T=\sum X_i\sim\operatorname{Gamma}(n,\lambda)$ has $E[1/T]=\frac{\lambda}{n-1}$ and $E[1/T^2]=\frac{\lambda^2}{(n-1)(n-2)}$. So $\frac{n-1}{T}$ is the UMVUE, with variance $\frac{(n-1)\lambda^2}{n-2}-\lambda^2=\frac{\lambda^2}{n-2}$. Since $I(\lambda)=\frac{1}{\lambda^2}$, the bound is $\frac{\lambda^2}{n}$: the efficiency is $\frac{n-2}{n}$.
4 $E[X_{(n)}^2]=\frac{n\theta^2}{n+2}$, so the UMVUE is $\frac{n+2}{n}X_{(n)}^2$.
5 $E[\bar X^2]=\mu^2+\frac1n$, so the UMVUE of $\mu^2$ is $\bar X^2-\frac1n$ (it can be negative). For $\Phi(c-\mu)$, Rao–Blackwellise $\mathbf 1\{X_1\le c\}$. Since $X_1-\bar X\sim N(0,1-\frac1n)$ is independent of $\bar X$, $$E[\mathbf 1\{X_1\le c\}\mid\bar X]=\Phi\left(\frac{c-\bar X}{\sqrt{1-1/n}}\right).$$
6 (a) $I=\frac1{\sigma^2}$, bound $\frac{\sigma^2}{n}$, attained by $\bar X$. (b) $I=\frac{1}{p(1-p)}$, bound $\frac{p(1-p)}{n}$, attained by $\hat p$. (c) $I=\frac1\lambda$, bound $\frac\lambda n$, attained by $\bar X$. All three are full-rank exponential families estimating the mean of the natural statistic.
7 $s=\frac{2(x-\theta)}{1+(x-\theta)^2}$, and $I=\int\frac{4u^2}{(1+u^2)^2}\cdot\frac{du}{\pi(1+u^2)}=\frac12$ (verified numerically). The MLE has asymptotic variance $\frac2n$, and the median $\frac{\pi^2}{4n}$ (asymptotic normality of sample quantiles, §4.1). The efficiency of the median is $\frac{2}{\pi^2/4}=\frac{8}{\pi^2}=0.81$.
8 $X_{(1)}-\theta\sim\operatorname{Exp}(n)$, so $E[X_{(1)}]=\theta+\frac1n$ and $\operatorname{Var}=\frac{1}{n^2}$. $X_{(1)}$ is complete sufficient (III.4), so Lehmann–Scheffé applies. The support depends on $\theta$, so (R2) fails and the bound is not valid. Variances of order $1/n^2$ are typical of such "endpoint" problems.
9 $\hat\lambda=120/50=2.4$. MLE of $e^{-\lambda}$: $e^{-2.4}=0.0907$. UMVUE: $\left(\frac{49}{50}\right)^{120}=0.0885$. By the delta method, $\operatorname{SE}\approx e^{-\hat\lambda}\sqrt{\hat\lambda/n}=0.0907\times\sqrt{0.048}=0.0199$.
🎯 Exam corner CSIR Part BCSIR Part CGATE ST

E1. CSIR Part B GATE ST Let $X_1,\dots,X_n$ ($n\ge2$) be iid $\operatorname{Poi}(\lambda)$ and $T=\sum X_i$. The UMVUE of $e^{-\lambda}$ is (A) $e^{-T/n}$; (B) $\left(1-\frac1n\right)^T$; (C) $\left(1-\frac1n\right)^{T/n}$; (D) the proportion of the $X_i$ equal to $0$.

Idea Rao–Blackwellise $\mathbf 1\{X_1=0\}$ given the complete sufficient statistic $T$.
Work $X_1\mid T\sim\operatorname{Bin}(T,\frac1n)$, so $P(X_1=0\mid T)=(1-\frac1n)^T$. (A) is the MLE, which is biased. (D) is unbiased but not a function of $T$, so it is not the UMVUE. (C) is not unbiased.
Answer (B).

E2. CSIR Part B Let $X_1,\dots,X_n$ be iid with density $f(x;\theta)=e^{-(x-\theta)}$ for $x\gt\theta$. The maximum likelihood estimator of $\theta$ is (A) $\bar X$; (B) $X_{(1)}$; (C) $\bar X-1$; (D) $X_{(1)}-\frac1n$.

Idea The support depends on $\theta$; write the likelihood with its indicator.
Work $L(\theta)=e^{n\theta-\sum x_i}\mathbf 1\{\theta\lt x_{(1)}\}$ increases up to $x_{(1)}$. (C) is the unbiased moment estimator, and (D) is the UMVUE. Neither maximises $L$.
Answer (B).

E3. CSIR Part B Let $X_1,\dots,X_n$ be iid $\operatorname{Ber}(p)$, $0\lt p\lt1$. The Cramér–Rao lower bound for the variance of unbiased estimators of $p(1-p)$ is (A) $\frac{p(1-p)}{n}$; (B) $\frac{(1-2p)^2p(1-p)}{n}$; (C) $\frac{(1-2p)^2}{np(1-p)}$; (D) $\frac{p^2(1-p)^2}{n}$.

Idea Bound $=\psi'(p)^2/(nI(p))$.
Work $\psi'(p)=1-2p$ and $I(p)=\frac{1}{p(1-p)}$, so the bound is $(1-2p)^2\frac{p(1-p)}{n}$. (C) divides by the information instead of multiplying by its reciprocal. Note the bound is $0$ at $p=\frac12$, yet no unbiased estimator has variance $0$ there: the bound need not be attained.
Answer (B).

E4. CSIR Part C GATE ST Let $X_1,\dots,X_n$ ($n\ge2$) be iid $U(0,\theta)$, $\theta\gt0$. Which of the following are true? (One or more options may be correct.) (A) $X_{(n)}$ is a complete sufficient statistic. (B) $2\bar X$ is the UMVUE of $\theta$. (C) The MLE of $\theta$ is unbiased. (D) The MLE of $\theta^2$ is $X_{(n)}^2$.

(A) True Example 5(b).
(B) False $2\bar X$ is unbiased but is not a function of $X_{(n)}$. Its variance $\frac{\theta^2}{3n}$ exceeds that of the UMVUE $\frac{n+1}{n}X_{(n)}$, which is $\frac{\theta^2}{n(n+2)}$, for $n\ge2$.
(C) False $E[X_{(n)}]=\frac{n}{n+1}\theta\lt\theta$.
(D) True Invariance (Theorem 4.14).
Answer (A) and (D).

E5. CSIR Part C Let $X_1,\dots,X_n$ ($n\ge2$) be iid $N(\mu,\sigma^2)$ with both parameters unknown. Which of the following are true? (A) $S^2$ attains the Cramér–Rao lower bound for unbiased estimators of $\sigma^2$. (B) $S^2$ is the UMVUE of $\sigma^2$. (C) $(\bar X,S^2)$ is a complete sufficient statistic. (D) $\bar X^2$ is an unbiased estimator of $\mu^2$.

(A) False $\operatorname{Var}(S^2)=\frac{2\sigma^4}{n-1}\gt\frac{2\sigma^4}{n}$ (Example 7(b)).
(B) True It is an unbiased function of the complete sufficient statistic; apply Lehmann–Scheffé.
(C) True The normal family is a full-rank two-parameter exponential family with natural statistic $(\sum X_i,\sum X_i^2)$, which is in one-to-one correspondence with $(\bar X,S^2)$.
(D) False $E[\bar X^2]=\mu^2+\frac{\sigma^2}{n}$. The UMVUE of $\mu^2$ is $\bar X^2-\frac{S^2}{n}$.
Answer (B) and (C).

E6. CSIR Part C Which of the following statements about estimators are true? (A) If $T_n$ is unbiased for $\theta$ and $\operatorname{Var}(T_n)\to0$, then $T_n$ is consistent. (B) Every consistent estimator is asymptotically unbiased. (C) If $T$ is unbiased for $\theta$, then $T^2$ is unbiased for $\theta^2$. (D) If the MLE is unique, it is a function of every sufficient statistic.

(A) True MSE $=\operatorname{Var}\to0$; apply Theorem 4.13(2).
(B) False Example 2(c): $T_n=\bar X$ with probability $1-\frac1n$ and $n^2$ otherwise.
(C) False $E[T^2]=\theta^2+\operatorname{Var}(T)\gt\theta^2$ unless $T$ is degenerate.
(D) True If $T$ is sufficient, $L(\theta)=g(T(x);\theta)h(x)$, so maximising $L$ is the same as maximising $g(T(x);\cdot)$, which depends on $x$ only through $T(x)$. If the maximiser is unique, it is a function of $T(x)$.
Answer (A) and (D).
🧑‍🏫 For the teacher

Suggested plan — 3 lectures × 60 min (a fourth for weaker groups).

  1. Criteria and methods. Definition 4.4 and Theorem 4.13, then Example 1(a) on the board (the unbiased estimator loses on MSE) and Example 2 as a quick poll. Method of moments and likelihood (Definitions 4.5–4.6), invariance (Theorem 4.14), the strategy box, Examples 3 and 4. Project Interactive 1: Bernoulli, then Uniform (the cliff), then Cauchy at $n=3$ to $6$ (several peaks).
  2. Sufficiency and completeness. The idea of "regenerating the sample"; the factorisation theorem with its discrete proof; minimal sufficiency via the ratio criterion; completeness and exponential families (Theorem 4.17 as a recipe); Basu's theorem with the $\bar X\perp S^2$ application. Example 5, with (c) and (d) as the two key non-examples.
  3. Optimal estimators and information. Rao–Blackwell and Lehmann–Scheffé with full proofs, the UMVUE strategy box, and Example 6. Fisher information, the Cramér–Rao proof via Cauchy–Schwarz, when equality holds, and the asymptotic normality of the MLE. Example 7, then project Interactive 2 cycling through the six estimators. End with E4–E6.

Discussion questions. (1) Would you rather have an unbiased estimator or one with smaller MSE? Does the answer depend on whether many estimates will be averaged later? (2) Why can the variance of an estimator for $U(0,\theta)$ fall below the Cramér–Rao "bound", and what does this say about information in non-regular problems? (3) The UMVUE of $e^{-\lambda}$ and the MLE $e^{-\bar X}$ differ for small $n$. Which would you report to a public-health office, and why?

Misconceptions to watch for. Differentiating a likelihood that has an indicator; assuming MLEs are unbiased or unique; confusing "minimal sufficient" with "complete"; applying the Cramér–Rao bound to uniform or shifted models; forgetting $\psi'(\theta)^2$ in the bound for a function of $\theta$; thinking a UMVUE must attain the bound; treating unbiasedness as preserved under nonlinear maps.

Suggested homework. I(2, 4, 7, 8), II(4, 5, 6, 8, 9), III(2, 4, 5, 7), IV(1, 3, 5, 7, 9); Exam corner E1, E3, E5.

🔬 Research–Teaching NexusResearch-tutored · optional, about 45 minutes

Research question. How many fish are in the lake? Compare the maximum likelihood estimator of a population size from a capture–recapture experiment with a slightly modified estimator, and find out which one ecologists should use.

Why it matters. Fisheries agencies, wildlife services and conservation projects around the world estimate population sizes by marking $M$ animals, releasing them, and later recording how many of a second catch of $n$ are marked. The same mathematics is used to estimate the number of people missed by a census, the size of hidden populations in epidemiology, and the number of undiscovered bugs in software.

What to do.

  1. If the population has $N$ animals, the number $m$ of marked animals in the second catch is hypergeometric (§2.1). Show that the likelihood ratio is $\frac{L(N)}{L(N-1)}=\frac{(N-M)(N-n)}{N(N-M-n+m)}$, and deduce that the MLE is the integer part of $Mn/m$ (the Lincoln–Petersen estimator). When $Mn/m$ is an integer there are two MLEs: explain why.
  2. Take $M=100$, $n=80$, $m=16$. Compute the MLE and Chapman's estimator $\frac{(M+1)(n+1)}{m+1}-1$.
  3. Explain why the Lincoln–Petersen estimator has infinite expectation for every $N$, and so no finite bias or MSE. (Hint: $P(m=0)\gt0$.)
  4. For $N=500$, compute the exact mean and standard deviation of Chapman's estimator by summing over the hypergeometric law, and the mean of the Lincoln–Petersen estimator conditional on $m\gt0$.

Tools: Python with scipy.stats.hypergeom, or a spreadsheet.

Hand in: Your derivation of the MLE, a table comparing the two estimators (mean, bias, SD) at $N=500$, and a recommendation in three sentences.

Going further. Read about how capture–recapture extends to several capture occasions (the Schnabel and Jolly–Seber models) and to estimating the undercount of a national census by a post-enumeration survey.

Instructor note: $L(N)/L(N-1)\ge1$ iff $N\le Mn/m$. With $M=100$, $n=80$, $m=16$, $Mn/m=500$ exactly, so $L(499)=L(500)$ and both are MLEs. Chapman gives $\frac{101\cdot81}{17}-1=480.2$. For $N=500$: $P(m=0)\approx2.9\times10^{-9}$, which is tiny but positive, so $E[Mn/m]=\infty$. Chapman's estimator has mean $500.0$ (it is exactly unbiased when $M+n\ge N$; here $M+n=180\lt500$, and it is still unbiased to within $10^{-6}$) and SD about $107$. The Lincoln–Petersen estimator conditional on $m\gt0$ has mean about $524$, an upward bias of roughly $5\%$.

🧪 Self-check quiz

0/5 correct

Q1. For an iid sample from $U(0,\theta)$, the maximum likelihood estimator of $\theta$ is

Q2. For an iid $\operatorname{Poi}(\lambda)$ sample, a sufficient statistic is

Q3. Conditioning an unbiased estimator on a sufficient statistic (Rao–Blackwell)

Q4. The Cramér–Rao lower bound for unbiased estimators of $\lambda$ from $n$ iid $\operatorname{Poi}(\lambda)$ observations is

Q5. For which family is the minimal sufficient statistic not complete?