4.2 Methods of Estimation and Properties of Estimators
How to turn a sample into an estimate, and how to tell a good estimate from a bad one: moments and maximum likelihood; bias, MSE and consistency; sufficiency and completeness; the Rao–Blackwell and Lehmann–Scheffé route to the best unbiased estimator; and the Cramér–Rao bound that no unbiased estimator can beat.
A model with an unknown parameter, $X_1,\dots,X_n$ iid with density or pmf $f(x;\theta)$, $\theta\in\Theta$, is the starting point of nearly every statistical analysis. The infection rate in a Poisson model of case counts, the mean lifetime in an exponential model of component failures, the proportion of defectives in a binomial model of inspections: in each case the data must be turned into a number $\hat\theta$, an estimate. There are infinitely many ways to do this. For a uniform sample on $(0,\theta)$, $2\bar X$, $X_{(n)}$, $\frac{n+1}{n}X_{(n)}$ and $(n+1)X_{(1)}$ all have some claim to estimate $\theta$. Two questions organise the subject: how do we find reasonable estimators, and how do we compare them?
This section answers both. The method of moments and the method of maximum likelihood produce estimators almost automatically. Bias, mean squared error and consistency are the first yardsticks. Sufficiency identifies which functions of the data carry all the information about $\theta$, and completeness makes the best unbiased estimator unique. The Rao–Blackwell and Lehmann–Scheffé theorems then construct the uniformly minimum variance unbiased estimator (UMVUE). The Fisher information and the Cramér–Rao inequality give a floor below which no unbiased estimator's variance can go, and the MLE reaches that floor asymptotically. CSIR Part B questions regularly ask for an MLE, a UMVUE or a Cramér–Rao bound. Part C questions test the exceptions: the uniform model where the bound "fails", curved families that are not complete, MLEs that are not unique or not unbiased, and unbiased estimators that do not exist.
Estimators: Bias, Mean Squared Error and Consistency
Let $X_1,\dots,X_n$ be iid from $f(x;\theta)$, $\theta\in\Theta$, and let $\psi(\theta)$ be a real function of the parameter. An estimator of $\psi(\theta)$ is any statistic $T=T_n(X_1,\dots,X_n)$; its value on the data is an estimate. Writing $E_\theta$ for expectations under $\theta$:
- the bias is $b_T(\theta)=E_\theta[T]-\psi(\theta)$, and $T$ is unbiased if $b_T(\theta)=0$ for every $\theta\in\Theta$; $T_n$ is asymptotically unbiased if $b_{T_n}(\theta)\to0$ for every $\theta$;
- the mean squared error is $\operatorname{MSE}_\theta(T)=E_\theta\left[(T-\psi(\theta))^2\right]$;
- $T_n$ is (weakly) consistent if $T_n\xrightarrow{P}\psi(\theta)$ under every $\theta$, strongly consistent if $T_n\to\psi(\theta)$ almost surely, and MSE-consistent if $\operatorname{MSE}_\theta(T_n)\to0$.
- $\operatorname{MSE}_\theta(T)=\operatorname{Var}_\theta(T)+b_T(\theta)^2$.
- If $b_{T_n}(\theta)\to0$ and $\operatorname{Var}_\theta(T_n)\to0$ for every $\theta$, then $T_n$ is MSE-consistent and hence consistent.
- If $T_n$ is consistent for $\theta$ and $g$ is continuous at $\theta$, then $g(T_n)$ is consistent for $g(\theta)$.
- If $E|X_1|^r\lt\infty$, the sample moment $m_r'=\frac1n\sum X_i^r$ is strongly consistent for $\mu_r'$. Hence every estimator of the form $h(m_1',\dots,m_k')$ with $h$ continuous at $(\mu_1',\dots,\mu_k')$ is consistent.
Proof. (1) Expand $(T-\psi)^2=\left((T-E T)+(ET-\psi)\right)^2$; the cross term has mean $0$. (2) By (1), $\operatorname{MSE}\to0$, and Chebyshev–Markov gives $P(|T_n-\psi|\gt\varepsilon)\le\operatorname{MSE}/\varepsilon^2\to0$. (3) is the continuous mapping theorem (§2.3). (4) is Kolmogorov's strong law (§2.3) applied to the iid variables $X_i^r$, followed by (3) in $k$ variables. ∎
Unbiasedness is a property of the average of $T$ over repeated samples. Consistency is a property of the whole sequence $T_n$ as $n\to\infty$. Neither implies the other (Example 2). Mean squared error is the most common single measure of accuracy: it rewards small variance and penalises bias, and the decomposition (1) shows the trade-off between them.
| estimator | bias | MSE$/\theta^2$ | $n=5$ |
|---|---|---|---|
| $2\bar X$ | $0$ | $\frac{1}{3n}$ | $0.0667$ |
| $X_{(n)}$ (MLE) | $-\frac{\theta}{n+1}$ | $\frac{2}{(n+1)(n+2)}$ | $0.0476$ |
| $\frac{n+1}{n}X_{(n)}$ | $0$ | $\frac{1}{n(n+2)}$ | $0.0286$ |
| $\frac{n+2}{n+1}X_{(n)}$ | $-\frac{\theta}{(n+1)^2}$ | $\frac{1}{(n+1)^2}$ | $0.0278$ |
The Method of Moments and Maximum Likelihood
The likelihood is not a density in $\theta$, and it does not integrate to $1$. It ranks parameter values by how probable they make the data that were actually observed. The MLE need not exist, need not be unique, need not solve the likelihood equation, and need not be unbiased (Example 4). One property, however, makes it very convenient.
Proof. For every $\lambda\in\Lambda$, $L^*(\lambda)\le\sup_\theta L(\theta)=L(\hat\theta)$. Since $\hat\theta$ itself belongs to $\{\theta:\tau(\theta)=\tau(\hat\theta)\}$, also $L(\hat\theta)\le L^*(\tau(\hat\theta))$. Hence $L^*(\lambda)\le L^*(\tau(\hat\theta))$ for all $\lambda$. ∎
For example, the MLE of $\sigma$ in a normal model is $\sqrt{\hat\sigma^2}$, and the MLE of $P(X\gt1)=e^{-\lambda}$ in an exponential model is $e^{-1/\bar X}$. Unbiasedness has no such property.
- Write $L(\theta)$ with all indicator functions. If the support depends on $\theta$ (uniform, shifted exponential, Pareto), the indicators restrict $\theta$ to an interval determined by $X_{(1)}$ or $X_{(n)}$.
- Non-regular case: on that interval, decide whether $L$ is monotone. If it is, the MLE is the appropriate endpoint, an order statistic. Do not differentiate across the indicator.
- Regular case: solve $\ell'(\theta)=0$. Check that it is a maximum ($\ell''\lt0$, or $\ell\to-\infty$ at the ends of $\Theta$) and compare with the boundary values.
- Several parameters: solve the score equations jointly, often by profiling (maximise over one parameter for each fixed value of the other).
- Functions of $\theta$: use invariance. Check uniqueness: a flat likelihood (as for $U(\theta,\theta+1)$) gives many MLEs.
Sufficiency, Minimal Sufficiency and Completeness
A statistic $T$ is sufficient if, once $T$ is known, the rest of the data carry no further information about $\theta$. In other words, the conditional law of the sample given $T$ does not involve $\theta$, so a statistician who knew only $T$ could regenerate a sample with exactly the same law as the original without knowing $\theta$.
Proof (discrete case). Write $f(x;\theta)=P_\theta(X=x)$. ($\Leftarrow$) Suppose the factorisation holds and fix $t$ with $P_\theta(T=t)\gt0$. For $x$ with $T(x)=t$, $$\begin{aligned}P_\theta(X=x\mid T=t)&=\frac{f(x;\theta)}{\sum_{y:T(y)=t}f(y;\theta)}\\&=\frac{g(t;\theta)h(x)}{g(t;\theta)\sum_{y:T(y)=t}h(y)}\\&=\frac{h(x)}{\sum_{y:T(y)=t}h(y)},\end{aligned}$$ which is free of $\theta$. For $T(x)\ne t$ the conditional probability is $0$. ($\Rightarrow$) If $T$ is sufficient, put $g(t;\theta)=P_\theta(T=t)$ and $h(x)=P(X=x\mid T=T(x))$, which is free of $\theta$ by assumption. Then $$f(x;\theta)=P_\theta(T=T(x))\,P(X=x\mid T=T(x))=g(T(x);\theta)h(x).$$ The continuous case needs measure-theoretic conditioning; the statement is the same, and we use it freely. ∎
Examples follow at once. For $\operatorname{Ber}(p)$, $\prod p^{x_i}(1-p)^{1-x_i}=p^{t}(1-p)^{n-t}$ with $t=\sum x_i$, so $\sum X_i$ is sufficient. For $N(\mu,\sigma^2)$, the density is $(2\pi\sigma^2)^{-n/2}\exp\left(-\frac{\sum x_i^2-2\mu\sum x_i+n\mu^2}{2\sigma^2}\right)$, so $(\sum X_i,\sum X_i^2)$, or equivalently $(\bar X,S^2)$, is sufficient. For $U(0,\theta)$, $\theta^{-n}\mathbf 1\{x_{(n)}\le\theta\}\mathbf 1\{x_{(1)}\gt0\}$ shows that $X_{(n)}$ is sufficient. The whole sample, and the vector of order statistics, are always sufficient; the question is how far the data can be reduced.
Proof (sketch). Sufficiency. For each value $t$ of $T$ choose one point $x_t$ with $T(x_t)=t$. By the criterion, $h(x)=f(x;\theta)/f(x_{T(x)};\theta)$ does not depend on $\theta$, so $f(x;\theta)=f(x_{T(x)};\theta)\,h(x)=g(T(x);\theta)h(x)$, and the factorisation theorem applies. Minimality. Let $U$ be sufficient, with $f=\tilde g(U(x);\theta)\tilde h(x)$. If $U(x)=U(y)$, then $f(x;\theta)/f(y;\theta)=\tilde h(x)/\tilde h(y)$ is free of $\theta$, so $T(x)=T(y)$ by the criterion. Thus $T$ is constant wherever $U$ is constant, so $T$ is a function of $U$. ∎
A statistic $T$ (or its family of laws) is complete if, for every measurable $g$, $$E_\theta[g(T)]=0\ \text{ for all }\theta\in\Theta\quad\Longrightarrow\quad P_\theta\left(g(T)=0\right)=1\ \text{ for all }\theta\in\Theta.$$ It is boundedly complete if this holds for every bounded $g$. A statistic $V$ is ancillary if its distribution does not depend on $\theta$.
Completeness says that the family of laws of $T$ is rich enough that no nonzero function of $T$ has mean zero for all $\theta$. In particular, there is at most one unbiased estimator of $\psi(\theta)$ that is a function of $T$.
Proof (sketch). The joint density is $$\prod_i h(x_i)\cdot\exp\Big(\sum_j\eta_j(\theta)\mathbf T_j(x)-nA(\theta)\Big),$$ so the factorisation theorem gives sufficiency. For completeness, the law of $\mathbf T$ has the form $c(\eta)e^{\eta\cdot t}\,\nu(dt)$ for a measure $\nu$ that does not depend on $\eta$. If $E_\eta[g(\mathbf T)]=0$ for all $\eta$ in an open rectangle, then $\int g^+(t)e^{\eta\cdot t}\nu(dt)=\int g^-(t)e^{\eta\cdot t}\nu(dt)$ there. Two Laplace transforms that agree on an open set are equal as measures (the uniqueness theorem for Laplace transforms, as for mgfs in §1.3). Hence $g^+=g^-$ $\nu$-a.e., i.e. $g(\mathbf T)=0$ a.s. Minimality follows from Theorem 4.16: the ratio $$\frac{f(x;\theta)}{f(y;\theta)}=\frac{\prod h(x_i)}{\prod h(y_i)}\exp\Big(\sum_j\eta_j(\theta)\left(\mathbf T_j(x)-\mathbf T_j(y)\right)\Big)$$ is free of $\theta$ iff $\mathbf T(x)=\mathbf T(y)$, because the full-rank condition rules out a fixed linear relation among the $\eta_j(\theta)$. ∎
The binomial, Poisson, geometric, negative binomial (known $r$), exponential, gamma, beta and normal families are all of full rank in their usual parametrisations. Uniform laws with unknown endpoints are not exponential families, since their support moves, and completeness there must be checked by hand.
Proof. Fix an event $\{V\in B\}$. Its probability $p_B=P_\theta(V\in B)$ does not depend on $\theta$ ($V$ is ancillary), and $\varphi(T)=P(V\in B\mid T)$ does not depend on $\theta$ ($T$ is sufficient), so $\varphi$ is a bounded statistic. Now $E_\theta[\varphi(T)-p_B]=0$ for all $\theta$, and bounded completeness gives $\varphi(T)=p_B$ a.s. Thus $P(V\in B\mid T)=P(V\in B)$ for every $B$, which is independence. ∎
Basu's theorem gives a second proof that $\bar X$ and $S^2$ are independent for normal samples. Fix $\sigma^2$. Then $\bar X$ is complete sufficient for $\mu$ (a full-rank one-parameter family), and $S^2$ is ancillary for $\mu$ because it is unchanged by shifts. Since $\sigma^2$ was arbitrary, independence holds for every $(\mu,\sigma^2)$.
Best Unbiased Estimators: Rao–Blackwell and Lehmann–Scheffé
Proof. Since $T$ is sufficient, the conditional law of the data given $T$, and hence $E[W\mid T]$, does not involve $\theta$. So $\varphi(T)$ is computable from the data: it is a statistic. By the tower property, $E_\theta[\varphi(T)]=E_\theta[W]=\psi(\theta)$. By the law of total variance (§1.4), $$\operatorname{Var}_\theta(W)=\operatorname{Var}_\theta\left(E[W\mid T]\right)+E_\theta\left[\operatorname{Var}(W\mid T)\right]\ge\operatorname{Var}_\theta(\varphi(T)),$$ with equality iff $\operatorname{Var}(W\mid T)=0$ a.s., i.e. $W$ is a function of $T$. ∎
Proof. Let $W$ be any unbiased estimator of $\psi(\theta)$ with finite variance, and let $\varphi(T)=E[W\mid T]$. By Rao–Blackwell, $\varphi(T)$ is unbiased and $\operatorname{Var}(\varphi(T))\le\operatorname{Var}(W)$. Both $\varphi(T)$ and $g(T)$ are unbiased functions of $T$, so $E_\theta[\varphi(T)-g(T)]=0$ for all $\theta$, and completeness gives $\varphi(T)=g(T)$ a.s. Hence $\operatorname{Var}_\theta(g(T))\le\operatorname{Var}_\theta(W)$ for every $\theta$. If $W$ is also a UMVUE, equality holds, and the equality case of Rao–Blackwell gives $W=\varphi(T)=g(T)$ a.s. ∎
- Find a complete sufficient statistic $T$: the natural statistic of a full-rank exponential family, or an order statistic in a uniform or shifted model (checked by hand).
- Route A — unbias a function of $T$. Compute $E[T]$, $E[T^2]$, $E[1/T]$, … and combine them to get an unbiased $g(T)$. For example, $E[T(T-1)]$ for Poisson and binomial counts gives $\lambda^2$ and $p^2$.
- Route B — Rao–Blackwellise. Start from any simple unbiased estimator, often an indicator such as $\mathbf 1\{X_1=0\}$ or $\mathbf 1\{X_1\le c\}$, and compute $E[\,\cdot\mid T]$ using the conditional law of $X_1$ given $T$.
- Either way, Lehmann–Scheffé says the result is the UMVUE. If no complete sufficient statistic exists, a UMVUE may fail to exist.
Fisher Information, the Cramér–Rao Bound and Asymptotics of the MLE
A one-parameter family $f(x;\theta)$ is regular if (R1) $\Theta$ is an open interval; (R2) the support $\{x:f(x;\theta)\gt0\}$ does not depend on $\theta$; (R3) $\theta\mapsto\log f(x;\theta)$ is twice differentiable; and (R4) $\int f(x;\theta)\,dx$ (or the sum) may be differentiated twice under the integral sign. The score is $s(x;\theta)=\frac{\partial}{\partial\theta}\log f(x;\theta)$, and the Fisher information in one observation is $$I(\theta)=E_\theta\left[s(X;\theta)^2\right].$$
An unbiased estimator whose variance equals the Cramér–Rao bound below for every $\theta$ is called efficient. The efficiency of an unbiased $T$ is the ratio of the bound to $\operatorname{Var}_\theta(T)$, and the relative efficiency of $T_1$ with respect to $T_2$ is $\operatorname{Var}(T_2)/\operatorname{Var}(T_1)$.
In a regular family:
- $E_\theta[s(X;\theta)]=0$, so $I(\theta)=\operatorname{Var}_\theta(s(X;\theta))$;
- $I(\theta)=-E_\theta\left[\dfrac{\partial^2}{\partial\theta^2}\log f(X;\theta)\right]$;
- a sample of $n$ iid observations has score $S_n=\sum_is(X_i;\theta)$ and information $I_n(\theta)=nI(\theta)$;
- if $\theta=h(\eta)$ with $h$ differentiable, the information about $\eta$ is $I(h(\eta))\,h'(\eta)^2$.
Proof. (1) Differentiate $\int f\,dx=1$: $0=\int\partial_\theta f\,dx=\int s\,f\,dx=E[s]$. (2) Differentiate again, using $\partial_\theta f=sf$: $0=\int(\partial_\theta s)f\,dx+\int s^2f\,dx$. (3) The score of the sample is the sum of $n$ iid mean-zero scores, so its variance is $nI(\theta)$. (4) By the chain rule the score in $\eta$ is $s\cdot h'(\eta)$. ∎
Let the family be regular with $0\lt I(\theta)\lt\infty$, and let $T$ be a statistic with $E_\theta[T]=\psi(\theta)$ and $\operatorname{Var}_\theta(T)\lt\infty$. Suppose $\int T(x)f_n(x;\theta)\,dx$ can be differentiated under the integral sign, where $f_n$ is the joint density. Then $$\operatorname{Var}_\theta(T)\ \ge\ \frac{\psi'(\theta)^2}{nI(\theta)}\qquad\text{for all }\theta.$$ Equality holds for all $\theta$ iff $S_n=a(\theta)\left(T-\psi(\theta)\right)$ a.s. for some function $a$. This happens exactly when the family is a one-parameter exponential family with natural statistic $\sum T_1(X_i)$ and $T$ is an affine function of it.
For a biased estimator with bias $b(\theta)$, the same argument gives $\operatorname{MSE}_\theta(T)\ge\dfrac{(1+b'(\theta))^2}{nI(\theta)}+b(\theta)^2$.
Proof. Differentiating $\psi(\theta)=\int T(x)f_n(x;\theta)\,dx$ under the integral sign gives $\psi'(\theta)=\int T\,S_n\,f_n\,dx=E_\theta[TS_n]$. Since $E_\theta[S_n]=0$, this is $\operatorname{Cov}_\theta(T,S_n)$. By the Cauchy–Schwarz inequality (§2.2), $$\psi'(\theta)^2=\operatorname{Cov}(T,S_n)^2\le\operatorname{Var}(T)\operatorname{Var}(S_n)=\operatorname{Var}(T)\cdot nI(\theta).$$ Equality in Cauchy–Schwarz holds iff $S_n$ and $T-\psi(\theta)$ are linearly dependent, i.e. $S_n=a(\theta)(T-\psi(\theta))$. Integrating this identity in $\theta$ gives $\log f_n=A_1(\theta)T+A_2(\theta)+c(x)$, an exponential family in $T$. ∎
The bound is not always attained. In a regular model, an efficient estimator exists only for $\psi(\theta)=E_\theta[T]$ with $T$ the natural statistic of an exponential family (and affine functions of it). So in the exponential model with rate $\lambda$, the mean $1/\lambda$ has an efficient estimator ($\bar X$), but $\lambda$ itself does not. The UMVUE $(n-1)/\sum X_i$ has variance $\frac{\lambda^2}{n-2}\gt\frac{\lambda^2}{n}$. A UMVUE need not be efficient. What is true is that the MLE attains the bound asymptotically.
Suppose the family is regular and identifiable ($\theta\ne\theta'$ implies $f(\cdot;\theta)\ne f(\cdot;\theta')$), $0\lt I(\theta)\lt\infty$, and the third derivative satisfies $\left|\frac{\partial^3}{\partial\theta^3}\log f(x;\theta)\right|\le M(x)$ near the true $\theta_0$, with $E_{\theta_0}[M(X)]\lt\infty$. Then, with probability tending to $1$, the likelihood equation has a root $\hat\theta_n$ that is consistent for $\theta_0$, and $$\sqrt n\left(\hat\theta_n-\theta_0\right)\xrightarrow{d}N\left(0,\ \frac{1}{I(\theta_0)}\right).$$ By the delta method, $\sqrt n\left(g(\hat\theta_n)-g(\theta_0)\right)\xrightarrow{d}N\left(0,\frac{g'(\theta_0)^2}{I(\theta_0)}\right)$ for differentiable $g$. The MLE is asymptotically efficient.
Proof (sketch). Consistency. By the strong law, $\frac1n\left[\ell_n(\theta)-\ell_n(\theta_0)\right]\to E_{\theta_0}\log\frac{f(X;\theta)}{f(X;\theta_0)}$. By Jensen's inequality and identifiability, this limit is strictly negative for $\theta\ne\theta_0$, because $E\log Y\lt\log EY=\log1=0$ for the likelihood ratio $Y$. So for small $\delta\gt0$, with probability tending to $1$, $\ell_n(\theta_0\pm\delta)\lt\ell_n(\theta_0)$, and $\ell_n$ has a local maximum, hence a root of $\ell_n'$, in $(\theta_0-\delta,\theta_0+\delta)$.
Normality. Expand the score about $\theta_0$: $$0=\ell_n'(\hat\theta)=\ell_n'(\theta_0)+(\hat\theta-\theta_0)\ell_n''(\theta_0)+\tfrac12(\hat\theta-\theta_0)^2\ell_n'''(\theta^*)$$ for some $\theta^*$ between $\hat\theta$ and $\theta_0$. Rearranging, $$\sqrt n(\hat\theta-\theta_0)=\frac{n^{-1/2}\ell_n'(\theta_0)}{-n^{-1}\ell_n''(\theta_0)-\tfrac12(\hat\theta-\theta_0)\,n^{-1}\ell_n'''(\theta^*)}.$$ The numerator is a normalised sum of iid scores with mean $0$ and variance $I(\theta_0)$, so it converges to $N(0,I(\theta_0))$ by the CLT. In the denominator, $-n^{-1}\ell_n''(\theta_0)\to I(\theta_0)$ by the WLLN and Theorem 4.21(2). The last term is bounded by $|\hat\theta-\theta_0|\cdot\frac1n\sum M(X_i)\xrightarrow{P}0$. Slutsky's theorem gives the limit $N(0,I/I^2)=N(0,1/I)$. ∎
In practice the theorem gives the standard error $1/\sqrt{nI(\hat\theta)}$, or $1/\sqrt{-\ell_n''(\hat\theta)}$ using the observed information. This is the width of the purple bell in Interactive 1, and it is the basis of the large-sample intervals of §4.3 and the likelihood ratio tests of §5.1.
- The MLE need not be unbiased: $\hat\sigma^2=SS/n$, $X_{(n)}$ for $U(0,\theta)$ and $1/\bar X$ for an exponential rate are all biased. It need not be unique ($U(\theta,\theta+1)$, the Laplace median for even $n$), it need not solve $\ell'=0$ (uniform, shifted exponential), and it need not exist when the supremum is not attained in $\Theta$ (for example, $\operatorname{Bin}(n,p)$ with $\Theta=(0,1)$ and all trials failures).
- Moment estimators can be impossible: $2\bar X$ can be smaller than $X_{(n)}$ for $U(0,\theta)$, and moment estimators of variances can be negative.
- Unbiased estimators may not exist ($1/p$ for binomial data) or may be absurd ($(-1)^X$ for $e^{-2\lambda}$). Unbiasedness is not preserved by nonlinear maps ($E[S]\lt\sigma$), whereas MLEs are invariant.
- Unbiased does not imply consistent ($X_1$), and consistent does not imply asymptotically unbiased (Example 2(c)).
- Minimal sufficient does not imply complete: $N(\theta,\theta^2)$ and $U(\theta,\theta+1)$. Complete sufficient does imply minimal sufficient (Exercise III.6).
- Curved exponential families are not complete: Theorem 4.17 needs the natural parameters to fill an open set.
- A UMVUE need not attain the Cramér–Rao bound: $S^2$ for $\sigma^2$, and $(n-1)/\sum X_i$ for an exponential rate. Efficiency is the exception, confined to exponential families.
- The Cramér–Rao bound needs regularity: for $U(0,\theta)$ and the shifted exponential the support depends on $\theta$, and estimators with variance of order $1/n^2$ exist.
- Biased estimators can beat the bound and the UMVUE in MSE: $SS/(n+1)$ has smaller MSE than $S^2$; $\frac{n+2}{n+1}X_{(n)}$ beats $\frac{n+1}{n}X_{(n)}$.
- Information adds over independent observations ($nI(\theta)$), but it changes under reparametrisation by the factor $h'(\eta)^2$. "The information" always refers to a particular parameter.
- A likelihood may have several local maxima (Cauchy location with small $n$). The root of $\ell'=0$ found by an algorithm need not be the global maximum, and Theorem 4.23 concerns a consistent root.
Exercises
- 1 Let $X_1,\dots,X_n$ have common mean $\mu$ and variance $\sigma^2$ (iid). Show that $\sum a_iX_i$ is unbiased for $\mu$ iff $\sum a_i=1$, and that among these $\bar X$ has the smallest variance.
- 2 For a normal sample with $n=5$, compute the MSEs of $S^2$ and of $\hat\sigma^2=SS/n$ in units of $\sigma^4$.
- 3 For $U(0,\theta)$, compare the variances of $2\bar X$ and $\frac{n+1}{n}X_{(n)}$. For which $n$ is the second strictly better?
- 4 Show that no unbiased estimator of $1/p$ exists for $X\sim\operatorname{Bin}(n,p)$, $0\lt p\lt1$.
- 5 For a single observation $X\sim\operatorname{Poi}(\lambda)$, show that $(-1)^X$ is the only unbiased estimator of $e^{-2\lambda}$, and find its variance.
- 6 Show that $X_{(n)}$ is consistent for $\theta$ in $U(0,\theta)$, and that $X_1$ is unbiased but not consistent for the mean of any non-degenerate population.
- 7 Give an example of a consistent estimator whose MSE tends to $\infty$.
- 8 Applied. Two laboratories measure the same quantity with independent unbiased estimators $T_1,T_2$ of variances $4$ and $9$. Find the unbiased combination $wT_1+(1-w)T_2$ of least variance and its variance.
- 1 Find the MME and the MLE of the rate $\lambda$ for an $\operatorname{Exp}(\lambda)$ sample.
- 2 Find the MMEs of the gamma parameters from $\bar x$ and $\hat\sigma^2$ (Example 3(b)), and explain why they are consistent.
- 3 Verify that the normal MLE of Example 3(a) is a global maximum by showing that the Hessian of $\ell$ at $(\bar x,\hat\sigma^2)$ is negative definite.
- 4 Find the MLE of $\theta$ for $U(-\theta,\theta)$.
- 5 Find the MLE of $\theta$ for $U(\theta,2\theta)$, $\theta\gt0$.
- 6 For the two-parameter exponential density $\frac1\sigma e^{-(x-\mu)/\sigma}$, $x\gt\mu$, find the MLE of $(\mu,\sigma)$.
- 7 For $\operatorname{Exp}(\lambda)$ data with $\bar x=2$, find the MLE of $P(X\gt1)$.
- 8 For a Cauchy location sample with $n=2$ and data $x_1=-a$, $x_2=a$, show that the likelihood has two maxima when $a\gt1$ and one when $a\le1$.
- 9 Applied (genetics). Under Hardy–Weinberg equilibrium the genotypes AA, Aa, aa occur with probabilities $\theta^2$, $2\theta(1-\theta)$, $(1-\theta)^2$. A sample of $100$ people has counts $36$, $48$, $16$. Find the MLE of $\theta$ and its approximate standard error.
- 1 Use the factorisation theorem to find a sufficient statistic for (a) $\operatorname{Gamma}(\alpha,\lambda)$ with both parameters unknown; (b) $U(\theta_1,\theta_2)$; (c) $\operatorname{Geo}(p)$.
- 2 Show that the Rayleigh family $f(x;\theta)=\frac{x}{\theta}e^{-x^2/(2\theta)}$, $x\gt0$, is a full-rank exponential family, find a complete sufficient statistic, and find the UMVUE of $\theta$.
- 3 Show that $\sum X_i$ is complete for a $\operatorname{Ber}(p)$ sample.
- 4 For $f(x;\theta)=e^{-(x-\theta)}$, $x\gt\theta$, show that $X_{(1)}$ is complete sufficient.
- 5 For $N(\theta,\theta)$ with $\theta\gt0$ (mean equal to variance), find a complete sufficient statistic. Is $\sum X_i$ sufficient?
- 6 Prove that a complete sufficient statistic is minimal sufficient, assuming a minimal sufficient statistic exists.
- 7 Use Basu's theorem to show that, for an $\operatorname{Exp}(\lambda)$ sample, $X_1/\sum X_i$ is independent of $\sum X_i$, and find $E[X_1/\sum X_i]$.
- 8 For a $U(0,\theta)$ sample, use Basu's theorem to show that $X_{(1)}/X_{(n)}$ is independent of $X_{(n)}$.
- 1 For a $\operatorname{Ber}(p)$ sample, find the UMVUE of $p^2$.
- 2 For a $\operatorname{Poi}(\lambda)$ sample, find the UMVUE of $\lambda^2$.
- 3 For an $\operatorname{Exp}(\lambda)$ sample with $n\ge3$, find the UMVUE of $\lambda$ and its variance, and compare with the Cramér–Rao bound.
- 4 For a $U(0,\theta)$ sample, find the UMVUE of $\theta^2$.
- 5 For $N(\mu,1)$, find the UMVUEs of $\mu^2$ and of $P(X\le c)=\Phi(c-\mu)$.
- 6 Compute the Fisher information and the Cramér–Rao bound for unbiased estimation of (a) $\mu$ in $N(\mu,\sigma^2)$ with $\sigma$ known; (b) $p$ in $\operatorname{Ber}(p)$; (c) $\lambda$ in $\operatorname{Poi}(\lambda)$. Which natural estimators attain them?
- 7 For the Cauchy location family $\frac{1}{\pi(1+(x-\theta)^2)}$, show that $I(\theta)=\frac12$, and find the asymptotic efficiency of the sample median.
- 8 For $f(x;\theta)=e^{-(x-\theta)}$, $x\gt\theta$, show that $X_{(1)}-\frac1n$ is the UMVUE of $\theta$ with variance $\frac{1}{n^2}$, and explain why this does not contradict the Cramér–Rao inequality.
- 9 Applied. A city records $120$ road accidents in $50$ days, modelled as iid $\operatorname{Poi}(\lambda)$ per day. Find the MLE and the UMVUE of the probability of an accident-free day, and an approximate standard error for the MLE.
E1. CSIR Part B GATE ST Let $X_1,\dots,X_n$ ($n\ge2$) be iid $\operatorname{Poi}(\lambda)$ and $T=\sum X_i$. The UMVUE of $e^{-\lambda}$ is (A) $e^{-T/n}$; (B) $\left(1-\frac1n\right)^T$; (C) $\left(1-\frac1n\right)^{T/n}$; (D) the proportion of the $X_i$ equal to $0$.
E2. CSIR Part B Let $X_1,\dots,X_n$ be iid with density $f(x;\theta)=e^{-(x-\theta)}$ for $x\gt\theta$. The maximum likelihood estimator of $\theta$ is (A) $\bar X$; (B) $X_{(1)}$; (C) $\bar X-1$; (D) $X_{(1)}-\frac1n$.
E3. CSIR Part B Let $X_1,\dots,X_n$ be iid $\operatorname{Ber}(p)$, $0\lt p\lt1$. The Cramér–Rao lower bound for the variance of unbiased estimators of $p(1-p)$ is (A) $\frac{p(1-p)}{n}$; (B) $\frac{(1-2p)^2p(1-p)}{n}$; (C) $\frac{(1-2p)^2}{np(1-p)}$; (D) $\frac{p^2(1-p)^2}{n}$.
E4. CSIR Part C GATE ST Let $X_1,\dots,X_n$ ($n\ge2$) be iid $U(0,\theta)$, $\theta\gt0$. Which of the following are true? (One or more options may be correct.) (A) $X_{(n)}$ is a complete sufficient statistic. (B) $2\bar X$ is the UMVUE of $\theta$. (C) The MLE of $\theta$ is unbiased. (D) The MLE of $\theta^2$ is $X_{(n)}^2$.
E5. CSIR Part C Let $X_1,\dots,X_n$ ($n\ge2$) be iid $N(\mu,\sigma^2)$ with both parameters unknown. Which of the following are true? (A) $S^2$ attains the Cramér–Rao lower bound for unbiased estimators of $\sigma^2$. (B) $S^2$ is the UMVUE of $\sigma^2$. (C) $(\bar X,S^2)$ is a complete sufficient statistic. (D) $\bar X^2$ is an unbiased estimator of $\mu^2$.
E6. CSIR Part C Which of the following statements about estimators are true? (A) If $T_n$ is unbiased for $\theta$ and $\operatorname{Var}(T_n)\to0$, then $T_n$ is consistent. (B) Every consistent estimator is asymptotically unbiased. (C) If $T$ is unbiased for $\theta$, then $T^2$ is unbiased for $\theta^2$. (D) If the MLE is unique, it is a function of every sufficient statistic.
Suggested plan — 3 lectures × 60 min (a fourth for weaker groups).
- Criteria and methods. Definition 4.4 and Theorem 4.13, then Example 1(a) on the board (the unbiased estimator loses on MSE) and Example 2 as a quick poll. Method of moments and likelihood (Definitions 4.5–4.6), invariance (Theorem 4.14), the strategy box, Examples 3 and 4. Project Interactive 1: Bernoulli, then Uniform (the cliff), then Cauchy at $n=3$ to $6$ (several peaks).
- Sufficiency and completeness. The idea of "regenerating the sample"; the factorisation theorem with its discrete proof; minimal sufficiency via the ratio criterion; completeness and exponential families (Theorem 4.17 as a recipe); Basu's theorem with the $\bar X\perp S^2$ application. Example 5, with (c) and (d) as the two key non-examples.
- Optimal estimators and information. Rao–Blackwell and Lehmann–Scheffé with full proofs, the UMVUE strategy box, and Example 6. Fisher information, the Cramér–Rao proof via Cauchy–Schwarz, when equality holds, and the asymptotic normality of the MLE. Example 7, then project Interactive 2 cycling through the six estimators. End with E4–E6.
Discussion questions. (1) Would you rather have an unbiased estimator or one with smaller MSE? Does the answer depend on whether many estimates will be averaged later? (2) Why can the variance of an estimator for $U(0,\theta)$ fall below the Cramér–Rao "bound", and what does this say about information in non-regular problems? (3) The UMVUE of $e^{-\lambda}$ and the MLE $e^{-\bar X}$ differ for small $n$. Which would you report to a public-health office, and why?
Misconceptions to watch for. Differentiating a likelihood that has an indicator; assuming MLEs are unbiased or unique; confusing "minimal sufficient" with "complete"; applying the Cramér–Rao bound to uniform or shifted models; forgetting $\psi'(\theta)^2$ in the bound for a function of $\theta$; thinking a UMVUE must attain the bound; treating unbiasedness as preserved under nonlinear maps.
Suggested homework. I(2, 4, 7, 8), II(4, 5, 6, 8, 9), III(2, 4, 5, 7), IV(1, 3, 5, 7, 9); Exam corner E1, E3, E5.
Research question. How many fish are in the lake? Compare the maximum likelihood estimator of a population size from a capture–recapture experiment with a slightly modified estimator, and find out which one ecologists should use.
Why it matters. Fisheries agencies, wildlife services and conservation projects around the world estimate population sizes by marking $M$ animals, releasing them, and later recording how many of a second catch of $n$ are marked. The same mathematics is used to estimate the number of people missed by a census, the size of hidden populations in epidemiology, and the number of undiscovered bugs in software.
What to do.
- If the population has $N$ animals, the number $m$ of marked animals in the second catch is hypergeometric (§2.1). Show that the likelihood ratio is $\frac{L(N)}{L(N-1)}=\frac{(N-M)(N-n)}{N(N-M-n+m)}$, and deduce that the MLE is the integer part of $Mn/m$ (the Lincoln–Petersen estimator). When $Mn/m$ is an integer there are two MLEs: explain why.
- Take $M=100$, $n=80$, $m=16$. Compute the MLE and Chapman's estimator $\frac{(M+1)(n+1)}{m+1}-1$.
- Explain why the Lincoln–Petersen estimator has infinite expectation for every $N$, and so no finite bias or MSE. (Hint: $P(m=0)\gt0$.)
- For $N=500$, compute the exact mean and standard deviation of Chapman's estimator by summing over the hypergeometric law, and the mean of the Lincoln–Petersen estimator conditional on $m\gt0$.
Tools: Python with scipy.stats.hypergeom, or a spreadsheet.
Hand in: Your derivation of the MLE, a table comparing the two estimators (mean, bias, SD) at $N=500$, and a recommendation in three sentences.
Going further. Read about how capture–recapture extends to several capture occasions (the Schnabel and Jolly–Seber models) and to estimating the undercount of a national census by a post-enumeration survey.
🧪 Self-check quiz
Q1. For an iid sample from $U(0,\theta)$, the maximum likelihood estimator of $\theta$ is
Q2. For an iid $\operatorname{Poi}(\lambda)$ sample, a sufficient statistic is
Q3. Conditioning an unbiased estimator on a sufficient statistic (Rao–Blackwell)
Q4. The Cramér–Rao lower bound for unbiased estimators of $\lambda$ from $n$ iid $\operatorname{Poi}(\lambda)$ observations is
Q5. For which family is the minimal sufficient statistic not complete?