4.3 Confidence Intervals and Elementary Bayesian Inference
How to report the uncertainty of an estimate: confidence intervals built from pivotal quantities for normal, exponential, uniform and binomial models, exact and large-sample, and what their coverage does and does not mean; then the Bayesian alternative, with priors, posteriors, conjugate families, Bayes estimators under squared and absolute loss, and credible intervals.
An estimate on its own is half an answer. A laboratory that reports "the mean tablet weight is $502.3$ mg" has said nothing about whether the true mean could be $499$ mg, which would put the batch out of specification, or whether it is certainly within $0.1$ mg of $502.3$. What decision makers need is a range of plausible values together with a statement of how reliable the range is. Statistics offers two answers. The frequentist answer is the confidence interval: a rule that turns the data into an interval, designed so that the rule captures the true parameter in a stated proportion, say $95\%$, of all samples. The Bayesian answer treats the parameter itself as uncertain, describes that uncertainty by a prior distribution, updates it by Bayes' theorem into a posterior distribution, and reports a credible interval that contains the parameter with posterior probability $95\%$.
This section develops both. The central tool for confidence intervals is the pivotal quantity, a function of the data and the parameter whose distribution is completely known. With the sampling distributions of §4.1 it gives the exact $z$, $t$, $\chi^2$ and $F$ intervals for normal samples, intervals for exponential and uniform models, and, through Slutsky's theorem and the asymptotics of the MLE (§4.2), large-sample intervals for proportions and other parameters. We prove that equal tails give the shortest interval for unimodal symmetric pivots and see why they do not for skewed ones. On the Bayesian side we derive the three conjugate families (beta–binomial, gamma–Poisson and normal–normal), prove that the Bayes estimator is the posterior mean under squared error and the posterior median under absolute error, and show that highest-posterior-density intervals are the shortest credible intervals. CSIR Part B questions ask for the endpoints or the length of an interval and for posterior means; Part C questions test interpretation, the properties of the posterior and the traps listed in the ⚠️ box.
Confidence Sets and Pivotal Quantities
Throughout, $X=(X_1,\dots,X_n)$ has a distribution $P_\theta$ that depends on an unknown parameter $\theta\in\Theta$; $x$ denotes an observed value of $X$. A function of $X$ alone is a statistic. Probabilities and expectations computed under $P_\theta$ are written $P_\theta$ and $E_\theta$.
Notation for quantiles. We use upper $\alpha$ points: $z_\alpha$, $t_{\nu,\alpha}$, $\chi^2_{\nu,\alpha}$ and $F_{\nu_1,\nu_2,\alpha}$ are the numbers exceeded with probability $\alpha$ by an $N(0,1)$, $t_\nu$, $\chi^2_\nu$ and $F_{\nu_1,\nu_2}$ variable. So $z_{0.025}=1.95996$, $\chi^2_{\nu,1-\alpha/2}$ is the lower $\alpha/2$ point of $\chi^2_\nu$, and $F_{\nu_1,\nu_2,1-\alpha}=1/F_{\nu_2,\nu_1,\alpha}$ (§4.1). All values quoted below were computed with scipy.stats.
Let $g(\theta)$ be a real parameter of interest and $0\lt\alpha\lt1$.
- A confidence set for $g(\theta)$ is a rule $x\mapsto C(x)\subseteq\mathbb{R}$ such that the event $\{g(\theta)\in C(X)\}$ is defined for each $\theta$. Its coverage probability is the function $\theta\mapsto P_\theta\left(g(\theta)\in C(X)\right)$, and its confidence coefficient is $\inf_{\theta\in\Theta}P_\theta\left(g(\theta)\in C(X)\right)$.
- $C$ has confidence level $1-\alpha$ if $P_\theta\left(g(\theta)\in C(X)\right)\ge1-\alpha$ for every $\theta\in\Theta$; it is exact if equality holds for every $\theta$, and conservative if only the inequality is guaranteed.
- If $C(X)=[L(X),U(X)]$ with statistics $L\le U$, it is a confidence interval (CI). $[L(X),\infty)$ is a lower confidence bound and $(-\infty,U(X)]$ an upper confidence bound.
- A sequence $C_n$ (one for each sample size $n$) is an asymptotic (approximate) $1-\alpha$ confidence set if $P_\theta\left(g(\theta)\in C_n(X)\right)\to1-\alpha$ for every fixed $\theta$.
The random object is the interval, not the parameter. Before the data are seen, "$[L(X),U(X)]$ contains $\theta$" is an event with probability at least $1-\alpha$. After the data are seen, the realised interval $[L(x),U(x)]$ is a fixed pair of numbers, and it either contains the fixed number $\theta$ or it does not; the $95\%$ describes the long-run success rate of the procedure. This is the frequentist interpretation, and Interactive 1 shows it at work. Note also that coverage is a function of $\theta$; an asymptotic interval converges to the nominal level at each $\theta$, but possibly not uniformly, and its confidence coefficient for a fixed $n$ may be far below $1-\alpha$ (pitfall 2).
Let $Q(X,\theta)$ be a pivot, let $G$ be its common cdf, and let $a\lt b$ be constants with $P(a\le Q\le b)\ge1-\alpha$ under $G$. Then
$$C(X)=\{\theta\in\Theta:\ a\le Q(X,\theta)\le b\}$$is a confidence set for $\theta$ of level $1-\alpha$, exact when $P(a\le Q\le b)=1-\alpha$. If $\Theta$ is an interval and, for every $x$, $\theta\mapsto Q(x,\theta)$ is continuous and strictly monotone, then $C(x)$ is an interval (the part of $\Theta$ between the solutions of $Q(x,\theta)=a$ and $Q(x,\theta)=b$, where these exist).
Proof. For every $\theta$, the events $\{\theta\in C(X)\}$ and $\{a\le Q(X,\theta)\le b\}$ are the same event, so $$P_\theta\left(\theta\in C(X)\right)=P_\theta\left(a\le Q(X,\theta)\le b\right)=G(b)-G(a^-)\ge1-\alpha,$$ and the middle quantity does not depend on $\theta$ because $Q$ is a pivot. For the second claim fix $x$ and let $h(\theta)=Q(x,\theta)$, a continuous strictly monotone function on the interval $\Theta$. The preimage of the interval $[a,b]$ under a continuous monotone function on an interval is an interval: if $\theta_1\lt\theta_2$ both satisfy $a\le h\le b$ and $\theta_1\lt\theta\lt\theta_2$, then $h(\theta)$ lies between $h(\theta_1)$ and $h(\theta_2)$, hence in $[a,b]$. Its endpoints are where $h$ crosses the levels $a$ and $b$, when it does; otherwise an endpoint of $\Theta$. ∎
Where do pivots come from? Three general sources cover almost every example in the syllabus.
Let $X_i=\mu+\sigma Z_i$ $(i=1,\dots,n)$, where $Z_1,\dots,Z_n$ are iid with a known law, $\mu\in\mathbb{R}$ and $\sigma\gt0$. Let $T$ and $S\gt0$ be statistics that are location–scale equivariant: $T(a+cx)=a+cT(x)$ and $S(a+cx)=cS(x)$ for all $a\in\mathbb{R}$, $c\gt0$. Then
$$\frac{T(X)-\mu}{S(X)},\qquad\frac{S(X)}{\sigma},\qquad\frac{T(X)-\mu}{\sigma}$$are pivots (the first for $\mu$ with $\sigma$ unknown, the second for $\sigma$, the third for $\mu$ when $\sigma$ is known). Examples: $T=\bar X$ or the sample median; $S$ = the sample standard deviation, the range or the mean absolute deviation.
Proof. With $Z=(Z_1,\dots,Z_n)$ we have $X=\mu+\sigma Z$ coordinatewise, so equivariance gives $T(X)=\mu+\sigma T(Z)$ and $S(X)=\sigma S(Z)$. Hence $$\frac{T(X)-\mu}{S(X)}=\frac{T(Z)}{S(Z)},\qquad\frac{S(X)}{\sigma}=S(Z),\qquad\frac{T(X)-\mu}{\sigma}=T(Z),$$ and the right-hand sides are functions of $Z$ alone, whose law is known and free of $(\mu,\sigma)$. ∎
Proof. By the probability integral transform (§1.3), $U_i=F(X_i;\theta)$ are iid $U(0,1)$ when $F$ is continuous, and so are $1-U_i$. For $t\gt0$, $P(-\log U_i\gt t)=P(U_i\lt e^{-t})=e^{-t}$, so $-\log U_i\sim\operatorname{Exp}(1)=\operatorname{Gamma}(1,1)$. By the reproductive property of the gamma law (§2.1), $-\sum\log U_i\sim\operatorname{Gamma}(n,1)$, and multiplying by $2$ divides the rate by $2$: $-2\sum\log U_i\sim\operatorname{Gamma}(n,\tfrac12)=\chi^2_{2n}$. ∎
For discrete data, such as a binomial count, no exact pivot exists, but one can still invert the distribution function of a statistic. The price is that the coverage is only guaranteed to be at least $1-\alpha$.
Let $T$ be a real statistic, write $F_\theta(t)=P_\theta(T\le t)$ and $\bar F_\theta(t)=P_\theta(T\ge t)$, and let $\alpha_1,\alpha_2\ge0$ with $\alpha_1+\alpha_2=\alpha$.
- For every $\theta$ and every $u\in(0,1)$: $P_\theta\left(F_\theta(T)\le u\right)\le u$ and $P_\theta\left(\bar F_\theta(T)\le u\right)\le u$, with equality if $T$ has a continuous distribution.
- Consequently $C(t)=\{\theta:\ F_\theta(t)\gt\alpha_1\ \text{and}\ \bar F_\theta(t)\gt\alpha_2\}$, evaluated at $t=T$, is a confidence set of level $1-\alpha$.
- If the family is stochastically increasing ($F_\theta(t)$ is nonincreasing in $\theta$ for every $t$, hence $\bar F_\theta(t)$ nondecreasing), then $C(t)$ is an interval $(\theta_L(t),\theta_U(t))$ up to endpoints, where $\bar F_{\theta_L}(t)=\alpha_2$ and $F_{\theta_U}(t)=\alpha_1$ (when these equations have solutions in $\Theta$).
Proof. (1) Fix $\theta$ and $u$ and write $F=F_\theta$. The set $A=\{t:F(t)\le u\}$ is an interval unbounded to the left, because $F$ is nondecreasing; let $t^*=\sup A$. If $t^*\in A$, then $P(T\in A)=P(T\le t^*)=F(t^*)\le u$. If $t^*\notin A$, then $A=(-\infty,t^*)$ and $P(T\in A)=P(T\lt t^*)=\lim_{t\uparrow t^*}F(t)\le u$. In both cases $P\left(F(T)\le u\right)=P(T\in A)\le u$. The argument for $\bar F$ is the same with the inequalities reversed. If $T$ is continuous, $F(T)\sim U(0,1)$ by the probability integral transform, which gives equality.
(2) $\theta\notin C(T)$ means $F_\theta(T)\le\alpha_1$ or $\bar F_\theta(T)\le\alpha_2$, so by (1) $$P_\theta\left(\theta\notin C(T)\right)\le P_\theta\left(F_\theta(T)\le\alpha_1\right)+P_\theta\left(\bar F_\theta(T)\le\alpha_2\right)\le\alpha_1+\alpha_2=\alpha.$$ (If $\alpha_1=0$ the first event has probability $P_\theta(F_\theta(T)=0)=0$, and similarly for $\alpha_2$.)
(3) For fixed $t$, $\{\theta:F_\theta(t)\gt\alpha_1\}$ is an initial segment of $\Theta$ because $F_\theta(t)$ decreases in $\theta$, and $\{\theta:\bar F_\theta(t)\gt\alpha_2\}$ is a final segment because $\bar F_\theta(t)$ increases. Their intersection is an interval, with endpoints where the two monotone functions cross the levels $\alpha_2$ and $\alpha_1$. ∎
Applied to $X\sim\operatorname{Bin}(n,p)$ with $\alpha_1=\alpha_2=\alpha/2$, and using the beta–binomial tail identity $P_p(X\ge x)=P\left(\operatorname{Beta}(x,n-x+1)\le p\right)$ of §2.1, Theorem 4.27 gives the Clopper–Pearson ("exact") interval: its lower end $p_L$ is the $\alpha/2$ quantile of $\operatorname{Beta}(x,n-x+1)$ ($p_L=0$ if $x=0$), and its upper end $p_U$ is the $1-\alpha/2$ quantile of $\operatorname{Beta}(x+1,n-x)$ ($p_U=1$ if $x=n$). Its coverage is at least $1-\alpha$ for every $p$; it is conservative, usually by a noticeable margin.
A pivot determines the probability content of $[a,b]$ but not the choice of $a$ and $b$. The natural criterion is length.
Let $f$ be a density that is unimodal: for some $m$, $f$ is nondecreasing on $(-\infty,m]$ and nonincreasing on $[m,\infty)$. Suppose $a^*\le m\le b^*$, $\int_{a^*}^{b^*}f=1-\alpha$ and $f(a^*)=f(b^*)\gt0$. Then every interval $[a,b]$ with $\int_a^bf\ge1-\alpha$ has $b-a\ge b^*-a^*$.
In particular, if the pivot $Q=(\hat\theta-\theta)/c$ (with $c\gt0$ known or a statistic not depending on $a,b$) has a symmetric unimodal density, then the equal-tailed interval $\hat\theta\pm c\,q_{\alpha/2}$ is the shortest interval of the form $[\hat\theta-bc,\hat\theta-ac]$.
Proof. Note first that unimodality gives $f(x)\le f(a^*)$ for $x\le a^*$, $f(x)\le f(b^*)$ for $x\ge b^*$, and $f(x)\ge f(a^*)$ for $a^*\le x\le b^*$ (on this interval $f$ is at least the smaller of its endpoint values). Suppose, for a contradiction, that $\int_a^bf\ge1-\alpha$ but $b-a\lt b^*-a^*$. Write $k=f(a^*)=f(b^*)$. Consider the case $a\le a^*$ (the case $a\gt a^*$ is symmetric); then $b\lt a+(b^*-a^*)\le b^*$.
If $b\le a^*$: the whole of $[a,b]$ lies left of $a^*$, so $\int_a^bf\le k(b-a)\lt k(b^*-a^*)\le\int_{a^*}^{b^*}f=1-\alpha$, a contradiction.
If $a^*\lt b\lt b^*$: $$\int_a^bf=\int_{a^*}^{b^*}f+\int_a^{a^*}f-\int_b^{b^*}f\le(1-\alpha)+k(a^*-a)-k(b^*-b)=(1-\alpha)+k\left[(b-a)-(b^*-a^*)\right]\lt1-\alpha,$$ using $f\le k$ on $[a,a^*]$ and $f\ge k$ on $[b,b^*]$. Again a contradiction. For the last statement, $\{a\le Q\le b\}=\{\hat\theta-bc\le\theta\le\hat\theta-ac\}$ has length $c(b-a)$, and for a symmetric unimodal density $a^*=-q_{\alpha/2}$, $b^*=q_{\alpha/2}$ satisfy the hypotheses. ∎
Two remarks. First, the theorem concerns the length $b-a$ on the scale of the pivot. For a scale pivot such as $Q=(n-1)S^2/\sigma^2$, the interval for $\sigma^2$ is $[(n-1)S^2/b,\ (n-1)S^2/a]$, whose length is proportional to $\frac1a-\frac1b$, and minimising this under $\int_a^bf=1-\alpha$ leads by a Lagrange multiplier to the condition $a^2f(a)=b^2f(b)$ instead (Exercise I.5). Second, for skewed pivots such as $\chi^2$ the equal-tailed choice is not the shortest, though it is the one used in practice because it is easy to tabulate and has equal error rates on both sides.
Intervals for Normal Means and Variances
For normal samples the pivots are exact, and their laws were derived in §4.1: if $X_1,\dots,X_n$ are iid $N(\mu,\sigma^2)$ with $n\ge2$, $\bar X=\frac1n\sum X_i$ and $S^2=\frac{1}{n-1}\sum(X_i-\bar X)^2$, then $\bar X\sim N(\mu,\sigma^2/n)$, $(n-1)S^2/\sigma^2\sim\chi^2_{n-1}$, the two are independent, and therefore $\sqrt n(\bar X-\mu)/S\sim t_{n-1}$.
Let $X_1,\dots,X_n$ be iid $N(\mu,\sigma^2)$, $n\ge2$. Each of the following is an exact $1-\alpha$ confidence interval.
- For $\mu$, $\sigma$ known: $\bar X\pm z_{\alpha/2}\,\dfrac{\sigma}{\sqrt n}$.
- For $\mu$, $\sigma$ unknown: $\bar X\pm t_{n-1,\alpha/2}\,\dfrac{S}{\sqrt n}$.
- For $\sigma^2$, $\mu$ unknown: $\left[\dfrac{(n-1)S^2}{\chi^2_{n-1,\alpha/2}},\ \dfrac{(n-1)S^2}{\chi^2_{n-1,1-\alpha/2}}\right]$; for $\sigma$, take square roots.
- For $\sigma^2$, $\mu$ known: $\left[\dfrac{\sum(X_i-\mu)^2}{\chi^2_{n,\alpha/2}},\ \dfrac{\sum(X_i-\mu)^2}{\chi^2_{n,1-\alpha/2}}\right]$.
The intervals (1) and (2) are the shortest of the form $\bar X-b\,c\le\mu\le\bar X-a\,c$ with $c=\sigma/\sqrt n$, respectively $c=S/\sqrt n$. The length of (1) is fixed; the length of (2) is random with $$E[\text{length}]=2t_{n-1,\alpha/2}\,c_n\frac{\sigma}{\sqrt n},\qquad c_n=E[S]/\sigma=\sqrt{\frac{2}{n-1}}\,\frac{\Gamma(n/2)}{\Gamma((n-1)/2)}\lt1.$$
Proof. (1) $Z=\sqrt n(\bar X-\mu)/\sigma\sim N(0,1)$ is a pivot and $\{|Z|\le z_{\alpha/2}\}=\{|\bar X-\mu|\le z_{\alpha/2}\sigma/\sqrt n\}$. (2) $T=\sqrt n(\bar X-\mu)/S\sim t_{n-1}$ is a pivot (it is also the location–scale pivot of Theorem 4.25), and $\{|T|\le t_{n-1,\alpha/2}\}$ rearranges to the stated interval. (3) $Q=(n-1)S^2/\sigma^2\sim\chi^2_{n-1}$ and $\{\chi^2_{n-1,1-\alpha/2}\le Q\le\chi^2_{n-1,\alpha/2}\}$ has probability $1-\alpha$; solve for $\sigma^2$, which reverses the order because $Q$ is decreasing in $\sigma^2$. Since $\sigma\mapsto\sigma^2$ is increasing on $(0,\infty)$, the event is the same as the one for $\sigma$ with square roots. (4) The $(X_i-\mu)/\sigma$ are iid $N(0,1)$, so $\sum(X_i-\mu)^2/\sigma^2\sim\chi^2_n$. The shortness claims follow from Theorem 4.28, since $N(0,1)$ and $t_{n-1}$ have symmetric unimodal densities. Finally $E[\text{length}]=2t_{n-1,\alpha/2}E[S]/\sqrt n$, and with $V=(n-1)S^2/\sigma^2\sim\chi^2_{n-1}=\operatorname{Gamma}\left(\frac{n-1}{2},\frac12\right)$, $$E\sqrt V=\int_0^\infty\frac{v^{1/2}v^{(n-1)/2-1}e^{-v/2}}{2^{(n-1)/2}\Gamma\left(\frac{n-1}{2}\right)}dv=\frac{2^{n/2}\Gamma(n/2)}{2^{(n-1)/2}\Gamma\left(\frac{n-1}{2}\right)}=\sqrt2\,\frac{\Gamma(n/2)}{\Gamma\left(\frac{n-1}{2}\right)},$$ so $E[S]=\sigma E\sqrt V/\sqrt{n-1}=c_n\sigma$. Jensen's inequality gives $E[S]\lt\sqrt{E[S^2]}=\sigma$, strictly because $S$ is not constant. ∎
Let $X_1,\dots,X_m$ be iid $N(\mu_1,\sigma_1^2)$ and $Y_1,\dots,Y_n$ iid $N(\mu_2,\sigma_2^2)$, independent of the $X$'s, with sample variances $S_1^2$, $S_2^2$. Exact $1-\alpha$ intervals:
- For $\mu_1-\mu_2$, variances known: $\bar X-\bar Y\pm z_{\alpha/2}\sqrt{\dfrac{\sigma_1^2}{m}+\dfrac{\sigma_2^2}{n}}$.
- For $\mu_1-\mu_2$, $\sigma_1=\sigma_2=\sigma$ unknown: $\bar X-\bar Y\pm t_{m+n-2,\alpha/2}\,S_p\sqrt{\dfrac1m+\dfrac1n}$, with the pooled variance $$S_p^2=\frac{(m-1)S_1^2+(n-1)S_2^2}{m+n-2}.$$
- For $\sigma_1^2/\sigma_2^2$: with $R=S_1^2/S_2^2$, $\left[\dfrac{R}{F_{m-1,n-1,\alpha/2}},\ R\,F_{n-1,m-1,\alpha/2}\right]$.
- (Paired data) If instead the observations come in pairs $(X_i,Y_i)$, $i=1,\dots,n$, whose differences $D_i=X_i-Y_i$ are iid $N(\mu_D,\sigma_D^2)$, then $\bar D\pm t_{n-1,\alpha/2}S_D/\sqrt n$ is an exact interval for $\mu_D=\mu_1-\mu_2$.
(Welch, statement.) If $\sigma_1^2\ne\sigma_2^2$ are unknown, there is no exact pivot for $\mu_1-\mu_2$ (the Behrens–Fisher problem). The approximate interval $\bar X-\bar Y\pm t_{\hat\nu,\alpha/2}\sqrt{S_1^2/m+S_2^2/n}$, with $\hat\nu=\dfrac{(S_1^2/m+S_2^2/n)^2}{\frac{(S_1^2/m)^2}{m-1}+\frac{(S_2^2/n)^2}{n-1}}$ degrees of freedom, has coverage close to $1-\alpha$ in practice.
Proof. (1) By independence, $\bar X-\bar Y$ is normal with mean $\mu_1-\mu_2$ and variance $\frac{\sigma_1^2}{m}+\frac{\sigma_2^2}{n}$; standardise. (2) Now $\bar X-\bar Y$ is normal with mean $\mu_1-\mu_2$ and variance $\sigma^2(\frac1m+\frac1n)$. The variables $(m-1)S_1^2/\sigma^2\sim\chi^2_{m-1}$ and $(n-1)S_2^2/\sigma^2\sim\chi^2_{n-1}$ are independent, so their sum $(m+n-2)S_p^2/\sigma^2\sim\chi^2_{m+n-2}$ by the reproductive property. Within each sample the mean is independent of the variance, and the samples are independent, so $(\bar X,\bar Y)$ is independent of $(S_1^2,S_2^2)$. Hence $$\frac{\bar X-\bar Y-(\mu_1-\mu_2)}{S_p\sqrt{1/m+1/n}}=\frac{Z}{\sqrt{\chi^2_{m+n-2}/(m+n-2)}}\sim t_{m+n-2},$$ where the unknown $\sigma$ cancels. (3) $\dfrac{S_1^2/\sigma_1^2}{S_2^2/\sigma_2^2}$ is a ratio of independent $\chi^2$ variables divided by their degrees of freedom, so it is $F_{m-1,n-1}$ (§4.1), and it equals $R\,\sigma_2^2/\sigma_1^2$. Solving $F_{m-1,n-1,1-\alpha/2}\le R\sigma_2^2/\sigma_1^2\le F_{m-1,n-1,\alpha/2}$ for $\sigma_1^2/\sigma_2^2$ and using the reciprocal relation $$\frac{1}{F_{m-1,n-1,1-\alpha/2}}=F_{n-1,m-1,\alpha/2}$$ gives the interval. (4) is Theorem 4.29(2) applied to the $D_i$. ∎
Pairing matters. If each patient is measured before and after a treatment, $X_i$ and $Y_i$ are positively correlated, and $\operatorname{Var}(D_i)=\sigma_1^2+\sigma_2^2-2\operatorname{Cov}(X_i,Y_i)$ can be far smaller than $\sigma_1^2+\sigma_2^2$. Treating such data as two independent samples is not merely inefficient; it uses a wrong variance and hence a wrong interval.
- Model and nuisance parameters. Which parameter do you want, and which other parameters are unknown? The pivot must not contain the unknown nuisance parameters.
- Start from a good statistic: a sufficient statistic or the MLE (§4.2): $\bar X$ and $S^2$ for the normal, $\sum X_i$ for the exponential and Poisson, $X_{(n)}$ for $U(0,\theta)$, $X_{(1)}$ for a shifted exponential.
- Standardise into a pivot: subtract a location, divide by a scale (Theorem 4.25), use $F(X;\theta)$ (Theorem 4.26), or invert the cdf of the statistic (Theorem 4.27, for discrete data).
- Choose the cut-offs: equal tails by default; for a symmetric unimodal pivot this is also the shortest; for a pivot supported on $[0,1]$ with increasing density, put the whole of $\alpha$ in one tail (Example 1(a)).
- Invert and check: solve for the parameter, check the direction of the inequalities, the parameter space (a probability cannot be negative) and the known constraints ($\theta\ge x_{(n)}$ for $U(0,\theta)$).
- Large $n$, no exact pivot? Use an asymptotic pivot (Theorem 4.31) and say that the level is approximate.
Large-Sample Intervals and Intervals for Proportions
Outside the normal model exact pivots are rare. But many estimators are asymptotically normal: the sample mean by the CLT, functions of means by the delta method, and MLEs by their asymptotic theory (§4.2). Each such limit gives an asymptotic pivot.
Let $\hat\theta_n$ be estimators with $\sqrt n(\hat\theta_n-\theta)\xrightarrow{d}N\left(0,\tau^2(\theta)\right)$ and $\tau(\theta)\gt0$ for every $\theta$, and let $\hat\tau_n\xrightarrow{P}\tau(\theta)$ for every $\theta$. Then each of the following is an asymptotic $1-\alpha$ confidence set for $\theta$:
- (Wald) $\hat\theta_n\pm z_{\alpha/2}\,\hat\tau_n/\sqrt n$;
- (Score type) $\left\{\theta:\ \sqrt n\,|\hat\theta_n-\theta|\le z_{\alpha/2}\,\tau(\theta)\right\}$;
- (Variance-stabilised) if $g$ is continuously differentiable and strictly increasing with $g'(\theta)\tau(\theta)=1$: $\left[g^{-1}\left(g(\hat\theta_n)-\frac{z_{\alpha/2}}{\sqrt n}\right),\ g^{-1}\left(g(\hat\theta_n)+\frac{z_{\alpha/2}}{\sqrt n}\right)\right]$.
For the MLE in a regular family (§4.2), $\tau^2(\theta)=1/I(\theta)$ with $I$ the Fisher information of one observation, and one may take $\hat\tau_n^2=1/I(\hat\theta_n)$.
Proof. (1) By Slutsky's theorem (§2.3), $W_n=\sqrt n(\hat\theta_n-\theta)/\hat\tau_n$ satisfies $W_n\xrightarrow{d}N(0,1)$, because $\tau(\theta)/\hat\tau_n\xrightarrow{P}1$. The interval contains $\theta$ iff $|W_n|\le z_{\alpha/2}$, and since $\pm z_{\alpha/2}$ are continuity points of $\Phi$, $$P_\theta(|W_n|\le z_{\alpha/2})\to\Phi(z_{\alpha/2})-\Phi(-z_{\alpha/2})=1-\alpha.$$ (2) For fixed $\theta$, $\tau(\theta)$ is a constant, so $\sqrt n(\hat\theta_n-\theta)/\tau(\theta)\xrightarrow{d}N(0,1)$ directly, and the set contains $\theta$ iff this quantity lies in $[-z_{\alpha/2},z_{\alpha/2}]$. (3) By the delta method, $\sqrt n\left(g(\hat\theta_n)-g(\theta)\right)\xrightarrow{d}N\left(0,g'(\theta)^2\tau^2(\theta)\right)=N(0,1)$, so $g(\hat\theta_n)\pm z_{\alpha/2}/\sqrt n$ is an asymptotic interval for $g(\theta)$, and applying the increasing function $g^{-1}$ preserves the event. ∎
All are asymptotic $1-\alpha$ intervals; $z=z_{\alpha/2}$.
- (Any mean) iid $X_i$ with mean $\mu$ and finite variance $\sigma^2\gt0$: $\bar X\pm z\,S/\sqrt n$. For two independent samples: $\bar X-\bar Y\pm z\sqrt{S_1^2/m+S_2^2/n}$.
- (Proportion) $X\sim\operatorname{Bin}(n,p)$, $\hat p=X/n$, $0\lt p\lt1$. Wald: $\hat p\pm z\sqrt{\hat p(1-\hat p)/n}$. Wilson (score): $$\frac{\hat p+\frac{z^2}{2n}\pm z\sqrt{\frac{\hat p(1-\hat p)}{n}+\frac{z^2}{4n^2}}}{1+\frac{z^2}{n}}.$$
- (Difference of proportions) independent $X\sim\operatorname{Bin}(m,p_1)$, $Y\sim\operatorname{Bin}(n,p_2)$: $\hat p_1-\hat p_2\pm z\sqrt{\dfrac{\hat p_1(1-\hat p_1)}{m}+\dfrac{\hat p_2(1-\hat p_2)}{n}}$.
- (Poisson mean) iid $\operatorname{Poi}(\lambda)$: Wald $\bar X\pm z\sqrt{\bar X/n}$; variance-stabilised $\left(\sqrt{\bar X}\pm\frac{z}{2\sqrt n}\right)^2$ (with the lower end replaced by $0$ if $\sqrt{\bar X}\lt z/(2\sqrt n)$).
Proof. (1) The CLT gives $\sqrt n(\bar X-\mu)\xrightarrow{d}N(0,\sigma^2)$ and the strong law gives $S\to\sigma$ a.s. (§2.3); apply Theorem 4.31(1). For two samples, $\bar X-\bar Y$ is a difference of independent asymptotically normal variables, and its standardised version converges to $N(0,1)$ (by characteristic functions) as $m,n\to\infty$; again replace the variances by consistent estimates. (2) $\hat p$ is the mean of $n$ Bernoulli variables, with $\tau^2(p)=p(1-p)$ and $\hat\tau^2=\hat p(1-\hat p)\to p(1-p)$; this is the Wald interval. For Wilson, apply Theorem 4.31(2): $(\hat p-p)^2\le z^2p(1-p)/n$ is the quadratic inequality $$\left(1+\tfrac{z^2}{n}\right)p^2-\left(2\hat p+\tfrac{z^2}{n}\right)p+\hat p^2\le0,$$ whose discriminant is $\left(2\hat p+\frac{z^2}{n}\right)^2-4\left(1+\frac{z^2}{n}\right)\hat p^2=\frac{4z^2}{n}\hat p(1-\hat p)+\frac{z^4}{n^2}$. The set of $p$ between the two roots is the stated interval. (3) is (1) for two Bernoulli samples. (4) $\tau^2(\lambda)=\lambda$, so $g(\lambda)=2\sqrt\lambda$ has $g'(\lambda)\tau(\lambda)=1$; Theorem 4.31(3) gives $2\sqrt\lambda\in2\sqrt{\bar X}\pm z/\sqrt n$. ∎
The Wilson interval is the set of $p$ that are not rejected by the large-sample test of §5.2 that uses the null standard error $\sqrt{p(1-p)/n}$. It always lies in $[0,1]$, it never collapses to a point, and its centre $\frac{\hat p+z^2/(2n)}{1+z^2/n}$ is pulled from $\hat p$ towards $\frac12$. For $95\%$ the centre is close to $\frac{x+2}{n+4}$, which explains the popular Agresti–Coull rule: add two successes and two failures, then use the Wald formula. The Wald interval, by contrast, behaves badly for small $n$ or $p$ near $0$ or $1$ (pitfall 2 and Interactive 1).
Bayesian Inference: Priors, Posteriors and Conjugate Families
The frequentist treats $\theta$ as a fixed unknown constant and makes probability statements only about the data. The Bayesian treats the unknown $\theta$ as the value of a random variable $\Theta$ whose distribution, the prior, describes what is believed about $\theta$ before the data are seen: from earlier studies, expert knowledge, or deliberately vague. Bayes' theorem (§1.2), in its version for densities, then turns the prior and the data into the posterior distribution, which contains everything the analysis has to say about $\theta$.
A Bayesian model consists of a prior density (or pmf) $\pi(\theta)$ on $\Theta$ and, for each $\theta$, the density (or pmf) $f(x\mid\theta)$ of the data given $\Theta=\theta$. Write $L(\theta;x)=f(x\mid\theta)$ for the likelihood.
- The marginal (prior predictive) density of the data is $m(x)=\int_\Theta f(x\mid\theta)\pi(\theta)\,d\theta$ (a sum if $\Theta$ is discrete).
- For $x$ with $0\lt m(x)\lt\infty$, the posterior density is $$\pi(\theta\mid x)=\frac{f(x\mid\theta)\,\pi(\theta)}{m(x)}.$$
- If a future observation $Y$ has density $f(y\mid\theta)$ and is conditionally independent of $X$ given $\Theta$, its posterior predictive density is $f(y\mid x)=\int_\Theta f(y\mid\theta)\,\pi(\theta\mid x)\,d\theta$.
- A nonnegative $\pi$ with $\int_\Theta\pi=\infty$ is an improper prior; it may still be used formally if $\int_\Theta f(x\mid\theta)\pi(\theta)\,d\theta\lt\infty$, and the resulting posterior is then a proper density.
- $\pi(\theta\mid x)$ is the conditional density of $\Theta$ given $X=x$, and $\pi(\theta\mid x)\propto L(\theta;x)\,\pi(\theta)$ as a function of $\theta$: posterior $\propto$ likelihood $\times$ prior.
- If $T$ is a sufficient statistic for $\theta$, then $\pi(\theta\mid x)$ depends on $x$ only through $T(x)$.
- (Sequential updating) If $x=(x',x'')$ where $X'$ and $X''$ are conditionally independent given $\Theta$, then $\pi(\theta\mid x)\propto f(x''\mid\theta)\,\pi(\theta\mid x')$: the posterior after $x'$ is the prior for $x''$.
- Multiplying the prior or the likelihood by a positive constant (one not depending on $\theta$) leaves the posterior unchanged.
Proof. (1) The pair $(\Theta,X)$ has joint density $\pi(\theta)f(x\mid\theta)$, so the marginal density of $X$ is $m(x)$ and the conditional density of $\Theta$ given $X=x$ is the joint density divided by $m(x)$ (§1.4). The factor $1/m(x)$ does not depend on $\theta$. (2) By the factorisation theorem (§4.2), $f(x\mid\theta)=g\left(T(x),\theta\right)h(x)$. For $h(x)\gt0$, $$\pi(\theta\mid x)=\frac{g(T(x),\theta)h(x)\pi(\theta)}{\int g(T(x),\theta')h(x)\pi(\theta')\,d\theta'}=\frac{g(T(x),\theta)\pi(\theta)}{\int g(T(x),\theta')\pi(\theta')\,d\theta'},$$ which involves $x$ only through $T(x)$. (3) $f(x\mid\theta)=f(x'\mid\theta)f(x''\mid\theta)$, so $$\pi(\theta\mid x)\propto f(x''\mid\theta)\left[f(x'\mid\theta)\pi(\theta)\right]\propto f(x''\mid\theta)\,\pi(\theta\mid x').$$ (4) The constant cancels between numerator and $m(x)$. ∎
In practice one computes a posterior by writing down likelihood $\times$ prior, dropping every factor that does not involve $\theta$, and recognising the kernel of a known density. This works smoothly when prior and posterior belong to the same family.
- (Beta–binomial) Let $X_1,\dots,X_n\mid p$ be iid $\operatorname{Ber}(p)$, $x=\sum x_i$, and let $p\sim\operatorname{Beta}(a,b)$. Then $p\mid x\sim\operatorname{Beta}(a+x,\ b+n-x)$, $$E[p\mid x]=\frac{a+x}{a+b+n}=w\,\frac xn+(1-w)\,\frac{a}{a+b},\qquad w=\frac{n}{a+b+n},$$ and the predictive probability of a success at trial $n+1$ is $P(X_{n+1}=1\mid x)=E[p\mid x]$. The same holds if only $X=\sum X_i\sim\operatorname{Bin}(n,p)$ is observed.
- (Gamma–Poisson) Let $X_1,\dots,X_n\mid\lambda$ be iid $\operatorname{Poi}(\lambda)$ and $\lambda\sim\operatorname{Gamma}(\alpha,\beta)$ (rate $\beta$). Then $\lambda\mid x\sim\operatorname{Gamma}\left(\alpha+\sum x_i,\ \beta+n\right)$, $$E[\lambda\mid x]=\frac{\alpha+\sum x_i}{\beta+n}=w\,\bar x+(1-w)\frac\alpha\beta,\qquad w=\frac{n}{\beta+n}.$$ A new count $Y$ has the negative binomial predictive law $$P(Y=y\mid x)=\frac{\Gamma(\alpha'+y)}{y!\,\Gamma(\alpha')}\left(\frac{\beta'}{\beta'+1}\right)^{\alpha'}\left(\frac{1}{\beta'+1}\right)^y,$$ where $\alpha'=\alpha+\sum x_i$ and $\beta'=\beta+n$. (For exponential data $X_i\mid\lambda\sim\operatorname{Exp}(\lambda)$, the gamma prior is also conjugate: $\lambda\mid x\sim\operatorname{Gamma}(\alpha+n,\ \beta+\sum x_i)$.)
- (Normal–normal) Let $X_1,\dots,X_n\mid\theta$ be iid $N(\theta,\sigma^2)$ with $\sigma^2$ known, and $\theta\sim N(\mu_0,\tau_0^2)$. Then $\theta\mid x\sim N(\mu_n,\tau_n^2)$ with $$\frac{1}{\tau_n^2}=\frac{1}{\tau_0^2}+\frac{n}{\sigma^2},\qquad\mu_n=\tau_n^2\left(\frac{\mu_0}{\tau_0^2}+\frac{n\bar x}{\sigma^2}\right)=w\,\bar x+(1-w)\,\mu_0,\quad w=\frac{n/\sigma^2}{n/\sigma^2+1/\tau_0^2}.$$ Precisions (reciprocal variances) add, and the posterior mean is the precision-weighted average of the prior mean and the sample mean. The predictive law of a new observation is $N(\mu_n,\ \sigma^2+\tau_n^2)$.
Proof. (1) Up to factors free of $p$, $$\pi(p\mid x)\propto p^{x}(1-p)^{n-x}\cdot p^{a-1}(1-p)^{b-1}=p^{a+x-1}(1-p)^{b+n-x-1},\qquad0\lt p\lt1,$$ the kernel of $\operatorname{Beta}(a+x,b+n-x)$, whose mean is $\frac{a+x}{a+b+n}$ (§2.1). The weighted-average form follows from $a+x=n\cdot\frac xn+(a+b)\cdot\frac{a}{a+b}$. For the predictive probability, $P(X_{n+1}=1\mid x)=\int_0^1p\,\pi(p\mid x)\,dp=E[p\mid x]$, by Definition 4.15(3). The binomial likelihood differs from the Bernoulli one by the factor $\binom nx$, which cancels (Theorem 4.33(4)).
(2) Up to factors free of $\lambda$, $$\prod_i\frac{e^{-\lambda}\lambda^{x_i}}{x_i!}\cdot\lambda^{\alpha-1}e^{-\beta\lambda}\propto\lambda^{\alpha+\sum x_i-1}e^{-(\beta+n)\lambda},$$ the $\operatorname{Gamma}(\alpha',\beta')$ kernel, with mean $\alpha'/\beta'$; and $\alpha+\sum x_i=n\bar x+\beta\cdot\frac{\alpha}{\beta}$. The predictive pmf is $$\int_0^\infty\frac{e^{-\lambda}\lambda^y}{y!}\cdot\frac{\beta'^{\alpha'}\lambda^{\alpha'-1}e^{-\beta'\lambda}}{\Gamma(\alpha')}\,d\lambda=\frac{\beta'^{\alpha'}}{y!\,\Gamma(\alpha')}\cdot\frac{\Gamma(\alpha'+y)}{(\beta'+1)^{\alpha'+y}},$$ which is the stated negative binomial pmf. For exponential data, $\prod\lambda e^{-\lambda x_i}\cdot\lambda^{\alpha-1}e^{-\beta\lambda}=\lambda^{\alpha+n-1}e^{-(\beta+\sum x_i)\lambda}$.
(3) Using $\sum(x_i-\theta)^2=\sum(x_i-\bar x)^2+n(\bar x-\theta)^2$ and dropping terms free of $\theta$, $$\log\pi(\theta\mid x)=-\frac{n(\theta-\bar x)^2}{2\sigma^2}-\frac{(\theta-\mu_0)^2}{2\tau_0^2}+c=-\frac12\left[\left(\frac{n}{\sigma^2}+\frac1{\tau_0^2}\right)\theta^2-2\theta\left(\frac{n\bar x}{\sigma^2}+\frac{\mu_0}{\tau_0^2}\right)\right]+c'.$$ Completing the square, this is $-\frac{(\theta-\mu_n)^2}{2\tau_n^2}+c''$ with $\tau_n$ and $\mu_n$ as stated, the log-kernel of $N(\mu_n,\tau_n^2)$. For the predictive law, $Y=\theta+\varepsilon$ with $\theta\mid x\sim N(\mu_n,\tau_n^2)$ and $\varepsilon\sim N(0,\sigma^2)$ independent of $\theta$ given the data, so $Y\mid x\sim N(\mu_n,\tau_n^2+\sigma^2)$. ∎
The weights $w$ have a common interpretation. In the beta–binomial case the prior acts like $a+b$ earlier trials containing $a$ successes; in the gamma–Poisson case like $\beta$ earlier periods with $\alpha$ events in total; in the normal case like $\sigma^2/\tau_0^2$ earlier observations. As $n\to\infty$ with the prior fixed, $w\to1$ and the posterior mean approaches the MLE: the data overwhelm the prior. Actuaries call $w$ the credibility factor.
Vague and improper priors. Letting $\tau_0\to\infty$ in the normal–normal model gives the flat improper prior $\pi(\theta)\propto1$ and the posterior $N(\bar x,\sigma^2/n)$. The uniform prior $\operatorname{Beta}(1,1)$ for $p$ gives the posterior mean $\frac{x+1}{n+2}$ (Laplace's rule of succession). But "flat" depends on the parametrisation: if $p\sim U(0,1)$, the log-odds $\ell=\log\frac{p}{1-p}$ has the logistic density $\frac{e^\ell}{(1+e^\ell)^2}$, which is far from flat. Jeffreys' prior $\pi(\theta)\propto\sqrt{I(\theta)}$, with $I$ the Fisher information, is invariant under reparametrisation; for the Bernoulli model $I(p)=\frac{1}{p(1-p)}$ and it is $\operatorname{Beta}(\frac12,\frac12)$; for the Poisson model it is the improper $\lambda^{-1/2}$. Improper priors must be used with care: the Haldane prior $\pi(p)\propto\frac{1}{p(1-p)}$ gives a proper posterior only when $0\lt x\lt n$ (Exercise IV.9).
Bayes Estimators and Credible Intervals
A posterior distribution is a complete answer, but often a single number or an interval is wanted. Decision theory says how to choose it: specify the cost of each possible error and minimise the expected cost.
- A loss function $L(\theta,a)\ge0$ is the cost of estimating $\theta$ by $a$. Standard choices: squared error $(\theta-a)^2$, absolute error $|\theta-a|$, and weighted squared error $w(\theta)(\theta-a)^2$.
- The risk of an estimator $\delta(X)$ is $R(\theta,\delta)=E_\theta\left[L(\theta,\delta(X))\right]$; for squared error it is the mean squared error.
- For a prior $\pi$, the Bayes risk of $\delta$ is $r(\pi,\delta)=\int_\Theta R(\theta,\delta)\,\pi(\theta)\,d\theta$, and a Bayes estimator $\delta_\pi$ minimises $r(\pi,\delta)$ over all estimators.
- The posterior expected loss of an action $a$ given $x$ is $\rho(a\mid x)=E\left[L(\Theta,a)\mid X=x\right]$, that is, $\rho(a\mid x)=\int_\Theta L(\theta,a)\,\pi(\theta\mid x)\,d\theta$.
- If, for every $x$, $\delta(x)$ minimises the posterior expected loss $\rho(a\mid x)$ over $a$, and $r(\pi,\delta)\lt\infty$, then $\delta$ is a Bayes estimator.
- Under squared error loss, with $E[\Theta^2\mid x]\lt\infty$, the Bayes estimator is the posterior mean $\delta(x)=E[\Theta\mid X=x]$, and the minimum posterior expected loss is the posterior variance $\operatorname{Var}(\Theta\mid x)$. Under weighted squared error $w(\theta)(\theta-a)^2$ it is $E[w(\Theta)\Theta\mid x]/E[w(\Theta)\mid x]$.
- Under absolute error loss, with $E[|\Theta|\mid x]\lt\infty$, any posterior median is a Bayes estimator.
Proof. (1) By the tower property (§1.4), applied to the joint law of $(\Theta,X)$, $$r(\pi,\delta)=E\left[L(\Theta,\delta(X))\right]=E\Big[E\left[L(\Theta,\delta(X))\mid X\right]\Big]=\int\rho\left(\delta(x)\mid x\right)m(x)\,dx.$$ If $\delta'$ is any other estimator, $\rho(\delta(x)\mid x)\le\rho(\delta'(x)\mid x)$ for every $x$, so $r(\pi,\delta)\le r(\pi,\delta')$.
(2) Write $\mu(x)=E[\Theta\mid x]$. Expanding around $\mu(x)$, the cross term has posterior mean $0$, so $$\rho(a\mid x)=E\left[(\Theta-\mu(x))^2\mid x\right]+\left(\mu(x)-a\right)^2=\operatorname{Var}(\Theta\mid x)+(\mu(x)-a)^2,$$ which is minimised exactly at $a=\mu(x)$. For the weighted loss, $\rho(a\mid x)$ equals $E[w\Theta^2\mid x]-2aE[w\Theta\mid x]+a^2E[w\mid x]$, a quadratic in $a$ with minimum at $a=E[w\Theta\mid x]/E[w\mid x]$.
(3) Let $F$ be the posterior cdf and $g(a)=E\left[|\Theta-a|\mid x\right]$. For an integrable variable, $E|\Theta-a|=\int_{-\infty}^aF(t)\,dt+\int_a^\infty\left(1-F(t)\right)dt$ (the tail formula of §1.3 applied to $(a-\Theta)^+$ and $(\Theta-a)^+$). Hence, for $a\lt b$, $$g(b)-g(a)=\int_a^bF(t)\,dt-\int_a^b(1-F(t))\,dt=\int_a^b\left(2F(t)-1\right)dt.$$ Let $m$ be a median: $F(m)\ge\frac12$ and $P(\Theta\lt m\mid x)\le\frac12$. For $t\ge m$, $2F(t)-1\ge0$, so $g(b)\ge g(m)$ for every $b\gt m$. For $t\lt m$, $F(t)\le P(\Theta\lt m\mid x)\le\frac12$, so $2F(t)-1\le0$ and $g(m)\le g(a)$ for every $a\lt m$. So $m$ minimises $g$; apply (1). ∎
For the conjugate families of Theorem 4.34, the Bayes estimators under squared error are therefore the shrinkage formulas $\frac{a+x}{a+b+n}$, $\frac{\alpha+\sum x_i}{\beta+n}$ and $\mu_n$. Under absolute error the normal posterior gives the same answer (mean $=$ median), but beta and gamma posteriors are skewed and their medians differ from their means; for $\operatorname{Gamma}(\alpha',\beta')$ with $\alpha'\ge1$ one always has mode $\lt$ median $\lt$ mean. Under the loss "$1$ if $|\theta-a|\gt\varepsilon$, $0$ otherwise", the Bayes estimator tends, as $\varepsilon\to0$, to the posterior mode (the MAP estimate) for smooth unimodal posteriors; with a flat prior the mode is the MLE.
Proof. Compute $E[\delta(X)\Theta]$ in two ways, with expectations over the joint law of $(\Theta,X)$. Conditioning on $X$ and using $E[\Theta\mid X]=\delta(X)$: $E[\delta(X)\Theta]=E\left[\delta(X)E[\Theta\mid X]\right]=E[\delta(X)^2]$. Conditioning on $\Theta$ and using unbiasedness $E[\delta(X)\mid\Theta]=\Theta$: $E[\delta(X)\Theta]=E\left[\Theta E[\delta(X)\mid\Theta]\right]=E[\Theta^2]$. Therefore $$E\left[(\delta(X)-\Theta)^2\right]=E[\delta(X)^2]-2E[\delta(X)\Theta]+E[\Theta^2]=E[\delta(X)\Theta]-2E[\delta(X)\Theta]+E[\delta(X)\Theta]=0.$$ (All the expectations are finite by Cauchy–Schwarz.) ∎
So $\bar X$ is not a Bayes estimator of a normal mean for any proper prior (its Bayes risk $\sigma^2/n$ is positive), and $X/n$ is not Bayes for a binomial $p$ under any proper prior. Both are limits of Bayes estimators ($\tau_0\to\infty$; $a,b\to0$) and are "generalised Bayes" for improper priors. Conversely, every proper-prior posterior mean in the conjugate families is biased: $E_p\left[\frac{a+X}{a+b+n}\right]=\frac{a+np}{a+b+n}\ne p$ unless $p=\frac{a}{a+b}$.
- A set $C(x)\subseteq\Theta$ is a $1-\alpha$ credible set if $P\left(\Theta\in C(x)\mid X=x\right)\ge1-\alpha$.
- The equal-tailed credible interval is $[q_{\alpha/2}(x),q_{1-\alpha/2}(x)]$, between the $\alpha/2$ and $1-\alpha/2$ quantiles of the posterior.
- The highest posterior density (HPD) region is $H(x)=\{\theta:\pi(\theta\mid x)\ge k\}$, with $k\gt0$ chosen so that $P(\Theta\in H(x)\mid x)=1-\alpha$.
- If $H$ is an HPD region with posterior probability $1-\alpha$ and $C$ is any set with $P(\Theta\in C\mid x)\ge1-\alpha$, then $\operatorname{length}(C)\ge\operatorname{length}(H)$.
- If the posterior density is unimodal, $H$ is an interval $[a,b]$ with $\pi(a\mid x)=\pi(b\mid x)$ (unless it reaches an endpoint of $\Theta$); if it is moreover symmetric, the HPD and equal-tailed intervals coincide.
Proof. (1) Write $\pi$ for $\pi(\cdot\mid x)$ and $|A|$ for the length (Lebesgue measure) of $A$. Since $\int_C\pi\ge1-\alpha=\int_H\pi$, subtracting $\int_{C\cap H}\pi$ from both sides gives $\int_{C\setminus H}\pi\ge\int_{H\setminus C}\pi$. On $C\setminus H$ we have $\pi\lt k$, and on $H\setminus C$ we have $\pi\ge k$. Hence $$k\,|C\setminus H|\ge\int_{C\setminus H}\pi\ge\int_{H\setminus C}\pi\ge k\,|H\setminus C|,$$ so $|C\setminus H|\ge|H\setminus C|$ (as $k\gt0$), and adding $|C\cap H|$ gives $|C|\ge|H|$. (2) For a unimodal density, a superlevel set $\{\pi\ge k\}$ is an interval containing the mode, and at an interior endpoint the density equals $k$ by continuity. For a symmetric unimodal density the interval is symmetric about the centre, so it cuts off equal tails. ∎
This is the Bayesian counterpart of Theorem 4.28, and the proof is the same idea as the Neyman–Pearson lemma of §5.1: to collect a fixed amount of probability in the least space, take the points where the density is highest.
Credible versus confidence intervals. A $95\%$ credible interval says "given these data and this prior, $\theta$ lies in $[a,b]$ with probability $0.95$", which is the statement many people wrongly attach to confidence intervals. The price is a prior. The two kinds of interval can agree numerically: with the flat prior in the normal model the posterior is $N(\bar x,\sigma^2/n)$, and the $95\%$ credible interval is exactly $\bar x\pm1.96\sigma/\sqrt n$. More generally, the Bernstein–von Mises theorem (statement only) says that in regular parametric models, for any prior with a continuous positive density at the true $\theta_0$, the posterior is approximately $N\left(\hat\theta_n,\ 1/(nI(\hat\theta_n))\right)$ for large $n$, in total variation, with probability tending to one. So for large samples credible intervals and Wald intervals nearly coincide, and the prior matters only for small or moderate $n$, or when it is very concentrated.
- "$P(2.1\le\mu\le3.4)=0.95$" is wrong for a realised confidence interval. Once computed, the interval is fixed and $\mu$ is fixed; the $95\%$ refers to the procedure over repeated samples. A probability statement about $\mu$ needs a prior: that is a credible interval.
- Asymptotic intervals have pointwise, not uniform, coverage. For each fixed $n$, the Wald interval for a proportion has confidence coefficient $0$: as $p\to0$, $P_p(X=0)=(1-p)^n\to1$, and when $x=0$ the interval is $\{0\}$, which misses $p$ (Exercise III.5).
- Using $z$ with an estimated $\sigma$ in small samples undercovers. $\bar x\pm1.96s/\sqrt n$ has exact coverage $P(|t_{n-1}|\le1.96)$, which is $0.878$ for $n=5$ and $0.918$ for $n=10$.
- Equal tails are not shortest for skewed pivots. For $\chi^2$ pivots and for $X_{(n)}/\theta$ in $U(0,\theta)$, the shortest interval has unequal tails (Examples 1 and 3); for scale parameters the right criterion is $a^2f(a)=b^2f(b)$, not $f(a)=f(b)$.
- The confidence level is an average over samples, not a conditional guarantee. For two observations from $U(\theta-\frac12,\theta+\frac12)$, $[X_{(1)},X_{(2)}]$ is a $50\%$ interval, yet it is certain to cover $\theta$ when the range exceeds $\frac12$ (Example 2).
- Overlapping intervals do not mean "no difference". If $\bar x_1=0$ and $\bar x_2=3$, each with standard error $1$, the individual $95\%$ intervals $[-1.96,1.96]$ and $[1.04,4.96]$ overlap, but the $95\%$ interval for the difference, $3\pm1.96\sqrt2=[0.23,5.77]$, excludes $0$.
- The pooled $t$ interval needs equal variances and independent samples. With unequal variances and unequal sample sizes its coverage can be far from nominal (use Welch); with paired data it uses the wrong variance (use differences).
- The $\chi^2$ interval for $\sigma^2$ is not robust. The $t$ interval for $\mu$ is approximately valid for non-normal data with large $n$ (CLT and Slutsky), but the $\chi^2$ interval for $\sigma^2$ is not: its large-sample coverage depends on the kurtosis, because $\operatorname{Var}(S^2)\approx(\mu_4-\sigma^4)/n$ (§2.3).
- A pivot may not contain nuisance parameters. $\sqrt n(\bar X-\mu)/\sigma$ is useless for $\mu$ when $\sigma$ is unknown; replacing $\sigma$ by $S$ changes the law from $N(0,1)$ to $t_{n-1}$.
- Bayes estimators are biased, and unbiased estimators are not Bayes under squared error with a proper prior (Theorem 4.36). Bias is no defect from the Bayesian point of view.
- The posterior mean, median and mode differ for skewed posteriors, and they answer different loss functions (Theorem 4.35); for $\operatorname{Beta}(3,1)$ they are $0.75$, $0.794$ and $1$.
- An improper prior can give an improper posterior. With the Haldane prior $\pi(p)\propto p^{-1}(1-p)^{-1}$ and $x=0$ successes, the posterior $\propto p^{-1}(1-p)^{n-1}$ is not integrable near $0$.
- "Flat" priors are not invariant. A uniform prior on $p$ is not uniform on the odds or the log-odds; Jeffreys' prior $\propto\sqrt{I(\theta)}$ is the parametrisation-invariant choice.
- Predictive probabilities are not plug-in probabilities. $P(Y=0\mid x)=\left(\frac{\beta'}{\beta'+1}\right)^{\alpha'}$ for the gamma–Poisson model is larger than $e^{-E[\lambda\mid x]}$, by Jensen's inequality (Example 6(b)).
Exercises
- 1 $X_1,\dots,X_n$ are iid with density $\theta x^{\theta-1}$ on $(0,1)$, $\theta\gt0$. Show that $-2\theta\sum\log X_i\sim\chi^2_{2n}$, and find the equal-tailed $95\%$ interval for $\theta$ when $n=10$ and $\sum\log x_i=-6.2$.
- 2 $X_1,\dots,X_n$ are iid $U(\theta,\theta+1)$. (a) Using the pivot $X_{(1)}-\theta$, find the shortest $1-\alpha$ interval of the form $[X_{(1)}-b,\ X_{(1)}-a]$. (b) Show that $[X_{(n)}-1,\ X_{(1)}]$ contains $\theta$ with probability $1$, and find its expected length. Compare the two lengths for $n=10$, $\alpha=0.05$.
- 3 $X_1,\dots,X_n$ are iid with density $e^{-(x-\theta)}$ for $x\gt\theta$. Show that $n(X_{(1)}-\theta)\sim\operatorname{Exp}(1)$ and find the shortest $95\%$ confidence interval for $\theta$ based on this pivot.
- 4 For $N(\mu,\sigma^2)$ data with $\sigma$ known, the interval $\left[\bar X-z_{\alpha/3}\frac{\sigma}{\sqrt n},\ \bar X+z_{2\alpha/3}\frac{\sigma}{\sqrt n}\right]$ also has exact level $1-\alpha$. Verify this, and find its length relative to the equal-tailed interval for $\alpha=0.05$.
- 5 Let $f$ be the $\chi^2_{n-1}$ density. Show that among intervals $\left[(n-1)S^2/b,\ (n-1)S^2/a\right]$ with $\int_a^bf=1-\alpha$, the shortest satisfies $a^2f(a)=b^2f(b)$. Check numerically that the equal-tailed choice for $n=25$, $\alpha=0.05$ does not satisfy this condition.
- 6 For the components of Example 1(b) ($n=10$, total time on test $1240$ hours), find a $95\%$ lower confidence bound for the mean lifetime $\theta$.
- 7 (Rule of three) If $X\sim\operatorname{Bin}(n,p)$ and $x=0$, show that the exact one-sided $1-\alpha$ upper bound for $p$ given by Theorem 4.27 is $1-\alpha^{1/n}$, and that it is at most $-\log\alpha/n$. Evaluate both for $n=50$, $\alpha=0.05$.
- 8 (Exact Poisson interval) For $X\sim\operatorname{Poi}(\lambda)$, use Theorem 4.27 and the identity $P_\lambda(X\ge x)=P\left(\operatorname{Gamma}(x,1)\le\lambda\right)$ $(x\ge1)$ of §2.1 to show that $\left[\frac12\chi^2_{2x,1-\alpha/2},\ \frac12\chi^2_{2x+2,\alpha/2}\right]$ is a confidence interval of level at least $1-\alpha$. Evaluate it for $x=7$, $\alpha=0.05$.
- 1 A sample of $16$ from a normal population has $\bar x=27.3$ and $s=3.6$. Find a $99\%$ confidence interval for $\mu$.
- 2 For the same data, find a $95\%$ confidence interval for $\sigma^2$.
- 3 Blood glucose measurements have known $\sigma=12$ mg/dL. How many patients are needed for a $95\%$ interval for the mean of half-width at most $2$ mg/dL?
- 4 For $n=5$ and $\alpha=0.05$, compute $c_5=E[S]/\sigma$ and the ratio of the expected length of the $t$ interval to the length of the $z$ interval with $\sigma$ known.
- 5 Eight patients' systolic blood pressures are measured before and after a drug. The differences (before − after) have $\bar d=6.25$ and $s_d=4.2$ mmHg. Find a $95\%$ interval for the mean reduction, and explain why the two-sample interval would be inappropriate.
- 6 Independent normal samples with a common variance give $m=8$, $\bar x=15.2$, $s_1=2.1$ and $n=10$, $\bar y=13.1$, $s_2=2.5$. Find the $95\%$ pooled interval for $\mu_1-\mu_2$.
- 7 For the data of part 6, find a $90\%$ interval for $\sigma_1^2/\sigma_2^2$.
- 8 For $n=10$ and $\alpha=0.05$, find the probability that the $t$ interval is shorter than the $z$ interval that uses the true $\sigma$.
Enter a sample size, mean and standard deviation; the lab computes the $t$ interval for $\mu$, the $\chi^2$ interval for $\sigma^2$ and the quantiles used. Use it to check Exercises II.1–2 and Example 3.
- 1 In a survey, $540$ of $1200$ voters favour a candidate. Find the $95\%$ Wald interval for the proportion $p$.
- 2 In a pilot study $x=3$ of $n=25$ seeds fail to germinate. Compute the $95\%$ Wald and Wilson intervals and comment.
- 3 A side effect occurs in $45$ of $200$ patients on drug A and in $30$ of $180$ on drug B. Find a $95\%$ interval for $p_A-p_B$.
- 4 Ten one-minute counts of a radioactive source total $400$. Assuming Poisson counts, find the $95\%$ Wald and variance-stabilised intervals for the rate $\lambda$ per minute, and compare them with the exact interval $[36.18,\ 44.12]$.
- 5 Prove that for every $n$ the confidence coefficient of the Wald interval for a binomial $p$ is $0$.
- 6 Fifty exponential lifetimes have mean $\bar x=3.2$ years. Compare the $95\%$ Wald interval for the mean $\theta$, the interval obtained from $\log\bar X$ by Theorem 4.31(3), and the exact $\chi^2$ interval.
- 7 How large a sample is needed for a $95\%$ margin of error of at most $0.02$ for a proportion (a) with no information about $p$, (b) if it is known that $p\le0.1$?
- 8 Using $\sqrt n(S^2-\sigma^2)\xrightarrow{d}N(0,2\sigma^4)$ for normal data (§2.3), find a variance-stabilising transformation of $S^2$ and an asymptotic $95\%$ interval for $\sigma$ when $n=100$, $s=5$. Compare with the exact interval $[4.390,\ 5.808]$.
- 1 With the uniform prior $\operatorname{Beta}(1,1)$, $9$ successes are observed in $10$ Bernoulli trials. Find the posterior and the predictive probability that trial $11$ is a success. Compare with the plug-in value $\hat p=0.9$.
- 2 $X_1,\dots,X_n\mid\lambda$ are iid $\operatorname{Exp}(\lambda)$ and $\lambda\sim\operatorname{Gamma}(\alpha,\beta)$. Find the posterior, and the Bayes estimators under squared error of $\lambda$ and of the mean lifetime $1/\lambda$.
- 3 $X_1,\dots,X_9\mid\theta$ are iid $N(\theta,9)$, the prior is $N(50,4)$, and $\bar x=53$. Find the posterior and a $95\%$ credible interval for $\theta$.
- 4 With a uniform prior, $2$ successes occur in $2$ trials. Find the Bayes estimates of $p$ under squared and absolute error, and the posterior mode.
- 5 Show that the Bayes risk of the posterior mean under squared error is $E\left[\operatorname{Var}(\Theta\mid X)\right]$. For $X\sim\operatorname{Bin}(n,p)$ with $p\sim U(0,1)$, show that the Bayes risk of $\frac{X+1}{n+2}$ is $\frac{1}{6(n+2)}$, while that of $\frac Xn$ is $\frac{1}{6n}$.
- 6 Prove that in the model $X_i\mid\theta\sim N(\theta,1)$ iid, $\bar X$ is not a Bayes estimator under squared error with respect to any proper prior with finite variance.
- 7 (Asymmetric loss) Let $L(\theta,a)=c_1(\theta-a)$ if $\theta\gt a$ and $c_2(a-\theta)$ if $\theta\le a$, with $c_1,c_2\gt0$. Prove that the Bayes estimator is a $\frac{c_1}{c_1+c_2}$ quantile of the posterior. What does it say for an inventory manager for whom running short costs three times as much as overstocking?
- 8 A posterior is $\operatorname{Exp}(10)$ (rate $10$). Find the $95\%$ HPD interval and the $95\%$ equal-tailed interval, and their lengths.
- 9 With the improper Haldane prior $\pi(p)\propto p^{-1}(1-p)^{-1}$ and $X\sim\operatorname{Bin}(n,p)$, show that the posterior is proper if and only if $0\lt x\lt n$, and that its mean is then the MLE $x/n$.
E1. CSIR Part B $X_1,\dots,X_5$ are iid $U(0,\theta)$, $\theta\gt0$. The constant $c$ for which $[X_{(5)},\ cX_{(5)}]$ is a $95\%$ confidence interval for $\theta$ is (A) $(0.95)^{-1/5}$; (B) $20^{1/5}$; (C) $1.95$; (D) $e^{0.05}$.
E2. CSIR Part B GATE ST The success probability $p$ of a Bernoulli experiment has prior $\operatorname{Beta}(2,3)$. In $10$ independent trials, $7$ successes are observed. The Bayes estimate of $p$ under squared error loss is (A) $0.70$; (B) $0.60$; (C) $8/13$; (D) $0.40$.
E3. CSIR Part B A random sample of size $16$ from $N(\mu,\sigma^2)$, both parameters unknown, has sample standard deviation $s=4$ (divisor $n-1$). The length of the usual $95\%$ confidence interval for $\mu$ is (A) $2\times1.96$; (B) $2\,t_{15,0.025}$; (C) $2\,t_{16,0.025}$; (D) $8\,t_{15,0.025}$.
E4. CSIR Part C Let $X_1,\dots,X_n$ ($n\ge2$) be iid $N(\mu,\sigma^2)$ with both parameters unknown, and let $[L,U]=\bar X\pm t_{n-1,0.025}\,S/\sqrt n$. Which of the following are true? (One or more options may be correct.) (A) $P_{\mu,\sigma}(L\le\mu\le U)=0.95$ for all $\mu$ and $\sigma$. (B) $E[U-L]=2t_{n-1,0.025}\,\sigma/\sqrt n$. (C) For every sample, the $99\%$ $t$ interval contains the $95\%$ $t$ interval. (D) The interval $\bar X\pm1.96\,S/\sqrt n$ also has coverage exactly $0.95$.
E5. CSIR Part C GATE ST Given $\lambda$, $X_1,\dots,X_n$ are iid $\operatorname{Poi}(\lambda)$, and the prior of $\lambda$ is $\operatorname{Gamma}(\alpha,\beta)$ with density proportional to $\lambda^{\alpha-1}e^{-\beta\lambda}$, where $\alpha\ge1$. Which of the following are true? (A) The posterior is $\operatorname{Gamma}\left(\alpha+\sum x_i,\ \beta+n\right)$. (B) The Bayes estimator under squared error is a weighted average of $\bar x$ and $\alpha/\beta$, with weight $n/(n+\beta)$ on $\bar x$. (C) The Bayes estimator under squared error is an unbiased estimator of $\lambda$. (D) The Bayes estimate under absolute error loss is smaller than the Bayes estimate under squared error loss.
E6. CSIR Part C Let $X_1,\dots,X_{10}$ be iid exponential with mean $\theta\gt0$, and $T=\sum X_i$. Which of the following are true? (A) $2T/\theta\sim\chi^2_{20}$. (B) $X_{(1)}/\theta$ is exponential with mean $1$. (C) The equal-tailed interval $\left[2T/\chi^2_{20,0.025},\ 2T/\chi^2_{20,0.975}\right]$ has the smallest length among all $95\%$ intervals of the form $[2T/b,\ 2T/a]$. (D) The expected length of the interval in (C) is smaller than that of the equal-tailed $95\%$ interval built from the pivot $10X_{(1)}/\theta$.
Suggested plan: 3 lectures × 60 min.
- Confidence sets and pivots. Definition 4.13 and the interpretation, then Interactive 1 projected with the $t$ rule (watch the coverage settle at $95\%$ while individual intervals hit and miss). Definition 4.14 and Theorem 4.24; Theorems 4.25 and 4.26 with one line each; Example 1 on the board. Theorem 4.28 with the full proof (draw the density and the two intervals). Example 2 as a class discussion. Theorem 4.27 can be stated and illustrated with Clopper–Pearson, proof set as reading.
- Normal and large-sample intervals. Theorem 4.29 (prove (2) and (3), mention $c_n$), Theorem 4.30 (prove (2) and (3)), the paired case, Example 3 and Example 4(b). Then Theorem 4.31 via Slutsky, Theorem 4.32 with the Wilson derivation, Example 5. Return to Interactive 1 with the Wald rule at $p=0.05$, $n=20$, then the Wilson rule.
- Bayesian inference. Definition 4.15, Theorem 4.33, the three conjugate pairs of Theorem 4.34 (do the normal–normal computation in full), Example 6(a) and Interactive 2 with a sceptical and a uniform prior. Definition 4.17 and Theorem 4.35 (prove (2) and (3)), Theorem 4.36, credible and HPD intervals with Theorem 4.37, Example 7. Finish with E4–E6.
Discussion questions. (1) A newspaper writes: "there is a $95\%$ chance that the true support lies between $42\%$ and $48\%$." Is this a confidence statement or a credible statement, and what would make it correct? (2) Example 2 shows that some samples carry more information than others. Should a reported confidence level depend on the observed sample? (3) Two analysts with different priors analyse the same clinical trial. Under what conditions will their posteriors agree, and how many patients would that take in Example 6(a)?
Misconceptions to watch for. Reading a realised confidence interval as a probability statement about the parameter; using $z$ instead of $t$ with an estimated $\sigma$; forgetting to reverse the inequalities when inverting a decreasing pivot (the $\chi^2$ interval for $\sigma^2$); assuming equal tails give the shortest interval; using the Wald interval for small counts; pooling variances or ignoring pairing without checking the design; believing that the posterior mean should be unbiased; confusing the posterior mean, median and mode; treating a flat prior as "no information" in every parametrisation.
Suggested homework. I(1, 2, 5, 8), II(1, 2, 5, 8), III(2, 4, 5, 6), IV(2, 3, 5, 7, 9); Exam corner E1, E4, E5.
Research question. How well do the standard $95\%$ intervals for a binomial proportion actually cover, as functions of $n$ and $p$? Is the textbook rule "use the Wald interval when $np\ge5$ and $n(1-p)\ge5$" a safe guide?
Why it matters. Proportions are everywhere: the rate of a rare adverse event in a vaccine trial, the share of voters in a poll, the defect rate on a production line, the prevalence of an infection in a district survey. Regulators and newspapers report these with $95\%$ intervals. An interval that claims $95\%$ but covers $88\%$ of the time understates uncertainty, and one that covers $99\%$ wastes sample size. Because the binomial law is discrete, the coverage of every method oscillates with $p$ and $n$ in a surprisingly irregular way.
What to do.
- For $n\in\{10,20,50,100\}$ and a fine grid of $p$ in $(0,1)$, compute the exact coverage $\sum_{x}\binom nxp^x(1-p)^{n-x}\mathbf{1}\{p\in C(x)\}$ of the Wald, Wilson, Agresti–Coull and Clopper–Pearson intervals. No simulation is needed.
- Plot coverage against $p$ for each $n$. Mark where $np\ge5$ and $n(1-p)\ge5$ hold, and check whether the Wald interval behaves there.
- Record the minimum and the average coverage over $p$, and the average length, for each method and each $n$.
- Use Interactive 1 to watch one bad case in repeated sampling, and explain in a paragraph why the coverage jumps as $p$ crosses the endpoints $C(x)$.
Tools: Python (numpy, scipy.stats) or R; Interactive 1 of this section; Theorem 4.27 for Clopper–Pearson.
Hand in: Coverage plots for the four methods, a table of minimum and average coverage and average length, and a one-page recommendation for a pharmacovigilance team that must report rates of rare events from samples of $50$ to $500$ patients.
Going further. Read about the systematic comparisons of binomial intervals in the statistical literature of the early 2000s, which made the Wilson and Agresti–Coull intervals standard; about Jeffreys' interval (the equal-tailed credible interval from the $\operatorname{Beta}(\frac12,\frac12)$ prior), whose frequentist coverage is excellent; and about the general idea of probability matching priors, priors for which credible intervals have nearly the right frequentist coverage.
🧪 Self-check quiz
Q1. A $95\%$ confidence interval for $\mu$ computed from a sample is $[2.1,\ 3.4]$. Which statement is correct?
Q2. For iid $U(0,\theta)$ data, which of the following is a pivotal quantity?
Q3. With a uniform prior on $p$, $3$ successes are observed in $4$ Bernoulli trials. The posterior mean of $p$ is
Q4. Under absolute error loss $|\theta-a|$, the Bayes estimator is
Q5. $X_1,\dots,X_4\mid\theta$ are iid $N(\theta,1)$ and $\theta\sim N(0,1)$. The posterior variance of $\theta$ is