Chapter 4 · Sampling Distributions and Estimation

4.3 Confidence Intervals and Elementary Bayesian Inference

How to report the uncertainty of an estimate: confidence intervals built from pivotal quantities for normal, exponential, uniform and binomial models, exact and large-sample, and what their coverage does and does not mean; then the Bayesian alternative, with priors, posteriors, conjugate families, Bayes estimators under squared and absolute loss, and credible intervals.

An estimate on its own is half an answer. A laboratory that reports "the mean tablet weight is $502.3$ mg" has said nothing about whether the true mean could be $499$ mg, which would put the batch out of specification, or whether it is certainly within $0.1$ mg of $502.3$. What decision makers need is a range of plausible values together with a statement of how reliable the range is. Statistics offers two answers. The frequentist answer is the confidence interval: a rule that turns the data into an interval, designed so that the rule captures the true parameter in a stated proportion, say $95\%$, of all samples. The Bayesian answer treats the parameter itself as uncertain, describes that uncertainty by a prior distribution, updates it by Bayes' theorem into a posterior distribution, and reports a credible interval that contains the parameter with posterior probability $95\%$.

This section develops both. The central tool for confidence intervals is the pivotal quantity, a function of the data and the parameter whose distribution is completely known. With the sampling distributions of §4.1 it gives the exact $z$, $t$, $\chi^2$ and $F$ intervals for normal samples, intervals for exponential and uniform models, and, through Slutsky's theorem and the asymptotics of the MLE (§4.2), large-sample intervals for proportions and other parameters. We prove that equal tails give the shortest interval for unimodal symmetric pivots and see why they do not for skewed ones. On the Bayesian side we derive the three conjugate families (beta–binomial, gamma–Poisson and normal–normal), prove that the Bayes estimator is the posterior mean under squared error and the posterior median under absolute error, and show that highest-posterior-density intervals are the shortest credible intervals. CSIR Part B questions ask for the endpoints or the length of an interval and for posterior means; Part C questions test interpretation, the properties of the posterior and the traps listed in the ⚠️ box.

Confidence Sets and Pivotal Quantities

Throughout, $X=(X_1,\dots,X_n)$ has a distribution $P_\theta$ that depends on an unknown parameter $\theta\in\Theta$; $x$ denotes an observed value of $X$. A function of $X$ alone is a statistic. Probabilities and expectations computed under $P_\theta$ are written $P_\theta$ and $E_\theta$.

Notation for quantiles. We use upper $\alpha$ points: $z_\alpha$, $t_{\nu,\alpha}$, $\chi^2_{\nu,\alpha}$ and $F_{\nu_1,\nu_2,\alpha}$ are the numbers exceeded with probability $\alpha$ by an $N(0,1)$, $t_\nu$, $\chi^2_\nu$ and $F_{\nu_1,\nu_2}$ variable. So $z_{0.025}=1.95996$, $\chi^2_{\nu,1-\alpha/2}$ is the lower $\alpha/2$ point of $\chi^2_\nu$, and $F_{\nu_1,\nu_2,1-\alpha}=1/F_{\nu_2,\nu_1,\alpha}$ (§4.1). All values quoted below were computed with scipy.stats.

📘 Definition 4.13 — Confidence sets, coverage and confidence level

Let $g(\theta)$ be a real parameter of interest and $0\lt\alpha\lt1$.

  1. A confidence set for $g(\theta)$ is a rule $x\mapsto C(x)\subseteq\mathbb{R}$ such that the event $\{g(\theta)\in C(X)\}$ is defined for each $\theta$. Its coverage probability is the function $\theta\mapsto P_\theta\left(g(\theta)\in C(X)\right)$, and its confidence coefficient is $\inf_{\theta\in\Theta}P_\theta\left(g(\theta)\in C(X)\right)$.
  2. $C$ has confidence level $1-\alpha$ if $P_\theta\left(g(\theta)\in C(X)\right)\ge1-\alpha$ for every $\theta\in\Theta$; it is exact if equality holds for every $\theta$, and conservative if only the inequality is guaranteed.
  3. If $C(X)=[L(X),U(X)]$ with statistics $L\le U$, it is a confidence interval (CI). $[L(X),\infty)$ is a lower confidence bound and $(-\infty,U(X)]$ an upper confidence bound.
  4. A sequence $C_n$ (one for each sample size $n$) is an asymptotic (approximate) $1-\alpha$ confidence set if $P_\theta\left(g(\theta)\in C_n(X)\right)\to1-\alpha$ for every fixed $\theta$.

The random object is the interval, not the parameter. Before the data are seen, "$[L(X),U(X)]$ contains $\theta$" is an event with probability at least $1-\alpha$. After the data are seen, the realised interval $[L(x),U(x)]$ is a fixed pair of numbers, and it either contains the fixed number $\theta$ or it does not; the $95\%$ describes the long-run success rate of the procedure. This is the frequentist interpretation, and Interactive 1 shows it at work. Note also that coverage is a function of $\theta$; an asymptotic interval converges to the nominal level at each $\theta$, but possibly not uniformly, and its confidence coefficient for a fixed $n$ may be far below $1-\alpha$ (pitfall 2).

📘 Definition 4.14 — Pivotal quantity
A pivotal quantity (pivot) for $\theta$ is a function $Q(X,\theta)$ of the data and the parameter whose distribution under $P_\theta$ is the same for every $\theta\in\Theta$ and is completely known. An asymptotic pivot is a function $Q_n(X,\theta)$ whose distribution under $P_\theta$ converges to a fixed known law for every $\theta$. A pivot is not a statistic: it involves $\theta$. But once $\theta$ is given it must be computable from the data, so it may not contain other unknown (nuisance) parameters.
📐 Theorem 4.24 — The pivotal method

Let $Q(X,\theta)$ be a pivot, let $G$ be its common cdf, and let $a\lt b$ be constants with $P(a\le Q\le b)\ge1-\alpha$ under $G$. Then

$$C(X)=\{\theta\in\Theta:\ a\le Q(X,\theta)\le b\}$$

is a confidence set for $\theta$ of level $1-\alpha$, exact when $P(a\le Q\le b)=1-\alpha$. If $\Theta$ is an interval and, for every $x$, $\theta\mapsto Q(x,\theta)$ is continuous and strictly monotone, then $C(x)$ is an interval (the part of $\Theta$ between the solutions of $Q(x,\theta)=a$ and $Q(x,\theta)=b$, where these exist).

Proof. For every $\theta$, the events $\{\theta\in C(X)\}$ and $\{a\le Q(X,\theta)\le b\}$ are the same event, so $$P_\theta\left(\theta\in C(X)\right)=P_\theta\left(a\le Q(X,\theta)\le b\right)=G(b)-G(a^-)\ge1-\alpha,$$ and the middle quantity does not depend on $\theta$ because $Q$ is a pivot. For the second claim fix $x$ and let $h(\theta)=Q(x,\theta)$, a continuous strictly monotone function on the interval $\Theta$. The preimage of the interval $[a,b]$ under a continuous monotone function on an interval is an interval: if $\theta_1\lt\theta_2$ both satisfy $a\le h\le b$ and $\theta_1\lt\theta\lt\theta_2$, then $h(\theta)$ lies between $h(\theta_1)$ and $h(\theta_2)$, hence in $[a,b]$. Its endpoints are where $h$ crosses the levels $a$ and $b$, when it does; otherwise an endpoint of $\Theta$. ∎

Where do pivots come from? Three general sources cover almost every example in the syllabus.

📐 Theorem 4.25 — Pivots in location and scale families

Let $X_i=\mu+\sigma Z_i$ $(i=1,\dots,n)$, where $Z_1,\dots,Z_n$ are iid with a known law, $\mu\in\mathbb{R}$ and $\sigma\gt0$. Let $T$ and $S\gt0$ be statistics that are location–scale equivariant: $T(a+cx)=a+cT(x)$ and $S(a+cx)=cS(x)$ for all $a\in\mathbb{R}$, $c\gt0$. Then

$$\frac{T(X)-\mu}{S(X)},\qquad\frac{S(X)}{\sigma},\qquad\frac{T(X)-\mu}{\sigma}$$

are pivots (the first for $\mu$ with $\sigma$ unknown, the second for $\sigma$, the third for $\mu$ when $\sigma$ is known). Examples: $T=\bar X$ or the sample median; $S$ = the sample standard deviation, the range or the mean absolute deviation.

Proof. With $Z=(Z_1,\dots,Z_n)$ we have $X=\mu+\sigma Z$ coordinatewise, so equivariance gives $T(X)=\mu+\sigma T(Z)$ and $S(X)=\sigma S(Z)$. Hence $$\frac{T(X)-\mu}{S(X)}=\frac{T(Z)}{S(Z)},\qquad\frac{S(X)}{\sigma}=S(Z),\qquad\frac{T(X)-\mu}{\sigma}=T(Z),$$ and the right-hand sides are functions of $Z$ alone, whose law is known and free of $(\mu,\sigma)$. ∎

📐 Theorem 4.26 — A pivot from any continuous cdf
If $X_1,\dots,X_n$ are iid with a continuous cdf $F(x;\theta)$, then $$Q(X,\theta)=-2\sum_{i=1}^n\log F(X_i;\theta)\sim\chi^2_{2n},$$ and likewise $-2\sum_{i=1}^n\log\left(1-F(X_i;\theta)\right)\sim\chi^2_{2n}$. Both are pivots.

Proof. By the probability integral transform (§1.3), $U_i=F(X_i;\theta)$ are iid $U(0,1)$ when $F$ is continuous, and so are $1-U_i$. For $t\gt0$, $P(-\log U_i\gt t)=P(U_i\lt e^{-t})=e^{-t}$, so $-\log U_i\sim\operatorname{Exp}(1)=\operatorname{Gamma}(1,1)$. By the reproductive property of the gamma law (§2.1), $-\sum\log U_i\sim\operatorname{Gamma}(n,1)$, and multiplying by $2$ divides the rate by $2$: $-2\sum\log U_i\sim\operatorname{Gamma}(n,\tfrac12)=\chi^2_{2n}$. ∎

For discrete data, such as a binomial count, no exact pivot exists, but one can still invert the distribution function of a statistic. The price is that the coverage is only guaranteed to be at least $1-\alpha$.

📐 Theorem 4.27 — Inverting the distribution function of a statistic

Let $T$ be a real statistic, write $F_\theta(t)=P_\theta(T\le t)$ and $\bar F_\theta(t)=P_\theta(T\ge t)$, and let $\alpha_1,\alpha_2\ge0$ with $\alpha_1+\alpha_2=\alpha$.

  1. For every $\theta$ and every $u\in(0,1)$: $P_\theta\left(F_\theta(T)\le u\right)\le u$ and $P_\theta\left(\bar F_\theta(T)\le u\right)\le u$, with equality if $T$ has a continuous distribution.
  2. Consequently $C(t)=\{\theta:\ F_\theta(t)\gt\alpha_1\ \text{and}\ \bar F_\theta(t)\gt\alpha_2\}$, evaluated at $t=T$, is a confidence set of level $1-\alpha$.
  3. If the family is stochastically increasing ($F_\theta(t)$ is nonincreasing in $\theta$ for every $t$, hence $\bar F_\theta(t)$ nondecreasing), then $C(t)$ is an interval $(\theta_L(t),\theta_U(t))$ up to endpoints, where $\bar F_{\theta_L}(t)=\alpha_2$ and $F_{\theta_U}(t)=\alpha_1$ (when these equations have solutions in $\Theta$).

Proof. (1) Fix $\theta$ and $u$ and write $F=F_\theta$. The set $A=\{t:F(t)\le u\}$ is an interval unbounded to the left, because $F$ is nondecreasing; let $t^*=\sup A$. If $t^*\in A$, then $P(T\in A)=P(T\le t^*)=F(t^*)\le u$. If $t^*\notin A$, then $A=(-\infty,t^*)$ and $P(T\in A)=P(T\lt t^*)=\lim_{t\uparrow t^*}F(t)\le u$. In both cases $P\left(F(T)\le u\right)=P(T\in A)\le u$. The argument for $\bar F$ is the same with the inequalities reversed. If $T$ is continuous, $F(T)\sim U(0,1)$ by the probability integral transform, which gives equality.

(2) $\theta\notin C(T)$ means $F_\theta(T)\le\alpha_1$ or $\bar F_\theta(T)\le\alpha_2$, so by (1) $$P_\theta\left(\theta\notin C(T)\right)\le P_\theta\left(F_\theta(T)\le\alpha_1\right)+P_\theta\left(\bar F_\theta(T)\le\alpha_2\right)\le\alpha_1+\alpha_2=\alpha.$$ (If $\alpha_1=0$ the first event has probability $P_\theta(F_\theta(T)=0)=0$, and similarly for $\alpha_2$.)

(3) For fixed $t$, $\{\theta:F_\theta(t)\gt\alpha_1\}$ is an initial segment of $\Theta$ because $F_\theta(t)$ decreases in $\theta$, and $\{\theta:\bar F_\theta(t)\gt\alpha_2\}$ is a final segment because $\bar F_\theta(t)$ increases. Their intersection is an interval, with endpoints where the two monotone functions cross the levels $\alpha_2$ and $\alpha_1$. ∎

Applied to $X\sim\operatorname{Bin}(n,p)$ with $\alpha_1=\alpha_2=\alpha/2$, and using the beta–binomial tail identity $P_p(X\ge x)=P\left(\operatorname{Beta}(x,n-x+1)\le p\right)$ of §2.1, Theorem 4.27 gives the Clopper–Pearson ("exact") interval: its lower end $p_L$ is the $\alpha/2$ quantile of $\operatorname{Beta}(x,n-x+1)$ ($p_L=0$ if $x=0$), and its upper end $p_U$ is the $1-\alpha/2$ quantile of $\operatorname{Beta}(x+1,n-x)$ ($p_U=1$ if $x=n$). Its coverage is at least $1-\alpha$ for every $p$; it is conservative, usually by a noticeable margin.

A pivot determines the probability content of $[a,b]$ but not the choice of $a$ and $b$. The natural criterion is length.

📐 Theorem 4.28 — Shortest intervals for unimodal pivots

Let $f$ be a density that is unimodal: for some $m$, $f$ is nondecreasing on $(-\infty,m]$ and nonincreasing on $[m,\infty)$. Suppose $a^*\le m\le b^*$, $\int_{a^*}^{b^*}f=1-\alpha$ and $f(a^*)=f(b^*)\gt0$. Then every interval $[a,b]$ with $\int_a^bf\ge1-\alpha$ has $b-a\ge b^*-a^*$.

In particular, if the pivot $Q=(\hat\theta-\theta)/c$ (with $c\gt0$ known or a statistic not depending on $a,b$) has a symmetric unimodal density, then the equal-tailed interval $\hat\theta\pm c\,q_{\alpha/2}$ is the shortest interval of the form $[\hat\theta-bc,\hat\theta-ac]$.

Proof. Note first that unimodality gives $f(x)\le f(a^*)$ for $x\le a^*$, $f(x)\le f(b^*)$ for $x\ge b^*$, and $f(x)\ge f(a^*)$ for $a^*\le x\le b^*$ (on this interval $f$ is at least the smaller of its endpoint values). Suppose, for a contradiction, that $\int_a^bf\ge1-\alpha$ but $b-a\lt b^*-a^*$. Write $k=f(a^*)=f(b^*)$. Consider the case $a\le a^*$ (the case $a\gt a^*$ is symmetric); then $b\lt a+(b^*-a^*)\le b^*$.

If $b\le a^*$: the whole of $[a,b]$ lies left of $a^*$, so $\int_a^bf\le k(b-a)\lt k(b^*-a^*)\le\int_{a^*}^{b^*}f=1-\alpha$, a contradiction.

If $a^*\lt b\lt b^*$: $$\int_a^bf=\int_{a^*}^{b^*}f+\int_a^{a^*}f-\int_b^{b^*}f\le(1-\alpha)+k(a^*-a)-k(b^*-b)=(1-\alpha)+k\left[(b-a)-(b^*-a^*)\right]\lt1-\alpha,$$ using $f\le k$ on $[a,a^*]$ and $f\ge k$ on $[b,b^*]$. Again a contradiction. For the last statement, $\{a\le Q\le b\}=\{\hat\theta-bc\le\theta\le\hat\theta-ac\}$ has length $c(b-a)$, and for a symmetric unimodal density $a^*=-q_{\alpha/2}$, $b^*=q_{\alpha/2}$ satisfy the hypotheses. ∎

Two remarks. First, the theorem concerns the length $b-a$ on the scale of the pivot. For a scale pivot such as $Q=(n-1)S^2/\sigma^2$, the interval for $\sigma^2$ is $[(n-1)S^2/b,\ (n-1)S^2/a]$, whose length is proportional to $\frac1a-\frac1b$, and minimising this under $\int_a^bf=1-\alpha$ leads by a Lagrange multiplier to the condition $a^2f(a)=b^2f(b)$ instead (Exercise I.5). Second, for skewed pivots such as $\chi^2$ the equal-tailed choice is not the shortest, though it is the one used in practice because it is easy to tabulate and has equal error rates on both sides.

✏️ Example 1 — Exact intervals for the uniform and exponential models
(a) $X_1,\dots,X_8$ are iid $U(0,\theta)$ and the largest observation is $x_{(8)}=4.2$. Using the pivot $X_{(n)}/\theta$, find the shortest $95\%$ confidence interval of the form $[X_{(n)}/b,\ X_{(n)}/a]$. (b) Ten electronic components are tested until failure; their lifetimes are iid exponential with mean $\theta$ hours, and the total time on test is $T=\sum x_i=1240$ hours. Find the equal-tailed $95\%$ confidence interval for $\theta$ and for the failure rate $\lambda=1/\theta$.
Pivot (a) $X_{(n)}/\theta$ is the maximum of $n$ iid $U(0,1)$ variables, with cdf $t^n$ on $[0,1]$ (§4.1). It is a pivot, and $\{a\le X_{(n)}/\theta\le b\}=\{X_{(n)}/b\le\theta\le X_{(n)}/a\}$ with $b^n-a^n=1-\alpha$, $0\le a\lt b\le1$.
Shortest (a) The density $nt^{n-1}$ is increasing, so Theorem 4.28 does not apply directly; minimise the length $X_{(n)}\left(\frac1a-\frac1b\right)$ instead. From $a^n=b^n-(1-\alpha)$ we get $\frac{da}{db}=\frac{b^{n-1}}{a^{n-1}}$, so $$\frac{d}{db}\left(\frac1a-\frac1b\right)=-\frac{b^{n-1}}{a^{n+1}}+\frac{1}{b^2}\lt0,$$ because $a\lt b$ gives $\frac{b^{n-1}}{a^{n+1}}\gt\frac{b^{n-1}}{b^{n+1}}=\frac{1}{b^2}$. The length decreases in $b$, so take $b=1$ and $a=\alpha^{1/n}$: the interval is $\left[X_{(n)},\ X_{(n)}\alpha^{-1/n}\right]$.
Numbers (a) $0.05^{1/8}=0.68766$, so $\theta\in[4.2,\ 4.2/0.68766]=[4.200,\ 6.108]$. The interval starts at the observed maximum, as it must: $\theta\ge x_{(8)}$ is certain.
Pivot (b) $X_i\sim\operatorname{Exp}(\lambda)$ with $\lambda=1/\theta$, so $\lambda X_i\sim\operatorname{Exp}(1)$ and $2\lambda T=2T/\theta\sim\chi^2_{2n}$ (the argument of Theorem 4.26; also a scale pivot by Theorem 4.25). With $n=10$, $\chi^2_{20,0.975}=9.5908$ and $\chi^2_{20,0.025}=34.1696$ (upper points).
Numbers (b) $P\left(9.5908\le2T/\theta\le34.1696\right)=0.95$ gives $$\theta\in\left[\frac{2480}{34.1696},\ \frac{2480}{9.5908}\right]=[72.58,\ 258.58]\ \text{hours},$$ and inverting, $\lambda\in[0.003867,\ 0.013778]$ failures per hour. The point estimate $\bar x=124$ hours is not the midpoint: the interval is skewed to the right, like the $\chi^2$ law.
Check The shortest interval of this form solves $a^2f(a)=b^2f(b)$ with $f$ the $\chi^2_{20}$ density: $a=10.612$, $b=39.561$, giving $[62.69,\ 233.70]$ of length $171.0$ hours against $186.0$ for the equal-tailed one. Both have exact coverage $95\%$.
✏️ Example 2 — What does a confidence level promise? (prove or disprove)
Let $X_1,X_2$ be iid $U\left(\theta-\frac12,\theta+\frac12\right)$ and $C=[X_{(1)},X_{(2)}]$. Prove or disprove: (a) $C$ is an exact $50\%$ confidence interval for $\theta$. (b) If the observed range satisfies $x_{(2)}-x_{(1)}\ge\frac12$, then $\theta\in C$ with certainty. (c) Given the range $R=r$ with $r\lt\frac12$, the conditional probability that $C$ covers $\theta$ is still $\frac12$. (d) A $95\%$ confidence interval for a population mean contains about $95\%$ of the observations.
(a) True $C$ misses $\theta$ exactly when both observations fall on the same side of $\theta$, which has probability $\left(\frac12\right)^2+\left(\frac12\right)^2=\frac12$ for every $\theta$. (Equivalently, $X_i-\theta$ is a location pivot, uniform on $(-\frac12,\frac12)$.)
(b) True Both observations lie in $(\theta-\frac12,\theta+\frac12)$. If $x_{(1)}\gt\theta$, then $x_{(2)}\lt\theta+\frac12\lt x_{(1)}+\frac12$, so the range is below $\frac12$; similarly if $x_{(2)}\lt\theta$. Hence a range of at least $\frac12$ forces $x_{(1)}\le\theta\le x_{(2)}$.
(c) False Write $M=\frac{X_{(1)}+X_{(2)}}{2}$. Given $R=r$, the pair is a uniformly placed segment of length $r$ inside the interval of length $1$, so $M$ is uniform on $\left[\theta-\frac12+\frac r2,\ \theta+\frac12-\frac r2\right]$, an interval of length $1-r$. $C$ covers $\theta$ iff $|M-\theta|\le\frac r2$, so $$P(\theta\in C\mid R=r)=\frac{r}{1-r}\quad(r\lt\tfrac12),\qquad=1\quad(r\ge\tfrac12).$$ Averaging against the density $2(1-r)$ of $R$ gives $\int_0^{1/2}2r\,dr+\int_{1/2}^12(1-r)\,dr=\frac14+\frac14=\frac12$, as in (a). The $50\%$ is an average over samples; for a particular sample the evidence can be much stronger (or weaker).
(d) False The interval $\bar x\pm t\,s/\sqrt n$ has width of order $s/\sqrt n$, while the observations spread over a width of order $s$. For $n=100$, the $95\%$ interval is about $0.4s$ wide and contains only a small fraction of the data. Covering $95\%$ of future observations is the job of a prediction or tolerance interval, such as $\bar x\pm t_{n-1,\alpha/2}\,s\sqrt{1+1/n}$ for one new normal observation.

Intervals for Normal Means and Variances

For normal samples the pivots are exact, and their laws were derived in §4.1: if $X_1,\dots,X_n$ are iid $N(\mu,\sigma^2)$ with $n\ge2$, $\bar X=\frac1n\sum X_i$ and $S^2=\frac{1}{n-1}\sum(X_i-\bar X)^2$, then $\bar X\sim N(\mu,\sigma^2/n)$, $(n-1)S^2/\sigma^2\sim\chi^2_{n-1}$, the two are independent, and therefore $\sqrt n(\bar X-\mu)/S\sim t_{n-1}$.

📐 Theorem 4.29 — Confidence intervals for one normal sample

Let $X_1,\dots,X_n$ be iid $N(\mu,\sigma^2)$, $n\ge2$. Each of the following is an exact $1-\alpha$ confidence interval.

  1. For $\mu$, $\sigma$ known: $\bar X\pm z_{\alpha/2}\,\dfrac{\sigma}{\sqrt n}$.
  2. For $\mu$, $\sigma$ unknown: $\bar X\pm t_{n-1,\alpha/2}\,\dfrac{S}{\sqrt n}$.
  3. For $\sigma^2$, $\mu$ unknown: $\left[\dfrac{(n-1)S^2}{\chi^2_{n-1,\alpha/2}},\ \dfrac{(n-1)S^2}{\chi^2_{n-1,1-\alpha/2}}\right]$; for $\sigma$, take square roots.
  4. For $\sigma^2$, $\mu$ known: $\left[\dfrac{\sum(X_i-\mu)^2}{\chi^2_{n,\alpha/2}},\ \dfrac{\sum(X_i-\mu)^2}{\chi^2_{n,1-\alpha/2}}\right]$.

The intervals (1) and (2) are the shortest of the form $\bar X-b\,c\le\mu\le\bar X-a\,c$ with $c=\sigma/\sqrt n$, respectively $c=S/\sqrt n$. The length of (1) is fixed; the length of (2) is random with $$E[\text{length}]=2t_{n-1,\alpha/2}\,c_n\frac{\sigma}{\sqrt n},\qquad c_n=E[S]/\sigma=\sqrt{\frac{2}{n-1}}\,\frac{\Gamma(n/2)}{\Gamma((n-1)/2)}\lt1.$$

Proof. (1) $Z=\sqrt n(\bar X-\mu)/\sigma\sim N(0,1)$ is a pivot and $\{|Z|\le z_{\alpha/2}\}=\{|\bar X-\mu|\le z_{\alpha/2}\sigma/\sqrt n\}$. (2) $T=\sqrt n(\bar X-\mu)/S\sim t_{n-1}$ is a pivot (it is also the location–scale pivot of Theorem 4.25), and $\{|T|\le t_{n-1,\alpha/2}\}$ rearranges to the stated interval. (3) $Q=(n-1)S^2/\sigma^2\sim\chi^2_{n-1}$ and $\{\chi^2_{n-1,1-\alpha/2}\le Q\le\chi^2_{n-1,\alpha/2}\}$ has probability $1-\alpha$; solve for $\sigma^2$, which reverses the order because $Q$ is decreasing in $\sigma^2$. Since $\sigma\mapsto\sigma^2$ is increasing on $(0,\infty)$, the event is the same as the one for $\sigma$ with square roots. (4) The $(X_i-\mu)/\sigma$ are iid $N(0,1)$, so $\sum(X_i-\mu)^2/\sigma^2\sim\chi^2_n$. The shortness claims follow from Theorem 4.28, since $N(0,1)$ and $t_{n-1}$ have symmetric unimodal densities. Finally $E[\text{length}]=2t_{n-1,\alpha/2}E[S]/\sqrt n$, and with $V=(n-1)S^2/\sigma^2\sim\chi^2_{n-1}=\operatorname{Gamma}\left(\frac{n-1}{2},\frac12\right)$, $$E\sqrt V=\int_0^\infty\frac{v^{1/2}v^{(n-1)/2-1}e^{-v/2}}{2^{(n-1)/2}\Gamma\left(\frac{n-1}{2}\right)}dv=\frac{2^{n/2}\Gamma(n/2)}{2^{(n-1)/2}\Gamma\left(\frac{n-1}{2}\right)}=\sqrt2\,\frac{\Gamma(n/2)}{\Gamma\left(\frac{n-1}{2}\right)},$$ so $E[S]=\sigma E\sqrt V/\sqrt{n-1}=c_n\sigma$. Jensen's inequality gives $E[S]\lt\sqrt{E[S^2]}=\sigma$, strictly because $S$ is not constant. ∎

📐 Theorem 4.30 — Two independent normal samples, and paired data

Let $X_1,\dots,X_m$ be iid $N(\mu_1,\sigma_1^2)$ and $Y_1,\dots,Y_n$ iid $N(\mu_2,\sigma_2^2)$, independent of the $X$'s, with sample variances $S_1^2$, $S_2^2$. Exact $1-\alpha$ intervals:

  1. For $\mu_1-\mu_2$, variances known: $\bar X-\bar Y\pm z_{\alpha/2}\sqrt{\dfrac{\sigma_1^2}{m}+\dfrac{\sigma_2^2}{n}}$.
  2. For $\mu_1-\mu_2$, $\sigma_1=\sigma_2=\sigma$ unknown: $\bar X-\bar Y\pm t_{m+n-2,\alpha/2}\,S_p\sqrt{\dfrac1m+\dfrac1n}$, with the pooled variance $$S_p^2=\frac{(m-1)S_1^2+(n-1)S_2^2}{m+n-2}.$$
  3. For $\sigma_1^2/\sigma_2^2$: with $R=S_1^2/S_2^2$, $\left[\dfrac{R}{F_{m-1,n-1,\alpha/2}},\ R\,F_{n-1,m-1,\alpha/2}\right]$.
  4. (Paired data) If instead the observations come in pairs $(X_i,Y_i)$, $i=1,\dots,n$, whose differences $D_i=X_i-Y_i$ are iid $N(\mu_D,\sigma_D^2)$, then $\bar D\pm t_{n-1,\alpha/2}S_D/\sqrt n$ is an exact interval for $\mu_D=\mu_1-\mu_2$.

(Welch, statement.) If $\sigma_1^2\ne\sigma_2^2$ are unknown, there is no exact pivot for $\mu_1-\mu_2$ (the Behrens–Fisher problem). The approximate interval $\bar X-\bar Y\pm t_{\hat\nu,\alpha/2}\sqrt{S_1^2/m+S_2^2/n}$, with $\hat\nu=\dfrac{(S_1^2/m+S_2^2/n)^2}{\frac{(S_1^2/m)^2}{m-1}+\frac{(S_2^2/n)^2}{n-1}}$ degrees of freedom, has coverage close to $1-\alpha$ in practice.

Proof. (1) By independence, $\bar X-\bar Y$ is normal with mean $\mu_1-\mu_2$ and variance $\frac{\sigma_1^2}{m}+\frac{\sigma_2^2}{n}$; standardise. (2) Now $\bar X-\bar Y$ is normal with mean $\mu_1-\mu_2$ and variance $\sigma^2(\frac1m+\frac1n)$. The variables $(m-1)S_1^2/\sigma^2\sim\chi^2_{m-1}$ and $(n-1)S_2^2/\sigma^2\sim\chi^2_{n-1}$ are independent, so their sum $(m+n-2)S_p^2/\sigma^2\sim\chi^2_{m+n-2}$ by the reproductive property. Within each sample the mean is independent of the variance, and the samples are independent, so $(\bar X,\bar Y)$ is independent of $(S_1^2,S_2^2)$. Hence $$\frac{\bar X-\bar Y-(\mu_1-\mu_2)}{S_p\sqrt{1/m+1/n}}=\frac{Z}{\sqrt{\chi^2_{m+n-2}/(m+n-2)}}\sim t_{m+n-2},$$ where the unknown $\sigma$ cancels. (3) $\dfrac{S_1^2/\sigma_1^2}{S_2^2/\sigma_2^2}$ is a ratio of independent $\chi^2$ variables divided by their degrees of freedom, so it is $F_{m-1,n-1}$ (§4.1), and it equals $R\,\sigma_2^2/\sigma_1^2$. Solving $F_{m-1,n-1,1-\alpha/2}\le R\sigma_2^2/\sigma_1^2\le F_{m-1,n-1,\alpha/2}$ for $\sigma_1^2/\sigma_2^2$ and using the reciprocal relation $$\frac{1}{F_{m-1,n-1,1-\alpha/2}}=F_{n-1,m-1,\alpha/2}$$ gives the interval. (4) is Theorem 4.29(2) applied to the $D_i$. ∎

Pairing matters. If each patient is measured before and after a treatment, $X_i$ and $Y_i$ are positively correlated, and $\operatorname{Var}(D_i)=\sigma_1^2+\sigma_2^2-2\operatorname{Cov}(X_i,Y_i)$ can be far smaller than $\sigma_1^2+\sigma_2^2$. Treating such data as two independent samples is not merely inefficient; it uses a wrong variance and hence a wrong interval.

🧭 Problem-solving strategy — building a confidence interval
  1. Model and nuisance parameters. Which parameter do you want, and which other parameters are unknown? The pivot must not contain the unknown nuisance parameters.
  2. Start from a good statistic: a sufficient statistic or the MLE (§4.2): $\bar X$ and $S^2$ for the normal, $\sum X_i$ for the exponential and Poisson, $X_{(n)}$ for $U(0,\theta)$, $X_{(1)}$ for a shifted exponential.
  3. Standardise into a pivot: subtract a location, divide by a scale (Theorem 4.25), use $F(X;\theta)$ (Theorem 4.26), or invert the cdf of the statistic (Theorem 4.27, for discrete data).
  4. Choose the cut-offs: equal tails by default; for a symmetric unimodal pivot this is also the shortest; for a pivot supported on $[0,1]$ with increasing density, put the whole of $\alpha$ in one tail (Example 1(a)).
  5. Invert and check: solve for the parameter, check the direction of the inequalities, the parameter space (a probability cannot be negative) and the known constraints ($\theta\ge x_{(n)}$ for $U(0,\theta)$).
  6. Large $n$, no exact pivot? Use an asymptotic pivot (Theorem 4.31) and say that the level is approximate.
✏️ Example 3 — Tablet weights: intervals for the mean and the standard deviation
A quality-control laboratory weighs $n=25$ tablets from a batch and finds $\bar x=502.3$ mg and $s=4.1$ mg. Assume normality. (a) Find a $95\%$ confidence interval for the batch mean $\mu$. Is the specification $500\le\mu\le505$ mg plausibly met? (b) Find $95\%$ intervals for $\sigma^2$ and $\sigma$. (c) If $\sigma=4.1$ mg were known, how large must $n$ be for the $95\%$ interval for $\mu$ to have half-width at most $1$ mg?
(a) $t_{24,0.025}=2.0639$ and $s/\sqrt n=4.1/5=0.82$, so the half-width is $2.0639\times0.82=1.6924$ and $$\mu\in[500.61,\ 503.99]\ \text{mg}.$$ The whole interval lies inside $[500,505]$, so the data are consistent with the specification. (With $\sigma$ known the half-width would be $1.96\times0.82=1.607$; the $t$ quantile pays for estimating $\sigma$.)
(b) $(n-1)s^2=24\times16.81=403.44$, $\chi^2_{24,0.025}=39.3641$ and $\chi^2_{24,0.975}=12.4012$. So $$\sigma^2\in\left[\tfrac{403.44}{39.3641},\ \tfrac{403.44}{12.4012}\right]=[10.249,\ 32.532],\qquad\sigma\in[3.201,\ 5.704]\ \text{mg}.$$ The interval for $\sigma$ is not symmetric about $s=4.1$.
(c) We need $1.96\times4.1/\sqrt n\le1$, i.e. $n\ge(1.96\times4.1)^2=64.57$, so $n=65$ tablets. In general $n\ge\left(z_{\alpha/2}\sigma/E\right)^2$ for half-width $E$: halving $E$ quadruples $n$.
Check The shortest interval for $\sigma^2$ of the form $[403.44/b,\ 403.44/a]$ solves $a^2f(a)=b^2f(b)$ for the $\chi^2_{24}$ density: $a=13.523$, $b=44.480$, giving $[9.070,\ 29.834]$, of length $20.76$ against $22.28$. The gain is modest, which is why the equal-tailed interval is standard.
✏️ Example 4 — Comparing two teaching methods
Two groups of students are taught by different methods and take the same test. Group 1: $m=10$, $\bar x=72.4$, $s_1=8.2$. Group 2: $n=12$, $\bar y=65.1$, $s_2=7.5$. Assume independent normal samples. (a) Find a $95\%$ interval for $\sigma_1^2/\sigma_2^2$. Is the equal-variance assumption reasonable? (b) Find the pooled $95\%$ interval for $\mu_1-\mu_2$. (c) Compare with Welch's interval. (d) What would change if the same $10$ students had been taught by both methods in succession?
(a) $R=8.2^2/7.5^2=67.24/56.25=1.1954$, $F_{9,11,0.025}=3.5879$ and $F_{11,9,0.025}=3.9121$. So $\sigma_1^2/\sigma_2^2\in[1.1954/3.5879,\ 1.1954\times3.9121]=[0.333,\ 4.676]$. The interval contains $1$ comfortably, so equal variances are plausible (though the interval is very wide: small samples say little about variances).
(b) $s_p^2=\frac{9\times67.24+11\times56.25}{20}=61.1955$, $s_p=7.8228$, and the standard error is $s_p\sqrt{\frac1{10}+\frac1{12}}=3.3495$. With $t_{20,0.025}=2.0860$: $$\mu_1-\mu_2\in7.3\pm6.987=[0.31,\ 14.29].$$ The interval excludes $0$, so the data favour method 1, by somewhere between a fraction of a mark and $14$ marks.
(c) Welch: $\sqrt{67.24/10+56.25/12}=3.3781$, $\hat\nu=18.55$, $t_{18.55,0.025}=2.0965$, giving $[0.22,\ 14.38]$, almost the same. When the sample variances are similar, the two methods agree; Welch's is the safe default when they are not.
(d) Then the data are paired: use the $10$ differences and Theorem 4.30(4), with $t_{9,0.025}$. The two-sample formulas would be invalid because each student's two scores are correlated (a strong student scores well under both methods).

Large-Sample Intervals and Intervals for Proportions

Outside the normal model exact pivots are rare. But many estimators are asymptotically normal: the sample mean by the CLT, functions of means by the delta method, and MLEs by their asymptotic theory (§4.2). Each such limit gives an asymptotic pivot.

📐 Theorem 4.31 — Asymptotic confidence intervals

Let $\hat\theta_n$ be estimators with $\sqrt n(\hat\theta_n-\theta)\xrightarrow{d}N\left(0,\tau^2(\theta)\right)$ and $\tau(\theta)\gt0$ for every $\theta$, and let $\hat\tau_n\xrightarrow{P}\tau(\theta)$ for every $\theta$. Then each of the following is an asymptotic $1-\alpha$ confidence set for $\theta$:

  1. (Wald) $\hat\theta_n\pm z_{\alpha/2}\,\hat\tau_n/\sqrt n$;
  2. (Score type) $\left\{\theta:\ \sqrt n\,|\hat\theta_n-\theta|\le z_{\alpha/2}\,\tau(\theta)\right\}$;
  3. (Variance-stabilised) if $g$ is continuously differentiable and strictly increasing with $g'(\theta)\tau(\theta)=1$: $\left[g^{-1}\left(g(\hat\theta_n)-\frac{z_{\alpha/2}}{\sqrt n}\right),\ g^{-1}\left(g(\hat\theta_n)+\frac{z_{\alpha/2}}{\sqrt n}\right)\right]$.

For the MLE in a regular family (§4.2), $\tau^2(\theta)=1/I(\theta)$ with $I$ the Fisher information of one observation, and one may take $\hat\tau_n^2=1/I(\hat\theta_n)$.

Proof. (1) By Slutsky's theorem (§2.3), $W_n=\sqrt n(\hat\theta_n-\theta)/\hat\tau_n$ satisfies $W_n\xrightarrow{d}N(0,1)$, because $\tau(\theta)/\hat\tau_n\xrightarrow{P}1$. The interval contains $\theta$ iff $|W_n|\le z_{\alpha/2}$, and since $\pm z_{\alpha/2}$ are continuity points of $\Phi$, $$P_\theta(|W_n|\le z_{\alpha/2})\to\Phi(z_{\alpha/2})-\Phi(-z_{\alpha/2})=1-\alpha.$$ (2) For fixed $\theta$, $\tau(\theta)$ is a constant, so $\sqrt n(\hat\theta_n-\theta)/\tau(\theta)\xrightarrow{d}N(0,1)$ directly, and the set contains $\theta$ iff this quantity lies in $[-z_{\alpha/2},z_{\alpha/2}]$. (3) By the delta method, $\sqrt n\left(g(\hat\theta_n)-g(\theta)\right)\xrightarrow{d}N\left(0,g'(\theta)^2\tau^2(\theta)\right)=N(0,1)$, so $g(\hat\theta_n)\pm z_{\alpha/2}/\sqrt n$ is an asymptotic interval for $g(\theta)$, and applying the increasing function $g^{-1}$ preserves the event. ∎

📐 Theorem 4.32 — Standard large-sample intervals

All are asymptotic $1-\alpha$ intervals; $z=z_{\alpha/2}$.

  1. (Any mean) iid $X_i$ with mean $\mu$ and finite variance $\sigma^2\gt0$: $\bar X\pm z\,S/\sqrt n$. For two independent samples: $\bar X-\bar Y\pm z\sqrt{S_1^2/m+S_2^2/n}$.
  2. (Proportion) $X\sim\operatorname{Bin}(n,p)$, $\hat p=X/n$, $0\lt p\lt1$. Wald: $\hat p\pm z\sqrt{\hat p(1-\hat p)/n}$. Wilson (score): $$\frac{\hat p+\frac{z^2}{2n}\pm z\sqrt{\frac{\hat p(1-\hat p)}{n}+\frac{z^2}{4n^2}}}{1+\frac{z^2}{n}}.$$
  3. (Difference of proportions) independent $X\sim\operatorname{Bin}(m,p_1)$, $Y\sim\operatorname{Bin}(n,p_2)$: $\hat p_1-\hat p_2\pm z\sqrt{\dfrac{\hat p_1(1-\hat p_1)}{m}+\dfrac{\hat p_2(1-\hat p_2)}{n}}$.
  4. (Poisson mean) iid $\operatorname{Poi}(\lambda)$: Wald $\bar X\pm z\sqrt{\bar X/n}$; variance-stabilised $\left(\sqrt{\bar X}\pm\frac{z}{2\sqrt n}\right)^2$ (with the lower end replaced by $0$ if $\sqrt{\bar X}\lt z/(2\sqrt n)$).

Proof. (1) The CLT gives $\sqrt n(\bar X-\mu)\xrightarrow{d}N(0,\sigma^2)$ and the strong law gives $S\to\sigma$ a.s. (§2.3); apply Theorem 4.31(1). For two samples, $\bar X-\bar Y$ is a difference of independent asymptotically normal variables, and its standardised version converges to $N(0,1)$ (by characteristic functions) as $m,n\to\infty$; again replace the variances by consistent estimates. (2) $\hat p$ is the mean of $n$ Bernoulli variables, with $\tau^2(p)=p(1-p)$ and $\hat\tau^2=\hat p(1-\hat p)\to p(1-p)$; this is the Wald interval. For Wilson, apply Theorem 4.31(2): $(\hat p-p)^2\le z^2p(1-p)/n$ is the quadratic inequality $$\left(1+\tfrac{z^2}{n}\right)p^2-\left(2\hat p+\tfrac{z^2}{n}\right)p+\hat p^2\le0,$$ whose discriminant is $\left(2\hat p+\frac{z^2}{n}\right)^2-4\left(1+\frac{z^2}{n}\right)\hat p^2=\frac{4z^2}{n}\hat p(1-\hat p)+\frac{z^4}{n^2}$. The set of $p$ between the two roots is the stated interval. (3) is (1) for two Bernoulli samples. (4) $\tau^2(\lambda)=\lambda$, so $g(\lambda)=2\sqrt\lambda$ has $g'(\lambda)\tau(\lambda)=1$; Theorem 4.31(3) gives $2\sqrt\lambda\in2\sqrt{\bar X}\pm z/\sqrt n$. ∎

The Wilson interval is the set of $p$ that are not rejected by the large-sample test of §5.2 that uses the null standard error $\sqrt{p(1-p)/n}$. It always lies in $[0,1]$, it never collapses to a point, and its centre $\frac{\hat p+z^2/(2n)}{1+z^2/n}$ is pulled from $\hat p$ towards $\frac12$. For $95\%$ the centre is close to $\frac{x+2}{n+4}$, which explains the popular Agresti–Coull rule: add two successes and two failures, then use the Wald formula. The Wald interval, by contrast, behaves badly for small $n$ or $p$ near $0$ or $1$ (pitfall 2 and Interactive 1).

✏️ Example 5 — Rare adverse reactions and an opinion poll
(a) In a trial, $x=6$ of $n=40$ patients have an adverse reaction. Compute the $95\%$ Wald, Wilson and Clopper–Pearson intervals for the reaction probability $p$. (b) In a second trial, $0$ of $30$ patients react. What do the Wald and Clopper–Pearson methods give? Find the exact $95\%$ upper confidence bound and compare it with the "rule of three" $3/n$. (c) Two independent polls of $600$ voters each find $312$ and $270$ supporters of a proposal. Find a $95\%$ interval for the difference of the two population proportions.
(a) Wald $\hat p=0.15$, standard error $\sqrt{0.15\times0.85/40}=0.05646$, so $0.15\pm0.1107=[0.0393,\ 0.2607]$.
(a) Wilson With $z^2/n=3.8415/40=0.09604$: centre $\frac{0.15+0.04802}{1.09604}=0.1807$, half-width $\frac{1.96\sqrt{0.0031875+0.0006}}{1.09604}=0.1101$, giving $[0.0706,\ 0.2907]$. The interval is shifted to the right of the Wald interval: for small $\hat p$, the binomial law is skewed and the lower end must stay further from $0$.
(a) Clopper–Pearson $p_L$ = the $0.025$ quantile of $\operatorname{Beta}(6,35)$ $=0.0571$ and $p_U$ = the $0.975$ quantile of $\operatorname{Beta}(7,34)$ $=0.2984$, so $[0.0571,\ 0.2984]$: the widest of the three, as expected of a conservative interval.
(b) $\hat p=0$ makes the Wald standard error $0$, and the "interval" is the single point $\{0\}$, which is absurd: zero reactions in $30$ patients is far from proof that $p=0$. Clopper–Pearson with $x=0$ gives $p_L=0$ and $p_U=1-0.025^{1/30}=0.1157$ (because $P_p(X\le0)=(1-p)^{30}=0.025$). The one-sided exact $95\%$ upper bound solves $(1-p)^{30}=0.05$: $p_U=1-0.05^{1/30}=0.0950$. The rule of three gives $3/30=0.1$, because $1-\alpha^{1/n}\approx\frac{-\log\alpha}{n}$ and $-\log0.05=2.996$.
(c) $\hat p_1=0.52$, $\hat p_2=0.45$, standard error $\sqrt{\frac{0.52\times0.48}{600}+\frac{0.45\times0.55}{600}}=0.02878$, so $p_1-p_2\in0.07\pm0.0564=[0.0136,\ 0.1264]$. The interval excludes $0$.
Interactive 1 — Confidence Intervals in Repeated Sampling

Each tick draws a fresh sample of size $n$ (a seeded generator) and computes one interval by the chosen rule. The newest $40$ intervals are drawn as horizontal bars, the newest at the top. Bars that cover the true parameter (dashed purple line) are blue; misses are red. The readouts compare the running coverage with the exact coverage of the rule. Try the "$z$ with $s$" rule for $n=5$, and the Wald rule for $p=0.05$ and $n=20$.

100.30
intervals drawn—
coverage so far—
exact coverage of the rule—
mean length so far—

Bayesian Inference: Priors, Posteriors and Conjugate Families

The frequentist treats $\theta$ as a fixed unknown constant and makes probability statements only about the data. The Bayesian treats the unknown $\theta$ as the value of a random variable $\Theta$ whose distribution, the prior, describes what is believed about $\theta$ before the data are seen: from earlier studies, expert knowledge, or deliberately vague. Bayes' theorem (§1.2), in its version for densities, then turns the prior and the data into the posterior distribution, which contains everything the analysis has to say about $\theta$.

📘 Definition 4.15 — Prior, posterior and predictive distributions

A Bayesian model consists of a prior density (or pmf) $\pi(\theta)$ on $\Theta$ and, for each $\theta$, the density (or pmf) $f(x\mid\theta)$ of the data given $\Theta=\theta$. Write $L(\theta;x)=f(x\mid\theta)$ for the likelihood.

  1. The marginal (prior predictive) density of the data is $m(x)=\int_\Theta f(x\mid\theta)\pi(\theta)\,d\theta$ (a sum if $\Theta$ is discrete).
  2. For $x$ with $0\lt m(x)\lt\infty$, the posterior density is $$\pi(\theta\mid x)=\frac{f(x\mid\theta)\,\pi(\theta)}{m(x)}.$$
  3. If a future observation $Y$ has density $f(y\mid\theta)$ and is conditionally independent of $X$ given $\Theta$, its posterior predictive density is $f(y\mid x)=\int_\Theta f(y\mid\theta)\,\pi(\theta\mid x)\,d\theta$.
  4. A nonnegative $\pi$ with $\int_\Theta\pi=\infty$ is an improper prior; it may still be used formally if $\int_\Theta f(x\mid\theta)\pi(\theta)\,d\theta\lt\infty$, and the resulting posterior is then a proper density.
📐 Theorem 4.33 — Basic properties of the posterior
  1. $\pi(\theta\mid x)$ is the conditional density of $\Theta$ given $X=x$, and $\pi(\theta\mid x)\propto L(\theta;x)\,\pi(\theta)$ as a function of $\theta$: posterior $\propto$ likelihood $\times$ prior.
  2. If $T$ is a sufficient statistic for $\theta$, then $\pi(\theta\mid x)$ depends on $x$ only through $T(x)$.
  3. (Sequential updating) If $x=(x',x'')$ where $X'$ and $X''$ are conditionally independent given $\Theta$, then $\pi(\theta\mid x)\propto f(x''\mid\theta)\,\pi(\theta\mid x')$: the posterior after $x'$ is the prior for $x''$.
  4. Multiplying the prior or the likelihood by a positive constant (one not depending on $\theta$) leaves the posterior unchanged.

Proof. (1) The pair $(\Theta,X)$ has joint density $\pi(\theta)f(x\mid\theta)$, so the marginal density of $X$ is $m(x)$ and the conditional density of $\Theta$ given $X=x$ is the joint density divided by $m(x)$ (§1.4). The factor $1/m(x)$ does not depend on $\theta$. (2) By the factorisation theorem (§4.2), $f(x\mid\theta)=g\left(T(x),\theta\right)h(x)$. For $h(x)\gt0$, $$\pi(\theta\mid x)=\frac{g(T(x),\theta)h(x)\pi(\theta)}{\int g(T(x),\theta')h(x)\pi(\theta')\,d\theta'}=\frac{g(T(x),\theta)\pi(\theta)}{\int g(T(x),\theta')\pi(\theta')\,d\theta'},$$ which involves $x$ only through $T(x)$. (3) $f(x\mid\theta)=f(x'\mid\theta)f(x''\mid\theta)$, so $$\pi(\theta\mid x)\propto f(x''\mid\theta)\left[f(x'\mid\theta)\pi(\theta)\right]\propto f(x''\mid\theta)\,\pi(\theta\mid x').$$ (4) The constant cancels between numerator and $m(x)$. ∎

In practice one computes a posterior by writing down likelihood $\times$ prior, dropping every factor that does not involve $\theta$, and recognising the kernel of a known density. This works smoothly when prior and posterior belong to the same family.

📘 Definition 4.16 — Conjugate family
A family $\mathcal P$ of prior distributions is conjugate for the model $\{f(x\mid\theta):\theta\in\Theta\}$ if, for every prior in $\mathcal P$ and every possible data set $x$, the posterior also belongs to $\mathcal P$.
📐 Theorem 4.34 — Three conjugate pairs
  1. (Beta–binomial) Let $X_1,\dots,X_n\mid p$ be iid $\operatorname{Ber}(p)$, $x=\sum x_i$, and let $p\sim\operatorname{Beta}(a,b)$. Then $p\mid x\sim\operatorname{Beta}(a+x,\ b+n-x)$, $$E[p\mid x]=\frac{a+x}{a+b+n}=w\,\frac xn+(1-w)\,\frac{a}{a+b},\qquad w=\frac{n}{a+b+n},$$ and the predictive probability of a success at trial $n+1$ is $P(X_{n+1}=1\mid x)=E[p\mid x]$. The same holds if only $X=\sum X_i\sim\operatorname{Bin}(n,p)$ is observed.
  2. (Gamma–Poisson) Let $X_1,\dots,X_n\mid\lambda$ be iid $\operatorname{Poi}(\lambda)$ and $\lambda\sim\operatorname{Gamma}(\alpha,\beta)$ (rate $\beta$). Then $\lambda\mid x\sim\operatorname{Gamma}\left(\alpha+\sum x_i,\ \beta+n\right)$, $$E[\lambda\mid x]=\frac{\alpha+\sum x_i}{\beta+n}=w\,\bar x+(1-w)\frac\alpha\beta,\qquad w=\frac{n}{\beta+n}.$$ A new count $Y$ has the negative binomial predictive law $$P(Y=y\mid x)=\frac{\Gamma(\alpha'+y)}{y!\,\Gamma(\alpha')}\left(\frac{\beta'}{\beta'+1}\right)^{\alpha'}\left(\frac{1}{\beta'+1}\right)^y,$$ where $\alpha'=\alpha+\sum x_i$ and $\beta'=\beta+n$. (For exponential data $X_i\mid\lambda\sim\operatorname{Exp}(\lambda)$, the gamma prior is also conjugate: $\lambda\mid x\sim\operatorname{Gamma}(\alpha+n,\ \beta+\sum x_i)$.)
  3. (Normal–normal) Let $X_1,\dots,X_n\mid\theta$ be iid $N(\theta,\sigma^2)$ with $\sigma^2$ known, and $\theta\sim N(\mu_0,\tau_0^2)$. Then $\theta\mid x\sim N(\mu_n,\tau_n^2)$ with $$\frac{1}{\tau_n^2}=\frac{1}{\tau_0^2}+\frac{n}{\sigma^2},\qquad\mu_n=\tau_n^2\left(\frac{\mu_0}{\tau_0^2}+\frac{n\bar x}{\sigma^2}\right)=w\,\bar x+(1-w)\,\mu_0,\quad w=\frac{n/\sigma^2}{n/\sigma^2+1/\tau_0^2}.$$ Precisions (reciprocal variances) add, and the posterior mean is the precision-weighted average of the prior mean and the sample mean. The predictive law of a new observation is $N(\mu_n,\ \sigma^2+\tau_n^2)$.

Proof. (1) Up to factors free of $p$, $$\pi(p\mid x)\propto p^{x}(1-p)^{n-x}\cdot p^{a-1}(1-p)^{b-1}=p^{a+x-1}(1-p)^{b+n-x-1},\qquad0\lt p\lt1,$$ the kernel of $\operatorname{Beta}(a+x,b+n-x)$, whose mean is $\frac{a+x}{a+b+n}$ (§2.1). The weighted-average form follows from $a+x=n\cdot\frac xn+(a+b)\cdot\frac{a}{a+b}$. For the predictive probability, $P(X_{n+1}=1\mid x)=\int_0^1p\,\pi(p\mid x)\,dp=E[p\mid x]$, by Definition 4.15(3). The binomial likelihood differs from the Bernoulli one by the factor $\binom nx$, which cancels (Theorem 4.33(4)).

(2) Up to factors free of $\lambda$, $$\prod_i\frac{e^{-\lambda}\lambda^{x_i}}{x_i!}\cdot\lambda^{\alpha-1}e^{-\beta\lambda}\propto\lambda^{\alpha+\sum x_i-1}e^{-(\beta+n)\lambda},$$ the $\operatorname{Gamma}(\alpha',\beta')$ kernel, with mean $\alpha'/\beta'$; and $\alpha+\sum x_i=n\bar x+\beta\cdot\frac{\alpha}{\beta}$. The predictive pmf is $$\int_0^\infty\frac{e^{-\lambda}\lambda^y}{y!}\cdot\frac{\beta'^{\alpha'}\lambda^{\alpha'-1}e^{-\beta'\lambda}}{\Gamma(\alpha')}\,d\lambda=\frac{\beta'^{\alpha'}}{y!\,\Gamma(\alpha')}\cdot\frac{\Gamma(\alpha'+y)}{(\beta'+1)^{\alpha'+y}},$$ which is the stated negative binomial pmf. For exponential data, $\prod\lambda e^{-\lambda x_i}\cdot\lambda^{\alpha-1}e^{-\beta\lambda}=\lambda^{\alpha+n-1}e^{-(\beta+\sum x_i)\lambda}$.

(3) Using $\sum(x_i-\theta)^2=\sum(x_i-\bar x)^2+n(\bar x-\theta)^2$ and dropping terms free of $\theta$, $$\log\pi(\theta\mid x)=-\frac{n(\theta-\bar x)^2}{2\sigma^2}-\frac{(\theta-\mu_0)^2}{2\tau_0^2}+c=-\frac12\left[\left(\frac{n}{\sigma^2}+\frac1{\tau_0^2}\right)\theta^2-2\theta\left(\frac{n\bar x}{\sigma^2}+\frac{\mu_0}{\tau_0^2}\right)\right]+c'.$$ Completing the square, this is $-\frac{(\theta-\mu_n)^2}{2\tau_n^2}+c''$ with $\tau_n$ and $\mu_n$ as stated, the log-kernel of $N(\mu_n,\tau_n^2)$. For the predictive law, $Y=\theta+\varepsilon$ with $\theta\mid x\sim N(\mu_n,\tau_n^2)$ and $\varepsilon\sim N(0,\sigma^2)$ independent of $\theta$ given the data, so $Y\mid x\sim N(\mu_n,\tau_n^2+\sigma^2)$. ∎

The weights $w$ have a common interpretation. In the beta–binomial case the prior acts like $a+b$ earlier trials containing $a$ successes; in the gamma–Poisson case like $\beta$ earlier periods with $\alpha$ events in total; in the normal case like $\sigma^2/\tau_0^2$ earlier observations. As $n\to\infty$ with the prior fixed, $w\to1$ and the posterior mean approaches the MLE: the data overwhelm the prior. Actuaries call $w$ the credibility factor.

Vague and improper priors. Letting $\tau_0\to\infty$ in the normal–normal model gives the flat improper prior $\pi(\theta)\propto1$ and the posterior $N(\bar x,\sigma^2/n)$. The uniform prior $\operatorname{Beta}(1,1)$ for $p$ gives the posterior mean $\frac{x+1}{n+2}$ (Laplace's rule of succession). But "flat" depends on the parametrisation: if $p\sim U(0,1)$, the log-odds $\ell=\log\frac{p}{1-p}$ has the logistic density $\frac{e^\ell}{(1+e^\ell)^2}$, which is far from flat. Jeffreys' prior $\pi(\theta)\propto\sqrt{I(\theta)}$, with $I$ the Fisher information, is invariant under reparametrisation; for the Bernoulli model $I(p)=\frac{1}{p(1-p)}$ and it is $\operatorname{Beta}(\frac12,\frac12)$; for the Poisson model it is the improper $\lambda^{-1/2}$. Improper priors must be used with care: the Haldane prior $\pi(p)\propto\frac{1}{p(1-p)}$ gives a proper posterior only when $0\lt x\lt n$ (Exercise IV.9).

✏️ Example 6 — Conjugate updating in two applications
(a) Before a small clinical study, earlier experience with similar drugs suggests a response rate around $0.3$, summarised by the prior $p\sim\operatorname{Beta}(3,7)$. In the study, $14$ of $30$ patients respond. Find the posterior, the posterior mean and its decomposition as a weighted average, a $95\%$ equal-tailed credible interval, and the predictive probability that the next two patients both respond. (b) A road junction had a long-run accident rate summarised by $\lambda\sim\operatorname{Gamma}(4,2)$ (prior mean $2$ per month). After a change in the traffic lights, $21$ accidents occur in $6$ months. Find the posterior of $\lambda$, its mean and variance, and the predictive probability of no accident next month.
(a) Posterior By Theorem 4.34(1), $p\mid x\sim\operatorname{Beta}(3+14,\ 7+16)=\operatorname{Beta}(17,23)$. The posterior mean is $\frac{17}{40}=0.425$, and with $w=\frac{30}{40}=0.75$: $0.75\times\frac{14}{30}+0.25\times0.3=0.35+0.075=0.425$. The estimate sits between the prior guess $0.3$ and the sample proportion $0.467$, three quarters of the way towards the data.
(a) Interval The $0.025$ and $0.975$ quantiles of $\operatorname{Beta}(17,23)$ are $0.2781$ and $0.5790$, so $P(0.278\le p\le0.579\mid x)=0.95$. For comparison, the Wald $95\%$ confidence interval from the study alone is $[0.288,\ 0.645]$: the prior pulls the interval down and narrows it.
(a) Prediction Given $p$, the next two responses are independent, so $P(\text{both}\mid x)=E[p^2\mid x]=\frac{17\times18}{40\times41}=0.1866$. This exceeds $E[p\mid x]^2=0.1806$ by $\operatorname{Var}(p\mid x)=0.00596$: the two future responses are not independent once $p$ is uncertain.
(b) By Theorem 4.34(2), $\lambda\mid x\sim\operatorname{Gamma}(4+21,\ 2+6)=\operatorname{Gamma}(25,8)$, with mean $\frac{25}{8}=3.125$ and variance $\frac{25}{64}=0.3906$ (standard deviation $0.625$). Here $w=\frac68=0.75$: $0.75\times3.5+0.25\times2=3.125$. The predictive probability of no accident is $\left(\frac{8}{9}\right)^{25}=0.0526$, a little larger than $e^{-3.125}=0.0439$, the value we would get by plugging in the posterior mean, because the predictive law also accounts for the uncertainty in $\lambda$.
Interactive 2 — Bayesian Updating for a Proportion

Bernoulli trials with true success probability $p$ (a seeded generator) arrive one at a time. The amber dashed curve is the prior, the green curve the likelihood (scaled to a density), and the purple curve the posterior $\operatorname{Beta}(a+x,\,b+n-x)$, with its $95\%$ equal-tailed credible interval shaded. The red line marks the true $p$. Compare a sceptical prior with a uniform one: early on they disagree; after a hundred trials they nearly coincide.

0.25
data: successes / trials—
posterior; weight on data—
posterior mean / median / mode—
95% credible vs 95% Wald—

Bayes Estimators and Credible Intervals

A posterior distribution is a complete answer, but often a single number or an interval is wanted. Decision theory says how to choose it: specify the cost of each possible error and minimise the expected cost.

📘 Definition 4.17 — Loss, risk and Bayes estimators
  1. A loss function $L(\theta,a)\ge0$ is the cost of estimating $\theta$ by $a$. Standard choices: squared error $(\theta-a)^2$, absolute error $|\theta-a|$, and weighted squared error $w(\theta)(\theta-a)^2$.
  2. The risk of an estimator $\delta(X)$ is $R(\theta,\delta)=E_\theta\left[L(\theta,\delta(X))\right]$; for squared error it is the mean squared error.
  3. For a prior $\pi$, the Bayes risk of $\delta$ is $r(\pi,\delta)=\int_\Theta R(\theta,\delta)\,\pi(\theta)\,d\theta$, and a Bayes estimator $\delta_\pi$ minimises $r(\pi,\delta)$ over all estimators.
  4. The posterior expected loss of an action $a$ given $x$ is $\rho(a\mid x)=E\left[L(\Theta,a)\mid X=x\right]$, that is, $\rho(a\mid x)=\int_\Theta L(\theta,a)\,\pi(\theta\mid x)\,d\theta$.
📐 Theorem 4.35 — Bayes estimators under squared and absolute loss
  1. If, for every $x$, $\delta(x)$ minimises the posterior expected loss $\rho(a\mid x)$ over $a$, and $r(\pi,\delta)\lt\infty$, then $\delta$ is a Bayes estimator.
  2. Under squared error loss, with $E[\Theta^2\mid x]\lt\infty$, the Bayes estimator is the posterior mean $\delta(x)=E[\Theta\mid X=x]$, and the minimum posterior expected loss is the posterior variance $\operatorname{Var}(\Theta\mid x)$. Under weighted squared error $w(\theta)(\theta-a)^2$ it is $E[w(\Theta)\Theta\mid x]/E[w(\Theta)\mid x]$.
  3. Under absolute error loss, with $E[|\Theta|\mid x]\lt\infty$, any posterior median is a Bayes estimator.

Proof. (1) By the tower property (§1.4), applied to the joint law of $(\Theta,X)$, $$r(\pi,\delta)=E\left[L(\Theta,\delta(X))\right]=E\Big[E\left[L(\Theta,\delta(X))\mid X\right]\Big]=\int\rho\left(\delta(x)\mid x\right)m(x)\,dx.$$ If $\delta'$ is any other estimator, $\rho(\delta(x)\mid x)\le\rho(\delta'(x)\mid x)$ for every $x$, so $r(\pi,\delta)\le r(\pi,\delta')$.

(2) Write $\mu(x)=E[\Theta\mid x]$. Expanding around $\mu(x)$, the cross term has posterior mean $0$, so $$\rho(a\mid x)=E\left[(\Theta-\mu(x))^2\mid x\right]+\left(\mu(x)-a\right)^2=\operatorname{Var}(\Theta\mid x)+(\mu(x)-a)^2,$$ which is minimised exactly at $a=\mu(x)$. For the weighted loss, $\rho(a\mid x)$ equals $E[w\Theta^2\mid x]-2aE[w\Theta\mid x]+a^2E[w\mid x]$, a quadratic in $a$ with minimum at $a=E[w\Theta\mid x]/E[w\mid x]$.

(3) Let $F$ be the posterior cdf and $g(a)=E\left[|\Theta-a|\mid x\right]$. For an integrable variable, $E|\Theta-a|=\int_{-\infty}^aF(t)\,dt+\int_a^\infty\left(1-F(t)\right)dt$ (the tail formula of §1.3 applied to $(a-\Theta)^+$ and $(\Theta-a)^+$). Hence, for $a\lt b$, $$g(b)-g(a)=\int_a^bF(t)\,dt-\int_a^b(1-F(t))\,dt=\int_a^b\left(2F(t)-1\right)dt.$$ Let $m$ be a median: $F(m)\ge\frac12$ and $P(\Theta\lt m\mid x)\le\frac12$. For $t\ge m$, $2F(t)-1\ge0$, so $g(b)\ge g(m)$ for every $b\gt m$. For $t\lt m$, $F(t)\le P(\Theta\lt m\mid x)\le\frac12$, so $2F(t)-1\le0$ and $g(m)\le g(a)$ for every $a\lt m$. So $m$ minimises $g$; apply (1). ∎

For the conjugate families of Theorem 4.34, the Bayes estimators under squared error are therefore the shrinkage formulas $\frac{a+x}{a+b+n}$, $\frac{\alpha+\sum x_i}{\beta+n}$ and $\mu_n$. Under absolute error the normal posterior gives the same answer (mean $=$ median), but beta and gamma posteriors are skewed and their medians differ from their means; for $\operatorname{Gamma}(\alpha',\beta')$ with $\alpha'\ge1$ one always has mode $\lt$ median $\lt$ mean. Under the loss "$1$ if $|\theta-a|\gt\varepsilon$, $0$ otherwise", the Bayes estimator tends, as $\varepsilon\to0$, to the posterior mode (the MAP estimate) for smooth unimodal posteriors; with a flat prior the mode is the MLE.

📐 Theorem 4.36 — An unbiased estimator is never Bayes under squared error
Let $\pi$ be a proper prior and suppose $\delta(X)=E[\Theta\mid X]$ is the Bayes estimator under squared error, with $E[\Theta^2]\lt\infty$ and $E[\delta(X)^2]\lt\infty$. If $\delta$ is also unbiased, $E_\theta[\delta(X)]=\theta$ for all $\theta$, then its Bayes risk is zero: $E\left[(\delta(X)-\Theta)^2\right]=0$. Hence in any model where every estimator has positive risk on a set of positive prior probability, no unbiased estimator is a Bayes estimator under squared error with respect to a proper prior.

Proof. Compute $E[\delta(X)\Theta]$ in two ways, with expectations over the joint law of $(\Theta,X)$. Conditioning on $X$ and using $E[\Theta\mid X]=\delta(X)$: $E[\delta(X)\Theta]=E\left[\delta(X)E[\Theta\mid X]\right]=E[\delta(X)^2]$. Conditioning on $\Theta$ and using unbiasedness $E[\delta(X)\mid\Theta]=\Theta$: $E[\delta(X)\Theta]=E\left[\Theta E[\delta(X)\mid\Theta]\right]=E[\Theta^2]$. Therefore $$E\left[(\delta(X)-\Theta)^2\right]=E[\delta(X)^2]-2E[\delta(X)\Theta]+E[\Theta^2]=E[\delta(X)\Theta]-2E[\delta(X)\Theta]+E[\delta(X)\Theta]=0.$$ (All the expectations are finite by Cauchy–Schwarz.) ∎

So $\bar X$ is not a Bayes estimator of a normal mean for any proper prior (its Bayes risk $\sigma^2/n$ is positive), and $X/n$ is not Bayes for a binomial $p$ under any proper prior. Both are limits of Bayes estimators ($\tau_0\to\infty$; $a,b\to0$) and are "generalised Bayes" for improper priors. Conversely, every proper-prior posterior mean in the conjugate families is biased: $E_p\left[\frac{a+X}{a+b+n}\right]=\frac{a+np}{a+b+n}\ne p$ unless $p=\frac{a}{a+b}$.

📘 Definition 4.18 — Credible sets and highest posterior density regions
  1. A set $C(x)\subseteq\Theta$ is a $1-\alpha$ credible set if $P\left(\Theta\in C(x)\mid X=x\right)\ge1-\alpha$.
  2. The equal-tailed credible interval is $[q_{\alpha/2}(x),q_{1-\alpha/2}(x)]$, between the $\alpha/2$ and $1-\alpha/2$ quantiles of the posterior.
  3. The highest posterior density (HPD) region is $H(x)=\{\theta:\pi(\theta\mid x)\ge k\}$, with $k\gt0$ chosen so that $P(\Theta\in H(x)\mid x)=1-\alpha$.
📐 Theorem 4.37 — HPD regions are the shortest credible sets
  1. If $H$ is an HPD region with posterior probability $1-\alpha$ and $C$ is any set with $P(\Theta\in C\mid x)\ge1-\alpha$, then $\operatorname{length}(C)\ge\operatorname{length}(H)$.
  2. If the posterior density is unimodal, $H$ is an interval $[a,b]$ with $\pi(a\mid x)=\pi(b\mid x)$ (unless it reaches an endpoint of $\Theta$); if it is moreover symmetric, the HPD and equal-tailed intervals coincide.

Proof. (1) Write $\pi$ for $\pi(\cdot\mid x)$ and $|A|$ for the length (Lebesgue measure) of $A$. Since $\int_C\pi\ge1-\alpha=\int_H\pi$, subtracting $\int_{C\cap H}\pi$ from both sides gives $\int_{C\setminus H}\pi\ge\int_{H\setminus C}\pi$. On $C\setminus H$ we have $\pi\lt k$, and on $H\setminus C$ we have $\pi\ge k$. Hence $$k\,|C\setminus H|\ge\int_{C\setminus H}\pi\ge\int_{H\setminus C}\pi\ge k\,|H\setminus C|,$$ so $|C\setminus H|\ge|H\setminus C|$ (as $k\gt0$), and adding $|C\cap H|$ gives $|C|\ge|H|$. (2) For a unimodal density, a superlevel set $\{\pi\ge k\}$ is an interval containing the mode, and at an interior endpoint the density equals $k$ by continuity. For a symmetric unimodal density the interval is symmetric about the centre, so it cuts off equal tails. ∎

This is the Bayesian counterpart of Theorem 4.28, and the proof is the same idea as the Neyman–Pearson lemma of §5.1: to collect a fixed amount of probability in the least space, take the points where the density is highest.

Credible versus confidence intervals. A $95\%$ credible interval says "given these data and this prior, $\theta$ lies in $[a,b]$ with probability $0.95$", which is the statement many people wrongly attach to confidence intervals. The price is a prior. The two kinds of interval can agree numerically: with the flat prior in the normal model the posterior is $N(\bar x,\sigma^2/n)$, and the $95\%$ credible interval is exactly $\bar x\pm1.96\sigma/\sqrt n$. More generally, the Bernstein–von Mises theorem (statement only) says that in regular parametric models, for any prior with a continuous positive density at the true $\theta_0$, the posterior is approximately $N\left(\hat\theta_n,\ 1/(nI(\hat\theta_n))\right)$ for large $n$, in total variation, with probability tending to one. So for large samples credible intervals and Wald intervals nearly coincide, and the prior matters only for small or moderate $n$, or when it is very concentrated.

✏️ Example 7 — Bayes estimates and credible intervals
(a) An instrument measures a length $\theta$ with $N(0,4)$ errors ($\sigma=2$ known). Four measurements average $\bar x=10.6$. The prior, from the manufacturer's specification, is $\theta\sim N(9,1)$. Find the Bayes estimates under squared and absolute error, and the $95\%$ HPD interval. Compare with the $95\%$ confidence interval. (b) For the accident rate of Example 6(b), with posterior $\operatorname{Gamma}(25,8)$, find the Bayes estimates under squared and absolute error, the posterior mode, and the $95\%$ equal-tailed and HPD intervals. (c) Show that $\bar X$ is not a Bayes estimator of $\lambda$ under squared error, for any proper prior on $(0,\infty)$ with $E[\lambda^2]\lt\infty$, when $X_1,\dots,X_n$ are iid $\operatorname{Poi}(\lambda)$.
(a) Prior precision $1$, data precision $n/\sigma^2=4/4=1$, so $\tau_n^2=\frac12$ and $\mu_n=\frac12(9)+\frac12(10.6)=9.8$. The posterior $N(9.8,\ 0.5)$ is symmetric, so both Bayes estimates equal $9.8$, and the HPD interval is the equal-tailed one: $9.8\pm1.96\sqrt{0.5}=[8.414,\ 11.186]$. The confidence interval $10.6\pm1.96\times2/2=[8.640,\ 12.560]$ is centred at $\bar x$ and wider, of length $3.92$ against $2.77$: the prior is worth as much as four more measurements.
(b) Point estimates Squared error: the posterior mean $25/8=3.125$. Absolute error: the posterior median $3.0834$. The mode is $(25-1)/8=3.0$. As expected for a right-skewed posterior, mode $\lt$ median $\lt$ mean.
(b) Intervals The $0.025$ and $0.975$ posterior quantiles give the equal-tailed interval $[2.022,\ 4.464]$, of length $2.441$. The HPD interval, found numerically by sliding a window of probability $0.95$ until the density is equal at both ends, is $[1.951,\ 4.371]$, of length $2.420$, with $\pi(1.951\mid x)=\pi(4.371\mid x)=0.0938$. It is shifted left, towards the mode.
(c) $\bar X$ is unbiased for $\lambda$. If it were the posterior mean for a proper prior, Theorem 4.36 would give $E[(\bar X-\lambda)^2]=0$. But $E[(\bar X-\lambda)^2]=E\left[E[(\bar X-\lambda)^2\mid\lambda]\right]=E[\lambda/n]\gt0$, since $\lambda\gt0$. Contradiction. (Note that $\bar x$ is the limit of $\frac{\alpha+n\bar x}{\beta+n}$ as $\alpha,\beta\to0$.)
⚠️ Counterexamples and pitfalls
  1. "$P(2.1\le\mu\le3.4)=0.95$" is wrong for a realised confidence interval. Once computed, the interval is fixed and $\mu$ is fixed; the $95\%$ refers to the procedure over repeated samples. A probability statement about $\mu$ needs a prior: that is a credible interval.
  2. Asymptotic intervals have pointwise, not uniform, coverage. For each fixed $n$, the Wald interval for a proportion has confidence coefficient $0$: as $p\to0$, $P_p(X=0)=(1-p)^n\to1$, and when $x=0$ the interval is $\{0\}$, which misses $p$ (Exercise III.5).
  3. Using $z$ with an estimated $\sigma$ in small samples undercovers. $\bar x\pm1.96s/\sqrt n$ has exact coverage $P(|t_{n-1}|\le1.96)$, which is $0.878$ for $n=5$ and $0.918$ for $n=10$.
  4. Equal tails are not shortest for skewed pivots. For $\chi^2$ pivots and for $X_{(n)}/\theta$ in $U(0,\theta)$, the shortest interval has unequal tails (Examples 1 and 3); for scale parameters the right criterion is $a^2f(a)=b^2f(b)$, not $f(a)=f(b)$.
  5. The confidence level is an average over samples, not a conditional guarantee. For two observations from $U(\theta-\frac12,\theta+\frac12)$, $[X_{(1)},X_{(2)}]$ is a $50\%$ interval, yet it is certain to cover $\theta$ when the range exceeds $\frac12$ (Example 2).
  6. Overlapping intervals do not mean "no difference". If $\bar x_1=0$ and $\bar x_2=3$, each with standard error $1$, the individual $95\%$ intervals $[-1.96,1.96]$ and $[1.04,4.96]$ overlap, but the $95\%$ interval for the difference, $3\pm1.96\sqrt2=[0.23,5.77]$, excludes $0$.
  7. The pooled $t$ interval needs equal variances and independent samples. With unequal variances and unequal sample sizes its coverage can be far from nominal (use Welch); with paired data it uses the wrong variance (use differences).
  8. The $\chi^2$ interval for $\sigma^2$ is not robust. The $t$ interval for $\mu$ is approximately valid for non-normal data with large $n$ (CLT and Slutsky), but the $\chi^2$ interval for $\sigma^2$ is not: its large-sample coverage depends on the kurtosis, because $\operatorname{Var}(S^2)\approx(\mu_4-\sigma^4)/n$ (§2.3).
  9. A pivot may not contain nuisance parameters. $\sqrt n(\bar X-\mu)/\sigma$ is useless for $\mu$ when $\sigma$ is unknown; replacing $\sigma$ by $S$ changes the law from $N(0,1)$ to $t_{n-1}$.
  10. Bayes estimators are biased, and unbiased estimators are not Bayes under squared error with a proper prior (Theorem 4.36). Bias is no defect from the Bayesian point of view.
  11. The posterior mean, median and mode differ for skewed posteriors, and they answer different loss functions (Theorem 4.35); for $\operatorname{Beta}(3,1)$ they are $0.75$, $0.794$ and $1$.
  12. An improper prior can give an improper posterior. With the Haldane prior $\pi(p)\propto p^{-1}(1-p)^{-1}$ and $x=0$ successes, the posterior $\propto p^{-1}(1-p)^{n-1}$ is not integrable near $0$.
  13. "Flat" priors are not invariant. A uniform prior on $p$ is not uniform on the odds or the log-odds; Jeffreys' prior $\propto\sqrt{I(\theta)}$ is the parametrisation-invariant choice.
  14. Predictive probabilities are not plug-in probabilities. $P(Y=0\mid x)=\left(\frac{\beta'}{\beta'+1}\right)^{\alpha'}$ for the gamma–Poisson model is larger than $e^{-E[\lambda\mid x]}$, by Jensen's inequality (Example 6(b)).

Exercises

I
Pivots and exact intervals.
  • 1 $X_1,\dots,X_n$ are iid with density $\theta x^{\theta-1}$ on $(0,1)$, $\theta\gt0$. Show that $-2\theta\sum\log X_i\sim\chi^2_{2n}$, and find the equal-tailed $95\%$ interval for $\theta$ when $n=10$ and $\sum\log x_i=-6.2$.
  • 2 $X_1,\dots,X_n$ are iid $U(\theta,\theta+1)$. (a) Using the pivot $X_{(1)}-\theta$, find the shortest $1-\alpha$ interval of the form $[X_{(1)}-b,\ X_{(1)}-a]$. (b) Show that $[X_{(n)}-1,\ X_{(1)}]$ contains $\theta$ with probability $1$, and find its expected length. Compare the two lengths for $n=10$, $\alpha=0.05$.
  • 3 $X_1,\dots,X_n$ are iid with density $e^{-(x-\theta)}$ for $x\gt\theta$. Show that $n(X_{(1)}-\theta)\sim\operatorname{Exp}(1)$ and find the shortest $95\%$ confidence interval for $\theta$ based on this pivot.
  • 4 For $N(\mu,\sigma^2)$ data with $\sigma$ known, the interval $\left[\bar X-z_{\alpha/3}\frac{\sigma}{\sqrt n},\ \bar X+z_{2\alpha/3}\frac{\sigma}{\sqrt n}\right]$ also has exact level $1-\alpha$. Verify this, and find its length relative to the equal-tailed interval for $\alpha=0.05$.
  • 5 Let $f$ be the $\chi^2_{n-1}$ density. Show that among intervals $\left[(n-1)S^2/b,\ (n-1)S^2/a\right]$ with $\int_a^bf=1-\alpha$, the shortest satisfies $a^2f(a)=b^2f(b)$. Check numerically that the equal-tailed choice for $n=25$, $\alpha=0.05$ does not satisfy this condition.
  • 6 For the components of Example 1(b) ($n=10$, total time on test $1240$ hours), find a $95\%$ lower confidence bound for the mean lifetime $\theta$.
  • 7 (Rule of three) If $X\sim\operatorname{Bin}(n,p)$ and $x=0$, show that the exact one-sided $1-\alpha$ upper bound for $p$ given by Theorem 4.27 is $1-\alpha^{1/n}$, and that it is at most $-\log\alpha/n$. Evaluate both for $n=50$, $\alpha=0.05$.
  • 8 (Exact Poisson interval) For $X\sim\operatorname{Poi}(\lambda)$, use Theorem 4.27 and the identity $P_\lambda(X\ge x)=P\left(\operatorname{Gamma}(x,1)\le\lambda\right)$ $(x\ge1)$ of §2.1 to show that $\left[\frac12\chi^2_{2x,1-\alpha/2},\ \frac12\chi^2_{2x+2,\alpha/2}\right]$ is a confidence interval of level at least $1-\alpha$. Evaluate it for $x=7$, $\alpha=0.05$.
1 $P(-\log X\gt t)=P(X\lt e^{-t})=e^{-\theta t}$, so $-\log X_i\sim\operatorname{Exp}(\theta)$ and $2\theta\sum(-\log X_i)\sim\chi^2_{2n}$ (this is Theorem 4.26 with $F(x)=x^\theta$). With $2\sum(-\log x_i)=12.4$: $\theta\in\left[\frac{\chi^2_{20,0.975}}{12.4},\frac{\chi^2_{20,0.025}}{12.4}\right]=\left[\frac{9.5908}{12.4},\frac{34.1696}{12.4}\right]=[0.773,\ 2.756]$.
2 (a) $X_{(1)}-\theta$ is the minimum of $n$ iid $U(0,1)$: density $n(1-t)^{n-1}$, decreasing on $[0,1]$. For a density that is decreasing on its support $[0,1]$, sliding an interval to the left never lowers its probability, so the shortest interval of given probability starts at $0$; hence $a=0$ and $1-(1-b)^n=1-\alpha$, i.e. $b=1-\alpha^{1/n}$: $\theta\in\left[X_{(1)}-(1-\alpha^{1/n}),\ X_{(1)}\right]$. (b) Every $X_i\in(\theta,\theta+1)$, so $X_{(1)}\gt\theta$ and $X_{(n)}-1\lt\theta$ always. $E[X_{(1)}]=\theta+\frac1{n+1}$ and $E[X_{(n)}]=\theta+\frac{n}{n+1}$, so the expected length is $1-\frac{n-1}{n+1}=\frac{2}{n+1}$. For $n=10$: $1-0.05^{0.1}=0.2589$ versus $0.1818$. The $100\%$ interval is shorter on average than the $95\%$ one, because the pivot in (a) ignores $X_{(n)}$, which also carries information. ($(X_{(1)},X_{(n)})$ is minimal sufficient.)
3 $P(n(X_{(1)}-\theta)\gt t)=P(X_1-\theta\gt t/n)^n=(e^{-t/n})^n=e^{-t}$. The $\operatorname{Exp}(1)$ density is decreasing, so the shortest interval for the pivot is $[0,-\log0.05]=[0,2.996]$, and $\theta\in\left[X_{(1)}-\frac{2.996}{n},\ X_{(1)}\right]$.
4 $P\left(-z_{2\alpha/3}\le Z\le z_{\alpha/3}\right)=1-\frac{2\alpha}{3}-\frac{\alpha}{3}=1-\alpha$ with $Z=\sqrt n(\bar X-\mu)/\sigma$, which rearranges to the interval. Its length is $(z_{0.01667}+z_{0.03333})\frac{\sigma}{\sqrt n}=(2.1280+1.8339)\frac{\sigma}{\sqrt n}=3.9620\frac{\sigma}{\sqrt n}$, against $3.9199\frac{\sigma}{\sqrt n}$: about $1.07\%$ longer, as Theorem 4.28 predicts.
5 Minimise $\frac1a-\frac1b$ subject to $G(a,b)=F(b)-F(a)-(1-\alpha)=0$. Differentiating the constraint, $f(b)\,db=f(a)\,da$, so along the constraint $\frac{d}{da}\left(\frac1a-\frac1b\right)=-\frac{1}{a^2}+\frac{1}{b^2}\frac{f(a)}{f(b)}$, which vanishes iff $a^2f(a)=b^2f(b)$. For $n-1=24$, the equal-tailed points $a=12.401$, $b=39.364$ give $a^2f(a)=2.04$ and $b^2f(b)=9.44$, far from equal; the shortest choice is $a=13.52$, $b=44.48$ (Example 3).
6 $P(2T/\theta\le\chi^2_{20,0.05})=0.95$ gives $\theta\ge2T/\chi^2_{20,0.05}=2480/31.4104=78.95$ hours.
7 With $T=X$ and $\alpha_1=\alpha$, $\alpha_2=0$, the upper end solves $P_p(X\le0)=(1-p)^n=\alpha$, so $p_U=1-\alpha^{1/n}$. Since $1-e^{-c}\le c$ for $c\ge0$, $1-\alpha^{1/n}=1-e^{\log\alpha/n}\le-\log\alpha/n$. For $n=50$: $1-0.05^{0.02}=0.0582$ against $2.996/50=0.0599\approx3/50=0.06$.
8 $\operatorname{Poi}(\lambda)$ is stochastically increasing in $\lambda$. The lower end solves $P_\lambda(X\ge x)=\frac\alpha2$, i.e. $P(\operatorname{Gamma}(x,1)\le\lambda)=\frac\alpha2$, so $\lambda_L$ is the lower $\frac\alpha2$ point of $\operatorname{Gamma}(x,1)=\frac12\chi^2_{2x}$: $\lambda_L=\frac12\chi^2_{2x,1-\alpha/2}$. The upper end solves $P_\lambda(X\le x)=\frac\alpha2$, i.e. $P_\lambda(X\ge x+1)=1-\frac\alpha2$, so $\lambda_U$ is the upper $\frac\alpha2$ point of $\frac12\chi^2_{2x+2}$. For $x=7$: $\left[\frac{5.6287}{2},\frac{28.8454}{2}\right]=[2.814,\ 14.423]$.
II
Normal-theory intervals.
  • 1 A sample of $16$ from a normal population has $\bar x=27.3$ and $s=3.6$. Find a $99\%$ confidence interval for $\mu$.
  • 2 For the same data, find a $95\%$ confidence interval for $\sigma^2$.
  • 3 Blood glucose measurements have known $\sigma=12$ mg/dL. How many patients are needed for a $95\%$ interval for the mean of half-width at most $2$ mg/dL?
  • 4 For $n=5$ and $\alpha=0.05$, compute $c_5=E[S]/\sigma$ and the ratio of the expected length of the $t$ interval to the length of the $z$ interval with $\sigma$ known.
  • 5 Eight patients' systolic blood pressures are measured before and after a drug. The differences (before − after) have $\bar d=6.25$ and $s_d=4.2$ mmHg. Find a $95\%$ interval for the mean reduction, and explain why the two-sample interval would be inappropriate.
  • 6 Independent normal samples with a common variance give $m=8$, $\bar x=15.2$, $s_1=2.1$ and $n=10$, $\bar y=13.1$, $s_2=2.5$. Find the $95\%$ pooled interval for $\mu_1-\mu_2$.
  • 7 For the data of part 6, find a $90\%$ interval for $\sigma_1^2/\sigma_2^2$.
  • 8 For $n=10$ and $\alpha=0.05$, find the probability that the $t$ interval is shorter than the $z$ interval that uses the true $\sigma$.
Check lab: normal-theory intervals (Theorem 4.29)

Enter a sample size, mean and standard deviation; the lab computes the $t$ interval for $\mu$, the $\chi^2$ interval for $\sigma^2$ and the quantiles used. Use it to check Exercises II.1–2 and Example 3.

t quantile; interval for μ—
χ² points (upper, lower)—
interval for σ²; for σ—
1 $t_{15,0.005}=2.9467$ and $s/\sqrt n=0.9$, so $27.3\pm2.652=[24.65,\ 29.95]$.
2 $15s^2=194.4$, $\chi^2_{15,0.025}=27.4884$, $\chi^2_{15,0.975}=6.2621$: $\sigma^2\in[7.072,\ 31.04]$.
3 $n\ge(1.96\times12/2)^2=138.3$, so $139$ patients.
4 $c_5=\sqrt{\frac24}\,\frac{\Gamma(5/2)}{\Gamma(2)}=0.7071\times1.3293=0.9400$, and the ratio is $\frac{t_{4,0.025}\,c_5}{z_{0.025}}=\frac{2.7764\times0.9400}{1.9600}=1.33$. Not knowing $\sigma$ costs a third more length on average at $n=5$.
5 $t_{7,0.025}=2.3646$, so $6.25\pm2.3646\times4.2/\sqrt8=6.25\pm3.511=[2.74,\ 9.76]$ mmHg. Before and after values on the same patient are correlated; the two-sample formula assumes independent samples and would use $\sigma_1^2+\sigma_2^2$ instead of the much smaller $\operatorname{Var}(D)$.
6 $s_p^2=\frac{7\times4.41+9\times6.25}{16}=5.445$, $s_p=2.3335$, standard error $2.3335\sqrt{\frac18+\frac1{10}}=1.1069$, $t_{16,0.025}=2.1199$: $2.1\pm2.346=[-0.25,\ 4.45]$, which contains $0$.
7 $R=4.41/6.25=0.7056$, $F_{7,9,0.05}=3.2927$, $F_{9,7,0.05}=3.6767$: $[0.7056/3.2927,\ 0.7056\times3.6767]=[0.214,\ 2.594]$.
8 $t_{9,0.025}S\lt1.96\sigma$ iff $\frac{9S^2}{\sigma^2}\lt9\left(\frac{1.96}{2.2622}\right)^2=6.756$, and $P(\chi^2_9\lt6.756)=0.337$. So about one sample in three gives a $t$ interval shorter than the "ideal" $z$ interval, although on average it is longer.
III
Large-sample intervals and proportions.
  • 1 In a survey, $540$ of $1200$ voters favour a candidate. Find the $95\%$ Wald interval for the proportion $p$.
  • 2 In a pilot study $x=3$ of $n=25$ seeds fail to germinate. Compute the $95\%$ Wald and Wilson intervals and comment.
  • 3 A side effect occurs in $45$ of $200$ patients on drug A and in $30$ of $180$ on drug B. Find a $95\%$ interval for $p_A-p_B$.
  • 4 Ten one-minute counts of a radioactive source total $400$. Assuming Poisson counts, find the $95\%$ Wald and variance-stabilised intervals for the rate $\lambda$ per minute, and compare them with the exact interval $[36.18,\ 44.12]$.
  • 5 Prove that for every $n$ the confidence coefficient of the Wald interval for a binomial $p$ is $0$.
  • 6 Fifty exponential lifetimes have mean $\bar x=3.2$ years. Compare the $95\%$ Wald interval for the mean $\theta$, the interval obtained from $\log\bar X$ by Theorem 4.31(3), and the exact $\chi^2$ interval.
  • 7 How large a sample is needed for a $95\%$ margin of error of at most $0.02$ for a proportion (a) with no information about $p$, (b) if it is known that $p\le0.1$?
  • 8 Using $\sqrt n(S^2-\sigma^2)\xrightarrow{d}N(0,2\sigma^4)$ for normal data (§2.3), find a variance-stabilising transformation of $S^2$ and an asymptotic $95\%$ interval for $\sigma$ when $n=100$, $s=5$. Compare with the exact interval $[4.390,\ 5.808]$.
1 $\hat p=0.45$, standard error $\sqrt{0.45\times0.55/1200}=0.01436$, margin $0.0281$: $[0.422,\ 0.478]$.
2 $\hat p=0.12$, Wald: $0.12\pm1.96\times0.065=[-0.007,\ 0.247]$, whose lower end is negative, an impossible value. Wilson: $[0.042,\ 0.300]$, inside $(0,1)$ and shifted right. With only $3$ failures the normal approximation behind Wald is poor; Wilson (or Clopper–Pearson) should be used.
3 $\hat p_A=0.225$, $\hat p_B=0.1667$, standard error $\sqrt{\frac{0.225\times0.775}{200}+\frac{0.1667\times0.8333}{180}}=0.04054$: $0.0583\pm0.0795=[-0.021,\ 0.138]$. The data do not establish a difference.
4 $\bar x=40$. Wald: $40\pm1.96\sqrt{40/10}=[36.08,\ 43.92]$. Variance-stabilised: $\left(\sqrt{40}\pm\frac{1.96}{2\sqrt{10}}\right)^2=(6.3246\pm0.3099)^2=[36.18,\ 44.02]$. The square-root interval is asymmetric like the exact one $[36.18,\ 44.12]$ and closer to it.
5 For $X=0$ the Wald interval is $\{0\}$, which does not contain any $p\gt0$. So its coverage at $p$ is at most $1-P_p(X=0)=1-(1-p)^n$, which tends to $0$ as $p\downarrow0$. Hence $\inf_{0\lt p\lt1}(\text{coverage})=0$, although the coverage tends to $1-\alpha$ for each fixed $p$ as $n\to\infty$.
6 Wald: $3.2\pm1.96\times3.2/\sqrt{50}=[2.313,\ 4.087]$. Log scale ($g=\log$, $g'(\theta)\tau(\theta)=\frac1\theta\cdot\theta=1$): $3.2e^{\pm1.96/\sqrt{50}}=[2.425,\ 4.222]$. Exact: $2n\bar x=320$ and $\left[\frac{320}{\chi^2_{100,0.025}},\frac{320}{\chi^2_{100,0.975}}\right]=\left[\frac{320}{129.561},\frac{320}{74.222}\right]=[2.470,\ 4.311]$. The log interval captures most of the right skew; Wald does not.
7 (a) $p(1-p)\le\frac14$, so $n\ge\left(\frac{1.96}{2\times0.02}\right)^2=2400.9$, i.e. $2401$. (b) $p(1-p)\le0.09$ on $[0,0.1]$, so $n\ge\frac{1.96^2\times0.09}{0.0004}=864.3$, i.e. $865$.
8 $\tau(\sigma^2)=\sqrt2\sigma^2$, so $g(v)=\frac{1}{\sqrt2}\log v$ has $g'\tau=1$; equivalently $\log S^2\pm z\sqrt{2/n}$ is an interval for $\log\sigma^2$. With $\log25=3.2189$ and $1.96\sqrt{0.02}=0.2772$: $\sigma^2\in[18.95,\ 32.99]$, so $\sigma\in[4.353,\ 5.743]$, close to the exact $[4.390,\ 5.808]$. (The Wald interval $25\pm1.96\times25\sqrt{0.02}$ gives $\sigma^2\in[18.07,\ 31.93]$, symmetric and worse.)
IV
Bayesian inference.
  • 1 With the uniform prior $\operatorname{Beta}(1,1)$, $9$ successes are observed in $10$ Bernoulli trials. Find the posterior and the predictive probability that trial $11$ is a success. Compare with the plug-in value $\hat p=0.9$.
  • 2 $X_1,\dots,X_n\mid\lambda$ are iid $\operatorname{Exp}(\lambda)$ and $\lambda\sim\operatorname{Gamma}(\alpha,\beta)$. Find the posterior, and the Bayes estimators under squared error of $\lambda$ and of the mean lifetime $1/\lambda$.
  • 3 $X_1,\dots,X_9\mid\theta$ are iid $N(\theta,9)$, the prior is $N(50,4)$, and $\bar x=53$. Find the posterior and a $95\%$ credible interval for $\theta$.
  • 4 With a uniform prior, $2$ successes occur in $2$ trials. Find the Bayes estimates of $p$ under squared and absolute error, and the posterior mode.
  • 5 Show that the Bayes risk of the posterior mean under squared error is $E\left[\operatorname{Var}(\Theta\mid X)\right]$. For $X\sim\operatorname{Bin}(n,p)$ with $p\sim U(0,1)$, show that the Bayes risk of $\frac{X+1}{n+2}$ is $\frac{1}{6(n+2)}$, while that of $\frac Xn$ is $\frac{1}{6n}$.
  • 6 Prove that in the model $X_i\mid\theta\sim N(\theta,1)$ iid, $\bar X$ is not a Bayes estimator under squared error with respect to any proper prior with finite variance.
  • 7 (Asymmetric loss) Let $L(\theta,a)=c_1(\theta-a)$ if $\theta\gt a$ and $c_2(a-\theta)$ if $\theta\le a$, with $c_1,c_2\gt0$. Prove that the Bayes estimator is a $\frac{c_1}{c_1+c_2}$ quantile of the posterior. What does it say for an inventory manager for whom running short costs three times as much as overstocking?
  • 8 A posterior is $\operatorname{Exp}(10)$ (rate $10$). Find the $95\%$ HPD interval and the $95\%$ equal-tailed interval, and their lengths.
  • 9 With the improper Haldane prior $\pi(p)\propto p^{-1}(1-p)^{-1}$ and $X\sim\operatorname{Bin}(n,p)$, show that the posterior is proper if and only if $0\lt x\lt n$, and that its mean is then the MLE $x/n$.
1 $p\mid x\sim\operatorname{Beta}(10,2)$, and $P(X_{11}=1\mid x)=\frac{10}{12}=0.833$ (Laplace's rule $\frac{x+1}{n+2}$), below the plug-in $0.9$: the uniform prior acts like one extra success and one extra failure.
2 Theorem 4.34(2): $\lambda\mid x\sim\operatorname{Gamma}(\alpha+n,\ \beta+\sum x_i)$. Under squared error, $\hat\lambda=\frac{\alpha+n}{\beta+\sum x_i}$. For $1/\lambda$: if $\lambda\sim\operatorname{Gamma}(a,b)$ with $a\gt1$, then $E[1/\lambda]=\int_0^\infty\frac{b^a\lambda^{a-2}e^{-b\lambda}}{\Gamma(a)}d\lambda=\frac{b\,\Gamma(a-1)}{\Gamma(a)}=\frac{b}{a-1}$, so the Bayes estimate of the mean is $\frac{\beta+\sum x_i}{\alpha+n-1}$ (for $\alpha+n\gt1$). Note that it is not $1/\hat\lambda$: Bayes estimators are not invariant under nonlinear reparametrisation.
3 Precisions: prior $\frac14$, data $\frac{9}{9}=1$; $\tau_n^2=\frac{1}{1.25}=0.8$ and $\mu_n=0.8\left(\frac{50}{4}+53\right)=52.4$. So $\theta\mid x\sim N(52.4,\ 0.8)$ and the $95\%$ credible interval is $52.4\pm1.96\sqrt{0.8}=[50.65,\ 54.15]$.
4 Posterior $\operatorname{Beta}(3,1)$, density $3p^2$, cdf $p^3$. Mean $\frac34$; median $2^{-1/3}=0.794$; mode $1$. The three answers differ because the posterior is skewed (to the left here).
5 By the tower property and Theorem 4.35(2), $r(\pi,\delta)=E\left[E[(\Theta-E[\Theta\mid X])^2\mid X]\right]=E[\operatorname{Var}(\Theta\mid X)]$. Under the uniform prior, $P(X=x)=\int_0^1\binom nxp^x(1-p)^{n-x}dp=\frac1{n+1}$ for each $x$, and $\operatorname{Var}(p\mid x)=\frac{(x+1)(n-x+1)}{(n+2)^2(n+3)}$. Since $\sum_{x=0}^n(x+1)(n-x+1)=\frac{(n+1)(n+2)(n+3)}{6}$, the average is $\frac{(n+2)(n+3)}{6}\cdot\frac{1}{(n+2)^2(n+3)}=\frac{1}{6(n+2)}$. For $X/n$: $r=E\left[\frac{p(1-p)}{n}\right]=\frac1n\left(\frac12-\frac13\right)=\frac{1}{6n}$, larger, as it must be.
6 $\bar X$ is unbiased and $E[(\bar X-\theta)^2]=E\left[E[(\bar X-\theta)^2\mid\theta]\right]=\frac1n\gt0$. If $\bar X$ were the posterior mean for a proper prior with $E[\theta^2]\lt\infty$ (then also $E[\bar X^2]=E[\theta^2]+\frac1n\lt\infty$), Theorem 4.36 would force this Bayes risk to be $0$.
7 With $F$ the posterior cdf, $\rho(a)=c_1\int_a^\infty(1-F)+c_2\int_{-\infty}^aF$, so as in Theorem 4.35(3), $\rho(b)-\rho(a)=\int_a^b\left[(c_1+c_2)F(t)-c_1\right]dt$. The integrand is $\le0$ below and $\ge0$ above any point $q$ with $P(\Theta\lt q)\le\frac{c_1}{c_1+c_2}\le F(q)$, so such a quantile minimises $\rho$. With shortage cost $c_1=3$ and overstock cost $c_2=1$, stock the $75\%$ quantile of the posterior (or predictive) demand: this is the "newsvendor" rule.
8 The density $10e^{-10\theta}$ is decreasing, so the HPD region is $[0,c]$ with $1-e^{-10c}=0.95$: $[0,\ 0.2996]$, of length $0.2996$. Equal-tailed: $\left[-\frac{\log0.975}{10},\ -\frac{\log0.025}{10}\right]=[0.00253,\ 0.3689]$, of length $0.3664$, more than $20\%$ longer.
9 Likelihood $\times$ prior $\propto p^{x-1}(1-p)^{n-x-1}$, which is integrable on $(0,1)$ iff $x-1\gt-1$ and $n-x-1\gt-1$, i.e. $0\lt x\lt n$. The posterior is then $\operatorname{Beta}(x,n-x)$ with mean $\frac{x}{n}$. For $x=0$ or $x=n$ the "posterior" does not exist, a warning that improper priors need checking.
🎯 Exam corner CSIR Part BCSIR Part CGATE ST

E1. CSIR Part B $X_1,\dots,X_5$ are iid $U(0,\theta)$, $\theta\gt0$. The constant $c$ for which $[X_{(5)},\ cX_{(5)}]$ is a $95\%$ confidence interval for $\theta$ is (A) $(0.95)^{-1/5}$; (B) $20^{1/5}$; (C) $1.95$; (D) $e^{0.05}$.

Idea $X_{(5)}/\theta$ is a pivot with cdf $t^5$ on $[0,1]$, and $\{X_{(5)}\le\theta\le cX_{(5)}\}=\{1/c\le X_{(5)}/\theta\le1\}$.
Compute $P(1/c\le X_{(5)}/\theta\le1)=1-c^{-5}=0.95$ gives $c^{-5}=0.05$, so $c=20^{1/5}\approx1.8206$. Option (A) comes from setting $1-c^{-5}$ equal to $0.05$ instead.
Answer (B).

E2. CSIR Part B GATE ST The success probability $p$ of a Bernoulli experiment has prior $\operatorname{Beta}(2,3)$. In $10$ independent trials, $7$ successes are observed. The Bayes estimate of $p$ under squared error loss is (A) $0.70$; (B) $0.60$; (C) $8/13$; (D) $0.40$.

Idea Posterior $\operatorname{Beta}(2+7,\ 3+3)=\operatorname{Beta}(9,6)$; squared error loss gives the posterior mean.
Compute $\frac{9}{15}=0.6$. Option (A) is the MLE, (C) the posterior mode $\frac{9-1}{15-2}$, (D) the prior mean.
Answer (B).

E3. CSIR Part B A random sample of size $16$ from $N(\mu,\sigma^2)$, both parameters unknown, has sample standard deviation $s=4$ (divisor $n-1$). The length of the usual $95\%$ confidence interval for $\mu$ is (A) $2\times1.96$; (B) $2\,t_{15,0.025}$; (C) $2\,t_{16,0.025}$; (D) $8\,t_{15,0.025}$.

Idea Theorem 4.29(2): length $2t_{n-1,\alpha/2}\,s/\sqrt n$.
Compute $s/\sqrt n=4/4=1$, so the length is $2t_{15,0.025}=4.263$. (A) uses the normal quantile although $\sigma$ is unknown, (C) has the wrong degrees of freedom, and (D) forgets to divide by $\sqrt n$.
Answer (B).

E4. CSIR Part C Let $X_1,\dots,X_n$ ($n\ge2$) be iid $N(\mu,\sigma^2)$ with both parameters unknown, and let $[L,U]=\bar X\pm t_{n-1,0.025}\,S/\sqrt n$. Which of the following are true? (One or more options may be correct.) (A) $P_{\mu,\sigma}(L\le\mu\le U)=0.95$ for all $\mu$ and $\sigma$. (B) $E[U-L]=2t_{n-1,0.025}\,\sigma/\sqrt n$. (C) For every sample, the $99\%$ $t$ interval contains the $95\%$ $t$ interval. (D) The interval $\bar X\pm1.96\,S/\sqrt n$ also has coverage exactly $0.95$.

(A) True $\sqrt n(\bar X-\mu)/S\sim t_{n-1}$ exactly, whatever $(\mu,\sigma)$ (Theorem 4.29).
(B) False $E[S]=c_n\sigma$ with $c_n\lt1$ by Jensen's inequality, so $E[U-L]=2t_{n-1,0.025}c_n\sigma/\sqrt n$ is strictly smaller than the stated value.
(C) True Both intervals are centred at $\bar x$ and $t_{n-1,0.005}\gt t_{n-1,0.025}$, so the half-widths are ordered for the same $s$.
(D) False Its coverage is $P(|t_{n-1}|\le1.96)\lt0.95$, because $t_{n-1,0.025}\gt1.96$; e.g. $0.878$ for $n=5$.
Answer (A) and (C).

E5. CSIR Part C GATE ST Given $\lambda$, $X_1,\dots,X_n$ are iid $\operatorname{Poi}(\lambda)$, and the prior of $\lambda$ is $\operatorname{Gamma}(\alpha,\beta)$ with density proportional to $\lambda^{\alpha-1}e^{-\beta\lambda}$, where $\alpha\ge1$. Which of the following are true? (A) The posterior is $\operatorname{Gamma}\left(\alpha+\sum x_i,\ \beta+n\right)$. (B) The Bayes estimator under squared error is a weighted average of $\bar x$ and $\alpha/\beta$, with weight $n/(n+\beta)$ on $\bar x$. (C) The Bayes estimator under squared error is an unbiased estimator of $\lambda$. (D) The Bayes estimate under absolute error loss is smaller than the Bayes estimate under squared error loss.

(A) True Theorem 4.34(2): $\lambda^{\sum x_i}e^{-n\lambda}\cdot\lambda^{\alpha-1}e^{-\beta\lambda}$ is a gamma kernel.
(B) True $\frac{\alpha+n\bar x}{\beta+n}=\frac{n}{n+\beta}\bar x+\frac{\beta}{n+\beta}\cdot\frac\alpha\beta$.
(C) False $E_\lambda\left[\frac{\alpha+n\bar X}{\beta+n}\right]=\frac{\alpha+n\lambda}{\beta+n}\ne\lambda$ unless $\lambda=\alpha/\beta$. (Theorem 4.36 says no proper-prior Bayes estimator here can be unbiased.)
(D) True Absolute error gives the posterior median, squared error the mean, and for every gamma law the median is below the mean (gamma laws are right-skewed; e.g. $\operatorname{Gamma}(25,8)$ has median $3.083$ and mean $3.125$).
Answer (A), (B) and (D).

E6. CSIR Part C Let $X_1,\dots,X_{10}$ be iid exponential with mean $\theta\gt0$, and $T=\sum X_i$. Which of the following are true? (A) $2T/\theta\sim\chi^2_{20}$. (B) $X_{(1)}/\theta$ is exponential with mean $1$. (C) The equal-tailed interval $\left[2T/\chi^2_{20,0.025},\ 2T/\chi^2_{20,0.975}\right]$ has the smallest length among all $95\%$ intervals of the form $[2T/b,\ 2T/a]$. (D) The expected length of the interval in (C) is smaller than that of the equal-tailed $95\%$ interval built from the pivot $10X_{(1)}/\theta$.

(A) True $X_i/\theta\sim\operatorname{Exp}(1)$, so $2T/\theta\sim\operatorname{Gamma}(10,\tfrac12)=\chi^2_{20}$.
(B) False The minimum of $10$ iid exponentials with mean $\theta$ is exponential with mean $\theta/10$; it is $10X_{(1)}/\theta$ that is $\operatorname{Exp}(1)$.
(C) False The shortest interval of this form satisfies $a^2f(a)=b^2f(b)$ for the $\chi^2_{20}$ density, which the equal-tailed points do not: the shortest uses $a=10.61$, $b=39.56$ (Example 1(b)).
(D) True The interval in (C) has expected length $20\theta\left(\frac{1}{9.5908}-\frac{1}{34.1696}\right)=1.50\theta$. From $Q=10X_{(1)}/\theta\sim\operatorname{Exp}(1)$ with $a=-\log0.975=0.0253$ and $b=-\log0.025=3.689$, the interval $[10X_{(1)}/b,\ 10X_{(1)}/a]$ has expected length $\theta\left(\frac1a-\frac1b\right)=39.2\theta$. A pivot built on a sufficient statistic does far better.
Answer (A) and (D).
🧑‍🏫 For the teacher

Suggested plan: 3 lectures × 60 min.

  1. Confidence sets and pivots. Definition 4.13 and the interpretation, then Interactive 1 projected with the $t$ rule (watch the coverage settle at $95\%$ while individual intervals hit and miss). Definition 4.14 and Theorem 4.24; Theorems 4.25 and 4.26 with one line each; Example 1 on the board. Theorem 4.28 with the full proof (draw the density and the two intervals). Example 2 as a class discussion. Theorem 4.27 can be stated and illustrated with Clopper–Pearson, proof set as reading.
  2. Normal and large-sample intervals. Theorem 4.29 (prove (2) and (3), mention $c_n$), Theorem 4.30 (prove (2) and (3)), the paired case, Example 3 and Example 4(b). Then Theorem 4.31 via Slutsky, Theorem 4.32 with the Wilson derivation, Example 5. Return to Interactive 1 with the Wald rule at $p=0.05$, $n=20$, then the Wilson rule.
  3. Bayesian inference. Definition 4.15, Theorem 4.33, the three conjugate pairs of Theorem 4.34 (do the normal–normal computation in full), Example 6(a) and Interactive 2 with a sceptical and a uniform prior. Definition 4.17 and Theorem 4.35 (prove (2) and (3)), Theorem 4.36, credible and HPD intervals with Theorem 4.37, Example 7. Finish with E4–E6.

Discussion questions. (1) A newspaper writes: "there is a $95\%$ chance that the true support lies between $42\%$ and $48\%$." Is this a confidence statement or a credible statement, and what would make it correct? (2) Example 2 shows that some samples carry more information than others. Should a reported confidence level depend on the observed sample? (3) Two analysts with different priors analyse the same clinical trial. Under what conditions will their posteriors agree, and how many patients would that take in Example 6(a)?

Misconceptions to watch for. Reading a realised confidence interval as a probability statement about the parameter; using $z$ instead of $t$ with an estimated $\sigma$; forgetting to reverse the inequalities when inverting a decreasing pivot (the $\chi^2$ interval for $\sigma^2$); assuming equal tails give the shortest interval; using the Wald interval for small counts; pooling variances or ignoring pairing without checking the design; believing that the posterior mean should be unbiased; confusing the posterior mean, median and mode; treating a flat prior as "no information" in every parametrisation.

Suggested homework. I(1, 2, 5, 8), II(1, 2, 5, 8), III(2, 4, 5, 6), IV(2, 3, 5, 7, 9); Exam corner E1, E4, E5.

🔬 Research–Teaching NexusResearch-tutored · optional, about 60 minutes

Research question. How well do the standard $95\%$ intervals for a binomial proportion actually cover, as functions of $n$ and $p$? Is the textbook rule "use the Wald interval when $np\ge5$ and $n(1-p)\ge5$" a safe guide?

Why it matters. Proportions are everywhere: the rate of a rare adverse event in a vaccine trial, the share of voters in a poll, the defect rate on a production line, the prevalence of an infection in a district survey. Regulators and newspapers report these with $95\%$ intervals. An interval that claims $95\%$ but covers $88\%$ of the time understates uncertainty, and one that covers $99\%$ wastes sample size. Because the binomial law is discrete, the coverage of every method oscillates with $p$ and $n$ in a surprisingly irregular way.

What to do.

  1. For $n\in\{10,20,50,100\}$ and a fine grid of $p$ in $(0,1)$, compute the exact coverage $\sum_{x}\binom nxp^x(1-p)^{n-x}\mathbf{1}\{p\in C(x)\}$ of the Wald, Wilson, Agresti–Coull and Clopper–Pearson intervals. No simulation is needed.
  2. Plot coverage against $p$ for each $n$. Mark where $np\ge5$ and $n(1-p)\ge5$ hold, and check whether the Wald interval behaves there.
  3. Record the minimum and the average coverage over $p$, and the average length, for each method and each $n$.
  4. Use Interactive 1 to watch one bad case in repeated sampling, and explain in a paragraph why the coverage jumps as $p$ crosses the endpoints $C(x)$.

Tools: Python (numpy, scipy.stats) or R; Interactive 1 of this section; Theorem 4.27 for Clopper–Pearson.

Hand in: Coverage plots for the four methods, a table of minimum and average coverage and average length, and a one-page recommendation for a pharmacovigilance team that must report rates of rare events from samples of $50$ to $500$ patients.

Going further. Read about the systematic comparisons of binomial intervals in the statistical literature of the early 2000s, which made the Wilson and Agresti–Coull intervals standard; about Jeffreys' interval (the equal-tailed credible interval from the $\operatorname{Beta}(\frac12,\frac12)$ prior), whose frequentist coverage is excellent; and about the general idea of probability matching priors, priors for which credible intervals have nearly the right frequentist coverage.

Instructor note: our exact computations give, at $n=20$: Wald $0.876$, Wilson $0.957$, Clopper–Pearson $0.989$ for $p=0.1$, and Wald only $0.639$ for $p=0.05$. At $n=100$, $p=0.01$: Wald $0.633$, Wilson $0.921$, Clopper–Pearson $0.982$. Even at $n=50$, $p=0.5$, where every rule of thumb is satisfied, the Wald coverage is $0.935$ (Wilson also $0.935$, Clopper–Pearson $0.967$). Over a fine grid of $p\in[0.005,0.995]$, the minimum coverage at $n=100$ was about $0.39$ for Wald, $0.89$ for Wilson and $0.95$ for Clopper–Pearson, with average coverages $0.929$, $0.951$ and $0.964$. Students should discover that the average coverage of Wilson is close to $95\%$ even though its minimum is not.

🧪 Self-check quiz

0/5 correct

Q1. A $95\%$ confidence interval for $\mu$ computed from a sample is $[2.1,\ 3.4]$. Which statement is correct?

Q2. For iid $U(0,\theta)$ data, which of the following is a pivotal quantity?

Q3. With a uniform prior on $p$, $3$ successes are observed in $4$ Bernoulli trials. The posterior mean of $p$ is

Q4. Under absolute error loss $|\theta-a|$, the Bayes estimator is

Q5. $X_1,\dots,X_4\mid\theta$ are iid $N(\theta,1)$ and $\theta\sim N(0,1)$. The posterior variance of $\theta$ is