Chapter 4 · Sampling Distributions and Estimation

4.1 Sampling Distributions and Order Statistics

Every statistic is a random variable with a law of its own. For normal samples the laws are exact, namely $\chi^2$, $t$ and $F$. For large samples the CLT and the delta method give normal approximations. For order statistics a counting argument gives everything, from the density of the median to the law of the range.

A quality engineer weighs $n=25$ tablets and finds an average of $\bar x=502.3$ mg and a standard deviation of $s=4.1$ mg. The question that matters is not what these numbers are but how far they could be from the truth. If the production line were sampled again, $\bar x$ and $s$ would change. They are values of the statistics $\bar X$ and $S$, which are functions of random variables and therefore random variables themselves. Their distributions, the sampling distributions, are the bridge between probability and inference. Every confidence interval of §4.3, every test of Chapter 5 and every comparison of estimators in §4.2 is a statement about a sampling distribution.

This section computes the sampling distributions that statistics uses most. Means and variances of $\bar X$ and $S^2$ hold for every population. For normal populations there is an exact theory: $\bar X$ and $S^2$ are independent, $(n-1)S^2/\sigma^2$ is $\chi^2_{n-1}$, and the studentised mean has Student's $t$ law. For other populations, the central limit theorem and the delta method of §2.3 supply asymptotic laws and standard errors. Finally, order statistics, the sorted sample values, have laws that follow from a single counting argument, and uniform and exponential samples give especially clean answers. CSIR Part B questions often ask you to identify a $\chi^2$, $t$ or $F$ law or to compute the mean of an order statistic. Part C questions ask which statements about independence and distribution are true, and many of the false options are statements that hold only for normal samples.

Statistics, Sampling Distributions and Standard Errors

📘 Definition 4.1 — Random sample, statistic, sampling distribution, standard error

A random sample of size $n$ from a distribution $F$ is a collection $X_1,\dots,X_n$ of iid random variables with common cdf $F$. A statistic is a random variable $T=t(X_1,\dots,X_n)$, where $t$ is a measurable function that does not involve unknown parameters. The distribution of $T$ is its sampling distribution, and the standard deviation $\sqrt{\operatorname{Var}(T)}$ is the standard error of $T$, written $\operatorname{SE}(T)$. When the standard error depends on unknown parameters, replacing them by estimates gives the estimated standard error $\widehat{\operatorname{SE}}(T)$.

The basic statistics are the sample mean $\bar X=\frac1n\sum_{i=1}^nX_i$, the sample variance $S^2=\frac{1}{n-1}\sum_{i=1}^n(X_i-\bar X)^2$ (note the divisor $n-1$), the sample standard deviation $S=\sqrt{S^2}$, and the sample moments $m_r'=\frac1n\sum X_i^r$ and $m_r=\frac1n\sum(X_i-\bar X)^r$. We write $\mu=E[X_1]$, $\sigma^2=\operatorname{Var}(X_1)$, and $\mu_r=E[(X_1-\mu)^r]$ for the central moments of the population.

The algebraic identity $$\sum_{i=1}^n(X_i-\bar X)^2=\sum_{i=1}^n(X_i-c)^2-n(\bar X-c)^2\qquad(c\in\mathbb{R})$$ (expand $X_i-c=(X_i-\bar X)+(\bar X-c)$ and use $\sum(X_i-\bar X)=0$) is used constantly below. With $c=\mu$ it explains the divisor $n-1$.

📐 Theorem 4.1 — Moments of $\bar X$ and $S^2$ for any population

Let $X_1,\dots,X_n$ be a random sample with $\sigma^2\lt\infty$ and $n\ge2$. Then

  1. $E[\bar X]=\mu$ and $\operatorname{Var}(\bar X)=\dfrac{\sigma^2}{n}$, so $\operatorname{SE}(\bar X)=\dfrac{\sigma}{\sqrt n}$;
  2. $E[S^2]=\sigma^2$, while $\hat\sigma^2=\frac1n\sum(X_i-\bar X)^2$ has $E[\hat\sigma^2]=\frac{n-1}{n}\sigma^2$;
  3. if $E[X_1^3]$ exists, then $\operatorname{Cov}(\bar X,S^2)=\dfrac{\mu_3}{n}$; in particular $\bar X$ and $S^2$ are uncorrelated for every symmetric population;
  4. if $E[X_1^4]$ exists, then $\operatorname{Var}(S^2)=\dfrac{\mu_4}{n}-\dfrac{(n-3)\,\sigma^4}{n(n-1)}$.

Proof. Since $S^2$ is unchanged when every $X_i$ is replaced by $X_i-\mu$, we may assume $\mu=0$ in (2)–(4). (1) By linearity $E[\bar X]=\mu$, and by independence $\operatorname{Var}(\bar X)=\frac{1}{n^2}\sum\operatorname{Var}(X_i)=\frac{\sigma^2}{n}$.

(2) The identity with $c=0$ gives $(n-1)S^2=\sum X_i^2-n\bar X^2$. Taking expectations, $E[\sum X_i^2]=n\sigma^2$ and $E[n\bar X^2]=n\operatorname{Var}(\bar X)=\sigma^2$, so $(n-1)E[S^2]=(n-1)\sigma^2$. Also $\hat\sigma^2=\frac{n-1}{n}S^2$.

(3) With $\mu=0$, $\operatorname{Cov}(\bar X,S^2)=E[\bar XS^2]$. Now $E[\bar X\sum_jX_j^2]=\frac1n\sum_i\sum_jE[X_iX_j^2]$. By independence and $E[X_i]=0$, only the $n$ terms with $i=j$ survive, so this equals $\mu_3$. Similarly $E[\bar X^3]=\frac{1}{n^3}\sum_{i,j,k}E[X_iX_jX_k]=\frac{n\mu_3}{n^3}$, since only the $n$ terms with $i=j=k$ survive. Hence $(n-1)E[\bar XS^2]=\mu_3-n\cdot\frac{\mu_3}{n^2}=\frac{(n-1)\mu_3}{n}$.

(4) (Sketch.) Write $(n-1)S^2=\frac{n-1}{n}\sum_iX_i^2-\frac1n\sum_{i\ne j}X_iX_j$. Square this and take expectations term by term. A product of four $X$'s has nonzero mean only when the indices pair up, giving $\mu_4$ (all four equal) or $\sigma^4$ (two distinct pairs). Collecting the terms gives $(n-1)^2E[S^4]=\frac{(n-1)^2}{n}\mu_4+\frac{(n-1)(n^2-2n+3)}{n}\sigma^4$. Subtracting $(n-1)^2\sigma^4$ yields (4). (Exercise I.3 checks the formula on a two-point population, and the widget below checks it by simulation.) ∎

For a normal population $\mu_4=3\sigma^4$, and (4) becomes $\operatorname{Var}(S^2)=\frac{2\sigma^4}{n-1}$. Theorem 4.2 below explains why. Writing $\kappa=\mu_4/\sigma^4$ for the kurtosis, (4) shows that $\operatorname{Var}(S^2)\approx(\kappa-1)\sigma^4/n$ for large $n$. Heavy-tailed populations therefore make $S^2$ far less precise than the normal theory suggests: the factor $\kappa-1$ is $2$ for normal, $0.8$ for uniform and $8$ for exponential populations.

✏️ Example 1 — Standard errors at work
A filling machine delivers volumes with standard deviation $\sigma=4$ ml. A sample of $n=64$ bottles is measured. (a) Find $\operatorname{SE}(\bar X)$ and, using the CLT, approximate $P(|\bar X-\mu|\le1)$. (b) How large must $n$ be for $\operatorname{SE}(\bar X)\le0.25$ ml? (c) If the volumes are normal, find $\operatorname{SE}(S^2)$. (d) In a survey of $n=400$ households, $20\%$ own a car. Find the estimated standard error of the sample proportion.
(a) $\operatorname{SE}(\bar X)=\sigma/\sqrt n=4/8=0.5$ ml. By the CLT, $(\bar X-\mu)/0.5$ is approximately $N(0,1)$, so $P(|\bar X-\mu|\le1)\approx P(|Z|\le2)=2\Phi(2)-1=0.9545$.
(b) $4/\sqrt n\le0.25\iff\sqrt n\ge16\iff n\ge256$. Halving the standard error needs four times as many observations: precision grows like $\sqrt n$.
(c) With $\mu_4=3\sigma^4$, Theorem 4.1(4) gives $\operatorname{Var}(S^2)=\frac{2\sigma^4}{n-1}=\frac{2\cdot256}{63}$, so $\operatorname{SE}(S^2)=16\sqrt{2/63}=2.85$ ml$^2$. The variance $\sigma^2=16$ is estimated with a relative error of about $18\%$, much worse than the relative error $0.5/\mu$ of the mean.
(d) The sample proportion is the mean of $n$ Bernoulli variables, so $\operatorname{SE}(\hat p)=\sqrt{p(1-p)/n}$. The estimated value is $\sqrt{0.2\times0.8/400}=0.02$, that is, two percentage points.

Sampling from a Normal Population: $\chi^2$, $t$ and $F$

For normal samples the sampling distributions are known exactly, and three families of laws appear. The chi-square law was introduced in §2.1 as a gamma law. The $t$ and $F$ laws are new.

📘 Definition 4.2 — The chi-square, Student $t$ and Snedecor $F$ laws
  1. For $k\in\mathbb{N}$, the chi-square law with $k$ degrees of freedom is $\chi^2_k=\operatorname{Gamma}(\tfrac k2,\tfrac12)$, with density $\dfrac{x^{k/2-1}e^{-x/2}}{2^{k/2}\Gamma(k/2)}$, $x\gt0$. It is the law of $Z_1^2+\dots+Z_k^2$ for iid $N(0,1)$ variables $Z_j$ (§2.1).
  2. If $Z\sim N(0,1)$ and $V\sim\chi^2_k$ are independent, then $T=\dfrac{Z}{\sqrt{V/k}}$ has Student's $t$ law with $k$ degrees of freedom, $t_k$.
  3. If $U\sim\chi^2_m$ and $V\sim\chi^2_n$ are independent, then $F=\dfrac{U/m}{V/n}$ has Snedecor's $F$ law with $(m,n)$ degrees of freedom, $F_{m,n}$. The order matters: $m$ belongs to the numerator.

Upper quantiles are written $\chi^2_{k,\alpha}$, $t_{k,\alpha}$ and $F_{m,n;\alpha}$, so that, for example, $P(T\gt t_{k,\alpha})=\alpha$.

📐 Theorem 4.2 — The fundamental theorem for normal samples

Let $X_1,\dots,X_n$ be iid $N(\mu,\sigma^2)$ with $n\ge2$. Then

  1. $\bar X\sim N(\mu,\sigma^2/n)$;
  2. $\dfrac{(n-1)S^2}{\sigma^2}=\dfrac{\sum(X_i-\bar X)^2}{\sigma^2}\sim\chi^2_{n-1}$;
  3. $\bar X$ is independent of the vector of deviations $(X_1-\bar X,\dots,X_n-\bar X)$, and hence of $S^2$, of the range $X_{(n)}-X_{(1)}$, of the mean deviation $\frac1n\sum|X_i-\bar X|$, and of every other function of the deviations.

Conversely (Geary's theorem, statement): if $X_1,\dots,X_n$ ($n\ge2$) are iid and $\bar X$ and $S^2$ are independent, then the common law is normal (or degenerate).

Proof. Put $Z_i=(X_i-\mu)/\sigma$, which are iid $N(0,1)$, and $Z=(Z_1,\dots,Z_n)^{\mathsf T}$. The joint density of $Z$ is $(2\pi)^{-n/2}\exp(-\tfrac12\|z\|^2)$, which depends on $z$ only through its length. Let $A$ be an orthogonal $n\times n$ matrix whose first row is $a_1=\frac{1}{\sqrt n}(1,\dots,1)$. One explicit choice is the Helmert matrix, whose $k$-th row for $k=2,\dots,n$ is $$a_k=\tfrac{1}{\sqrt{k(k-1)}}\,(\underbrace{1,\dots,1}_{k-1},\,-(k-1),\,0,\dots,0).$$ Each row has length $1$ because $\frac{(k-1)+(k-1)^2}{k(k-1)}=1$. Each $a_k$ with $k\ge2$ has entries summing to $0$, so it is orthogonal to $a_1$. For $2\le j\lt k$, $a_k$ is constant on the first $j$ coordinates, which are the only nonzero ones of $a_j$, so $a_j\cdot a_k$ is a multiple of the sum of the entries of $a_j$, which is $0$.

Let $Y=AZ$. The map $z\mapsto Az$ is a bijection with $|\det A|=1$ and $\|Az\|=\|z\|$. By the Jacobian formula (§1.4) $Y$ has density $(2\pi)^{-n/2}\exp(-\tfrac12\|y\|^2)$, so $Y_1,\dots,Y_n$ are iid $N(0,1)$. Now $Y_1=a_1\cdot Z=\sqrt n\,\bar Z$. Since $A^{\mathsf T}A=I$, we have $Z=\sum_kY_ka_k^{\mathsf T}$, and the $k=1$ term is $\bar Z\,(1,\dots,1)^{\mathsf T}$. Hence $$(Z_1-\bar Z,\dots,Z_n-\bar Z)^{\mathsf T}=\sum_{k=2}^nY_ka_k^{\mathsf T},$$ which is a function of $(Y_2,\dots,Y_n)$ alone. Since $Y_1$ is independent of $(Y_2,\dots,Y_n)$, $\bar Z$ is independent of the deviations, which proves (3) because $X_i-\bar X=\sigma(Z_i-\bar Z)$ and $\bar X=\mu+\sigma\bar Z$. By orthonormality of the $a_k$, $$\sum_i(Z_i-\bar Z)^2=\Big\|\sum_{k\ge2}Y_ka_k\Big\|^2=\sum_{k=2}^nY_k^2\sim\chi^2_{n-1},$$ which is (2). Finally $\bar X=\mu+\sigma Y_1/\sqrt n\sim N(\mu,\sigma^2/n)$, which is (1). ∎

The proof shows where the "lost" degree of freedom goes. The $n$ deviations satisfy one linear constraint, $\sum(X_i-\bar X)=0$, so they live in an $(n-1)$-dimensional subspace, and their squared length is a sum of $n-1$ independent squared standard normals.

📐 Theorem 4.3 — Densities and moments of the $t$ and $F$ laws
  1. $t_k$ has density $$f(t)=\frac{\Gamma\left(\frac{k+1}{2}\right)}{\sqrt{k\pi}\,\Gamma\left(\frac k2\right)}\left(1+\frac{t^2}{k}\right)^{-(k+1)/2},\qquad t\in\mathbb{R}.$$ It is symmetric about $0$, and $E|T|^r\lt\infty$ if and only if $r\lt k$. Hence $E[T]=0$ for $k\ge2$, and $\operatorname{Var}(T)=\dfrac{k}{k-2}$ for $k\ge3$; for $k=2$ the variance is infinite.
  2. $F_{m,n}$ has density $$f(x)=\frac{(m/n)^{m/2}\,x^{m/2-1}}{B\left(\frac m2,\frac n2\right)\left(1+\frac{m}{n}x\right)^{(m+n)/2}},\qquad x\gt0.$$ Its mean is $E[F]=\dfrac{n}{n-2}$ for $n\gt2$ (it does not depend on $m$), and $\operatorname{Var}(F)=\dfrac{2n^2(m+n-2)}{m(n-2)^2(n-4)}$ for $n\gt4$.

Proof. (1) Conditionally on $V=v$, $T=Z\sqrt{k/v}$ is $N(0,k/v)$, with density $\sqrt{v/k}\,\varphi(t\sqrt{v/k})$. Averaging over $V$, $$f(t)=\int_0^\infty\sqrt{\tfrac{v}{2\pi k}}\,e^{-t^2v/(2k)}\,\frac{v^{k/2-1}e^{-v/2}}{2^{k/2}\Gamma(k/2)}\,dv=\frac{1}{\sqrt{2\pi k}\,2^{k/2}\Gamma(k/2)}\int_0^\infty v^{\frac{k+1}{2}-1}e^{-\frac v2\left(1+\frac{t^2}{k}\right)}dv.$$ The last integral is a gamma integral, equal to $\Gamma(\frac{k+1}2)\left(\frac{2}{1+t^2/k}\right)^{(k+1)/2}$, and $2^{(k+1)/2}/(\sqrt{2\pi k}\,2^{k/2})=1/\sqrt{k\pi}$. Moments: by independence, $E|T|^r=k^{r/2}E|Z|^r\,E[V^{-r/2}]$. By the gamma moment formula of §2.1, $E[V^{-r/2}]\lt\infty$ iff $-r/2\gt-k/2$, i.e. $r\lt k$. Finally $E[T^2]=k\,E[Z^2]\,E[V^{-1}]=k\cdot\frac{1}{k-2}$.

(2) Since $\chi^2_j=\operatorname{Gamma}(\frac j2,\frac12)$, the gamma-ratio theorem of §2.1 shows that $B=U/(U+V)\sim\operatorname{Beta}(\frac m2,\frac n2)$. Now $F=\frac nm\cdot\frac{B}{1-B}$, an increasing bijection of $(0,1)$ onto $(0,\infty)$ with inverse $b=\frac{mx}{n+mx}$ and $\frac{db}{dx}=\frac{mn}{(n+mx)^2}$. Substituting $b$ and $1-b=\frac{n}{n+mx}$ into the beta density and multiplying by $\frac{db}{dx}$ gives $$\frac{(mx)^{m/2-1}n^{n/2-1}}{B(\frac m2,\frac n2)(n+mx)^{(m+n)/2-2}}\cdot\frac{mn}{(n+mx)^2},$$ which simplifies to the stated density. For the moments, $E[F]=\frac nm E[U]E[V^{-1}]=\frac nm\cdot m\cdot\frac{1}{n-2}$ and $E[F^2]=\frac{n^2}{m^2}E[U^2]E[V^{-2}]=\frac{n^2}{m^2}\cdot\frac{m(m+2)}{(n-2)(n-4)}$. Then $$\operatorname{Var}(F)=\frac{n^2\left[(m+2)(n-2)-m(n-4)\right]}{m(n-2)^2(n-4)}=\frac{2n^2(m+n-2)}{m(n-2)^2(n-4)}.\ ∎$$

📐 Theorem 4.4 — Relations among the $\chi^2$, $t$, $F$, beta, Cauchy and normal laws
  1. $\chi^2_k$ has mean $k$, variance $2k$ and mgf $(1-2t)^{-k/2}$ for $t\lt\frac12$. Independent chi-squares add: $\chi^2_a+\chi^2_b=\chi^2_{a+b}$. Also $\chi^2_2=\operatorname{Exp}(\frac12)$, and $(\chi^2_k-k)/\sqrt{2k}\xrightarrow{d}N(0,1)$ as $k\to\infty$.
  2. If $T\sim t_k$, then $T^2\sim F_{1,k}$. Also $t_1$ is the standard Cauchy law, and $t_k\xrightarrow{d}N(0,1)$ as $k\to\infty$.
  3. If $F\sim F_{m,n}$, then $1/F\sim F_{n,m}$. Consequently the lower quantile is $F_{m,n;1-\alpha}=1/F_{n,m;\alpha}$.
  4. If $F\sim F_{m,n}$, then $\dfrac{mF}{n+mF}\sim\operatorname{Beta}\left(\frac m2,\frac n2\right)$; in particular $F_{2,2}$ has cdf $\frac{x}{1+x}$.
  5. If $F_n\sim F_{m,n}$ with $m$ fixed, then $mF_n\xrightarrow{d}\chi^2_m$ as $n\to\infty$.

Proof. (1) is §2.1 (gamma moments, the gamma mgf and the reproductive property) together with the CLT applied to a sum of $k$ iid $\chi^2_1$ variables. (2) $T^2=\frac{Z^2/1}{V/k}$ and $Z^2\sim\chi^2_1$ is independent of $V$. For $k=1$ the density in Theorem 4.3 is $\frac{\Gamma(1)}{\sqrt\pi\,\Gamma(1/2)}(1+t^2)^{-1}=\frac{1}{\pi(1+t^2)}$. For the limit, write $T_k=Z/\sqrt{V_k/k}$ where $V_k$ is a sum of $k$ iid $\chi^2_1$ variables. By the weak law, $V_k/k\xrightarrow{P}1$, and Slutsky's theorem (§2.3) gives $T_k\xrightarrow{d}Z$. (3) $1/F=\frac{V/n}{U/m}$ is of the defining form with the roles exchanged. Hence $P(F\le x)=P(1/F\ge1/x)$; putting $x=1/F_{n,m;\alpha}$ makes the right side $\alpha$. (4) This is the variable $B$ of the proof of Theorem 4.3. For $m=n=2$, $B$ is uniform, so $P(F\le x)=P\left(B\le\frac{x}{1+x}\right)=\frac{x}{1+x}$. (5) $mF_n=U/(V_n/n)$ and $V_n/n\xrightarrow{P}1$; apply Slutsky. ∎

📐 Theorem 4.5 — Pivotal statistics from normal samples
  1. (One sample) If $X_1,\dots,X_n$ are iid $N(\mu,\sigma^2)$, then $\dfrac{\sqrt n(\bar X-\mu)}{S}\sim t_{n-1}$. If $\mu$ is known, $\dfrac{1}{\sigma^2}\sum(X_i-\mu)^2\sim\chi^2_n$.
  2. (Two samples, common variance) Let $X_1,\dots,X_{n_1}$ be iid $N(\mu_1,\sigma^2)$ and $Y_1,\dots,Y_{n_2}$ iid $N(\mu_2,\sigma^2)$, independent of each other. Define the pooled variance $S_p^2=\dfrac{(n_1-1)S_1^2+(n_2-1)S_2^2}{n_1+n_2-2}$. Then $$\frac{(\bar X-\bar Y)-(\mu_1-\mu_2)}{S_p\sqrt{\frac{1}{n_1}+\frac{1}{n_2}}}\sim t_{n_1+n_2-2}.$$
  3. (Variance ratio) With the same two samples but variances $\sigma_1^2,\sigma_2^2$: $\dfrac{S_1^2/\sigma_1^2}{S_2^2/\sigma_2^2}\sim F_{n_1-1,\,n_2-1}$.

Proof. (1) $Z=\sqrt n(\bar X-\mu)/\sigma\sim N(0,1)$ and $V=(n-1)S^2/\sigma^2\sim\chi^2_{n-1}$ are independent by Theorem 4.2, and $Z/\sqrt{V/(n-1)}=\sqrt n(\bar X-\mu)/S$. The second statement is Definition 4.2(1) applied to $(X_i-\mu)/\sigma$. (2) $\bar X-\bar Y\sim N\left(\mu_1-\mu_2,\sigma^2(\frac1{n_1}+\frac1{n_2})\right)$. By Theorem 4.2 applied to each sample and by the independence of the samples, the four statistics $\bar X,S_1^2,\bar Y,S_2^2$ are mutually independent, and $(n_1+n_2-2)S_p^2/\sigma^2$ is a sum of independent $\chi^2_{n_1-1}$ and $\chi^2_{n_2-1}$ variables, hence $\chi^2_{n_1+n_2-2}$. Divide the standardised difference by $\sqrt{S_p^2/\sigma^2}$. (3) The ratio is $\frac{\chi^2_{n_1-1}/(n_1-1)}{\chi^2_{n_2-1}/(n_2-1)}$ with independent numerator and denominator. ∎

These three statistics are pivots: they depend on the unknown parameters, but their laws do not. That is exactly what is needed to build the confidence intervals of §4.3 and the $t$- and $F$-tests of Chapter 5.

✏️ Example 2 — Computing with $\chi^2$, $t$ and $F$
A random sample of $n=10$ is drawn from $N(\mu,\sigma^2)$. (a) Find $P(S^2\gt2\sigma^2)$. (b) Find $c$ such that $P(|\bar X-\mu|\le cS)=0.95$. (c) A second independent sample of size $16$ is drawn from a normal law with the same variance. Find $k_1,k_2$ with $P(S_1^2/S_2^2\gt k_1)=0.05$ and $P(S_1^2/S_2^2\lt k_2)=0.05$.
(a) $P(S^2\gt2\sigma^2)=P\left(\frac{9S^2}{\sigma^2}\gt18\right)=P(\chi^2_9\gt18)=0.0352$ (scipy: chi2.sf(18, 9)). For comparison, $\chi^2_{9,0.05}=16.92$.
(b) $P(|\bar X-\mu|\le cS)=P(|T|\le c\sqrt{10})$ with $T\sim t_9$. We need $c\sqrt{10}=t_{9,0.025}=2.262$, so $c=2.262/3.162=0.7154$.
(c) With equal variances, $S_1^2/S_2^2\sim F_{9,15}$. The upper point is $k_1=F_{9,15;0.05}=2.588$. The lower point is $k_2=F_{9,15;0.95}=1/F_{15,9;0.05}=1/3.006=0.3327$ by Theorem 4.4(3).
Check The $F_{9,15}$ law has mean $\frac{15}{13}=1.15$. The interval $(0.333,\,2.588)$ is far from symmetric about the mean, as expected for a right-skewed law.
Interactive 1 — Is the Normal Theory Exact? Simulating $S^2$ and the $t$ Statistic

Samples of size $n$ are drawn repeatedly from the chosen population (mean $0$, variance $1$, seeded generator), and the blue histogram collects the values of the chosen statistic. The purple curve is the normal-theory law: $\chi^2_{n-1}$ for $(n-1)S^2/\sigma^2$ and $t_{n-1}$ for $T=\sqrt n\,\bar X/S$. The dashed curve for $T$ is $N(0,1)$. For a normal population the histogram matches the purple curve for every $n$ (Theorems 4.2 and 4.5). For other populations the variance statistic stays wrong however large $n$ is, because its variance is governed by the kurtosis (Theorem 4.1). The $t$ statistic, by contrast, becomes approximately normal (Theorem 4.6). Watch the tail rate in the readouts.

8
samples drawn—
simulated mean, variance—
normal theory mean, variance—
beyond the 5% point (nominal 0.05)—
✏️ Example 3 — Prove or disprove: $\bar X$ and $S^2$ are independent
Let $X_1,X_2$ be iid, so that $\bar X=\frac{X_1+X_2}{2}$ and $S^2=\frac{(X_1-X_2)^2}{2}$. Decide whether $\bar X$ and $S^2$ are independent when the common law is (a) $N(\mu,\sigma^2)$; (b) $\operatorname{Exp}(1)$; (c) $U(-1,1)$; (d) $\operatorname{Ber}(\frac12)$. In which cases are they uncorrelated?
(a) Independent $(X_1+X_2,X_1-X_2)$ is a linear image of a normal vector, so it is bivariate normal. Its covariance is $\operatorname{Var}X_1-\operatorname{Var}X_2=0$, so the two components are independent (§1.4), and hence so are $\bar X$ and $S^2$. This is Theorem 4.2 with $n=2$.
(b) Dependent Both variables are nonnegative, so $|X_1-X_2|\le X_1+X_2=2\bar X$, i.e. $S^2\le2\bar X^2$. Hence $P(\bar X\lt1,\ S^2\gt2)=0$, while $P(\bar X\lt1)\gt0$ and $P(S^2\gt2)=P(|X_1-X_2|\gt2)\gt0$. They are not even uncorrelated: Theorem 4.1(3) gives $\operatorname{Cov}=\mu_3/2=1$, the third central moment of $\operatorname{Exp}(1)$ being $2$.
(c) Uncorrelated but dependent The law is symmetric, so $\mu_3=0$ and $\operatorname{Cov}(\bar X,S^2)=0$. But both $X_i\in[-1,1]$, so if $|\bar X|\gt\frac12$ the two values lie on the same side within an interval of length less than $1$: if $\bar X\gt\frac12$, then $X_1,X_2\gt0$ and $|X_1-X_2|\lt1$. Thus $P(|\bar X|\gt\tfrac12,\ S^2\gt\tfrac12)=0$, although both events have positive probability.
(d) Dependent $S^2=\frac12$ exactly when $X_1\ne X_2$, which happens exactly when $\bar X=\frac12$. So $S^2$ is a function of $\bar X$, namely $S^2=\frac12\mathbf 1\{\bar X=\frac12\}$, and $P(S^2=\tfrac12\mid\bar X=\tfrac12)=1\ne P(S^2=\tfrac12)=\tfrac12$. The law is symmetric about $\frac12$, so again $\operatorname{Cov}=0$.
Moral Independence of $\bar X$ and $S^2$ is a property of the normal law alone (Geary's theorem), and zero correlation ($\mu_3=0$) is much weaker. A Part C option such as "for any population, $\bar X$ and $S^2$ are independent" is false.

Asymptotic Distributions and Standard Errors

For most statistics from non-normal populations, exact sampling distributions are out of reach. For large $n$, however, the CLT, Slutsky's theorem and the delta method of §2.3 give normal approximations whose variances provide the standard errors.

📐 Theorem 4.6 — Asymptotic laws of sample moments

Let $X_1,X_2,\dots$ be iid.

  1. If $E[X_1^{2r}]\lt\infty$, then $\sqrt n\,(m_r'-\mu_r')\xrightarrow{d}N\left(0,\ \mu_{2r}'-\mu_r'^2\right)$, where $\mu_r'=E[X_1^r]$.
  2. If $\mu_4\lt\infty$, then $\sqrt n\,(S^2-\sigma^2)\xrightarrow{d}N(0,\ \mu_4-\sigma^4)$, and the same holds for $\hat\sigma^2$.
  3. If also $\sigma\gt0$, then $\sqrt n\,(S-\sigma)\xrightarrow{d}N\left(0,\ \dfrac{\mu_4-\sigma^4}{4\sigma^2}\right)$. For a normal population the limits in (2) and (3) are $N(0,2\sigma^4)$ and $N(0,\sigma^2/2)$.
  4. If $0\lt\sigma^2\lt\infty$, the studentised mean satisfies $\sqrt n(\bar X-\mu)/S\xrightarrow{d}N(0,1)$ for every population.

Proof. (1) The variables $X_i^r$ are iid with mean $\mu_r'$ and variance $\mu_{2r}'-\mu_r'^2$; apply the CLT. (2) By the identity with $c=\mu$, $$\hat\sigma^2=\frac1n\sum(X_i-\mu)^2-(\bar X-\mu)^2.$$ The $W_i=(X_i-\mu)^2$ are iid with mean $\sigma^2$ and variance $\mu_4-\sigma^4$, so $\sqrt n(\bar W-\sigma^2)\xrightarrow{d}N(0,\mu_4-\sigma^4)$. Next, $\sqrt n(\bar X-\mu)^2=\left[\sqrt n(\bar X-\mu)\right](\bar X-\mu)$ is a product of a term bounded in probability and a term tending to $0$ in probability, so it tends to $0$ in probability. By Slutsky, $\sqrt n(\hat\sigma^2-\sigma^2)$ has the same limit. Finally $\sqrt n(S^2-\hat\sigma^2)=\frac{\sqrt n}{n-1}\hat\sigma^2\xrightarrow{P}0$. (3) Apply the delta method with $g(x)=\sqrt x$ and $g'(\sigma^2)=\frac1{2\sigma}$; for the normal case use $\mu_4=3\sigma^4$. (4) $S\xrightarrow{P}\sigma$ by (2) and the continuous mapping theorem; then apply the CLT and Slutsky. ∎

🧭 Problem-solving strategy — standard errors you should know
statisticestimatesstandard error (exact or large-sample)
$\bar X$$\mu$$\sigma/\sqrt n$ (exact)
$\hat p$ (sample proportion)$p$$\sqrt{p(1-p)/n}$ (exact)
$\bar X_1-\bar X_2$ (independent samples)$\mu_1-\mu_2$$\sqrt{\sigma_1^2/n_1+\sigma_2^2/n_2}$ (exact)
$S^2$$\sigma^2$$\sqrt{\operatorname{Var}(S^2)}$ from Theorem 4.1(4); $\sigma^2\sqrt{2/(n-1)}$ if normal
$S$$\sigma$$\approx\sqrt{(\mu_4-\sigma^4)/(4\sigma^2n)}$; $\approx\sigma/\sqrt{2n}$ if normal
sample $p$-quantile$\xi_p$$\approx\sqrt{p(1-p)}\,/\,(f(\xi_p)\sqrt n)$ (Theorem 4.12)
sample medianmedian $m$$\approx1/(2f(m)\sqrt n)$; $\approx\sigma\sqrt{\pi/(2n)}$ if normal
$g(\bar X)$$g(\mu)$$\approx|g'(\mu)|\,\sigma/\sqrt n$ (delta method)
sample correlation $r$$\rho$$\approx(1-\rho^2)/\sqrt n$ (bivariate normal; §7.2)

Replace unknown quantities by consistent estimates to get estimated standard errors. Slutsky's theorem guarantees that the studentised statistic is still asymptotically $N(0,1)$.

✏️ Example 4 — Delta-method standard errors
(a) A normal sample of size $n=50$ has true $\sigma=10$. Find the approximate standard error of $S$. (b) For a sample from an exponential law with mean $\theta$, find the asymptotic variance of $S^2$ and compare it with the normal-theory value. (c) Daily counts are $\operatorname{Poi}(\lambda)$ and $P(X=0)=e^{-\lambda}$ is estimated by $e^{-\bar X}$. Find its asymptotic standard error when $\lambda=2$ and $n=100$.
(a) By Theorem 4.6(3), $\operatorname{SE}(S)\approx\sigma/\sqrt{2n}=10/10=1$.
(b) For $\operatorname{Exp}$ with mean $\theta$, $\sigma^2=\theta^2$ and $\mu_4=9\theta^4$. Theorem 4.6(2) gives the asymptotic variance $(\mu_4-\sigma^4)/n=8\theta^4/n$, four times the normal-theory value $2\theta^4/n$. A normal-theory interval for $\sigma^2$ would be half as wide as it should be.
(c) $\sqrt n(\bar X-\lambda)\xrightarrow{d}N(0,\lambda)$, and $g(x)=e^{-x}$ has $g'(\lambda)=-e^{-\lambda}$. So $\operatorname{SE}(e^{-\bar X})\approx e^{-\lambda}\sqrt{\lambda/n}=e^{-2}\sqrt{0.02}=0.0191$.

Order Statistics

📘 Definition 4.3 — Order statistics, range, median, midrange and spacings

Sorting a sample $X_1,\dots,X_n$ into increasing order gives the order statistics $X_{(1)}\le X_{(2)}\le\dots\le X_{(n)}$. Thus $X_{(1)}=\min X_i$ and $X_{(n)}=\max X_i$. Derived statistics:

  • the sample range $R=X_{(n)}-X_{(1)}$ and the midrange $\frac12(X_{(1)}+X_{(n)})$;
  • the sample median: $X_{(k+1)}$ if $n=2k+1$, and $\frac12(X_{(k)}+X_{(k+1)})$ if $n=2k$; more generally the sample $p$-quantile $X_{(\lceil np\rceil)}$;
  • the spacings $X_{(i)}-X_{(i-1)}$, $i=2,\dots,n$.

If $F$ is continuous, ties occur with probability $0$, so $X_{(1)}\lt\dots\lt X_{(n)}$ almost surely.

Order statistics are not independent: they are tied together by the ordering, and they are not identically distributed either. Yet one counting idea gives all their laws. The number of observations at or below $x$, $$N(x)=\#\{i:X_i\le x\},$$ is $\operatorname{Bin}(n,F(x))$, and $X_{(r)}\le x$ exactly when $N(x)\ge r$.

📐 Theorem 4.7 — The distribution of the $r$-th order statistic

Let $X_1,\dots,X_n$ be iid with cdf $F$ and let $1\le r\le n$.

  1. For every $F$ (discrete, continuous or mixed), $$P(X_{(r)}\le x)=\sum_{j=r}^n\binom nj F(x)^j\left(1-F(x)\right)^{n-j}=\frac{n!}{(r-1)!\,(n-r)!}\int_0^{F(x)}t^{r-1}(1-t)^{n-r}\,dt.$$
  2. If $F$ has a density $f$, then $X_{(r)}$ has density $$f_{(r)}(x)=\frac{n!}{(r-1)!\,(n-r)!}\,F(x)^{r-1}\left(1-F(x)\right)^{n-r}f(x).$$

For $r=n$ and $r=1$ this gives $F_{(n)}=F^n$ and $1-F_{(1)}=(1-F)^n$, the maximum and minimum laws of §1.4.

Proof. (1) The events $\{X_i\le x\}$ are independent with probability $F(x)$, so $N(x)\sim\operatorname{Bin}(n,F(x))$. Since $\{X_{(r)}\le x\}=\{N(x)\ge r\}$, the first formula follows. The second is the beta–binomial tail identity of §2.1: $$\sum_{j\ge r}\binom nj p^j(1-p)^{n-j}=\frac{n!}{(r-1)!(n-r)!}\int_0^pt^{r-1}(1-t)^{n-r}dt.$$ (2) If $F$ has density $f$, then $F$ is absolutely continuous with $F'=f$ almost everywhere. Differentiating the integral form by the chain rule gives the density. ∎

A heuristic worth remembering. For $X_{(r)}$ to lie in $[x,x+dx]$, the sample must split into $r-1$ values below $x$, one value in $[x,x+dx]$ and $n-r$ values above. The multinomial probability of this is $$\frac{n!}{(r-1)!\,1!\,(n-r)!}\,F(x)^{r-1}\,f(x)\,dx\,\left(1-F(x)\right)^{n-r}.$$ The same "boxes" argument gives every joint density below.

📐 Theorem 4.8 — Joint densities of order statistics

Let $X_1,\dots,X_n$ be iid with density $f$ and cdf $F$.

  1. The vector $(X_{(1)},\dots,X_{(n)})$ has density $n!\,f(y_1)f(y_2)\cdots f(y_n)$ on $\{y_1\lt y_2\lt\dots\lt y_n\}$, and $0$ elsewhere.
  2. For $1\le r\lt s\le n$, the pair $(X_{(r)},X_{(s)})$ has density, for $x\lt y$, $$\frac{n!}{(r-1)!\,(s-r-1)!\,(n-s)!}\,F(x)^{r-1}\left[F(y)-F(x)\right]^{s-r-1}\left[1-F(y)\right]^{n-s}f(x)f(y).$$ In particular $(X_{(1)},X_{(n)})$ has density $n(n-1)\left[F(y)-F(x)\right]^{n-2}f(x)f(y)$ for $x\lt y$.

Proof. (1) Up to a null set, $\mathbb{R}^n$ splits into the $n!$ regions $D_\pi=\{x_{\pi(1)}\lt\dots\lt x_{\pi(n)}\}$, one for each permutation $\pi$. On $D_\pi$ the sorting map is the linear map $x\mapsto(x_{\pi(1)},\dots,x_{\pi(n)})$, which has $|\det|=1$ and carries $D_\pi$ onto the ordered region $D=\{y_1\lt\dots\lt y_n\}$. For a Borel set $B\subseteq D$, $$P\left((X_{(1)},\dots,X_{(n)})\in B\right)=\sum_\pi P\left((X_{\pi(1)},\dots,X_{\pi(n)})\in B\right)=n!\int_B\prod_if(y_i)\,dy,$$ because each $(X_{\pi(1)},\dots,X_{\pi(n)})$ has the same density $\prod f(y_i)$ as $(X_1,\dots,X_n)$. (2) Integrate the density in (1) over all coordinates except $y_r=x$ and $y_s=y$. The coordinates split into three blocks. For the $r-1$ ordered coordinates below $x$, $$\int_{y_1\lt\dots\lt y_{r-1}\lt x}\prod_{i\lt r}f(y_i)\,dy=\frac{F(x)^{r-1}}{(r-1)!}.$$ This holds because the unrestricted integral over $(-\infty,x)^{r-1}$ equals $F(x)^{r-1}$, and by symmetry it is shared equally among the $(r-1)!$ orderings. Likewise the $s-r-1$ coordinates in $(x,y)$ contribute $\frac{[F(y)-F(x)]^{s-r-1}}{(s-r-1)!}$, and the $n-s$ coordinates above $y$ contribute $\frac{[1-F(y)]^{n-s}}{(n-s)!}$. Multiply by $n!f(x)f(y)$. ∎

📐 Theorem 4.9 — Uniform order statistics are beta

Let $U_1,\dots,U_n$ be iid $U(0,1)$.

  1. $U_{(r)}\sim\operatorname{Beta}(r,\,n-r+1)$, so $E[U_{(r)}]=\dfrac{r}{n+1}$ and $\operatorname{Var}(U_{(r)})=\dfrac{r(n-r+1)}{(n+1)^2(n+2)}$.
  2. For $r\lt s$, $U_{(s)}-U_{(r)}\sim\operatorname{Beta}(s-r,\,n-s+r+1)$ and $\operatorname{Cov}(U_{(r)},U_{(s)})=\dfrac{r(n-s+1)}{(n+1)^2(n+2)}$.
  3. The range $R=U_{(n)}-U_{(1)}\sim\operatorname{Beta}(n-1,2)$, with density $n(n-1)w^{n-2}(1-w)$ on $(0,1)$, cdf $nw^{n-1}-(n-1)w^n$ and mean $\frac{n-1}{n+1}$.
  4. (Distribution-free) If $X_1,\dots,X_n$ are iid with a continuous cdf $F$, then $F(X_{(r)})\sim\operatorname{Beta}(r,n-r+1)$. For any $F$, $(X_{(1)},\dots,X_{(n)})$ has the same law as $(F^{-1}(U_{(1)}),\dots,F^{-1}(U_{(n)}))$, where $F^{-1}$ is the quantile function.

Proof. (1) With $F(x)=x$ on $(0,1)$, Theorem 4.7(2) gives the density $\frac{n!}{(r-1)!(n-r)!}x^{r-1}(1-x)^{n-r}$, which is the $\operatorname{Beta}(r,n-r+1)$ density since $B(r,n-r+1)=\frac{(r-1)!(n-r)!}{n!}$. The moments follow from the beta moments of §2.1 with $a=r$, $b=n-r+1$, $a+b=n+1$.

(2) By Theorem 4.8(2), $(U_{(r)},U_{(s)})$ has density $c\,x^{r-1}(y-x)^{s-r-1}(1-y)^{n-s}$ on $0\lt x\lt y\lt1$. Change variables to $(x,w)$ with $w=y-x$; the Jacobian is $1$. Integrating out $x\in(0,1-w)$ and substituting $x=(1-w)t$, $$\int_0^{1-w}x^{r-1}(1-w-x)^{n-s}dx=(1-w)^{n-s+r}B(r,n-s+1),$$ so $W$ has a density proportional to $w^{s-r-1}(1-w)^{n-s+r}$, which is $\operatorname{Beta}(s-r,n-s+r+1)$. For the covariance, let $D=(n+1)^2(n+2)$. Using $2\operatorname{Cov}=\operatorname{Var}U_{(r)}+\operatorname{Var}U_{(s)}-\operatorname{Var}W$ and the beta variances, $$2D\operatorname{Cov}=r(n-r+1)+s(n-s+1)-(s-r)(n-s+r+1)=2r(n+1-s),$$ after expanding $(s-r)(n-s+r+1)=(s-r)(n+1)-(s-r)^2$.

(3) is (2) with $r=1$, $s=n$; integrate the density for the cdf. (4) For continuous $F$, the $F(X_i)$ are iid $U(0,1)$ (the probability integral transform, §1.3). Since $F$ is nondecreasing, sorting commutes with applying $F$, so $F(X_{(r)})$ is the $r$-th order statistic of a uniform sample. Similarly the $F^{-1}(U_i)$ are iid with cdf $F$ (the quantile transform, §1.3), and $F^{-1}$ is nondecreasing. ∎

Part (4) is the source of distribution-free results. For instance, $F(X_{(n)})-F(X_{(1)})$, the population proportion covered by the sample range, is $\operatorname{Beta}(n-1,2)$ for every continuous $F$. The nonparametric tests of §5.3 rely on this kind of fact.

📐 Theorem 4.10 — The law of the sample range

If $X_1,\dots,X_n$ ($n\ge2$) are iid with density $f$ and cdf $F$, then the range $R=X_{(n)}-X_{(1)}$ has, for $w\gt0$, $$f_R(w)=n(n-1)\int_{-\infty}^{\infty}f(x)f(x+w)\left[F(x+w)-F(x)\right]^{n-2}dx,\qquad P(R\le w)=n\int_{-\infty}^\infty f(x)\left[F(x+w)-F(x)\right]^{n-1}dx.$$

Proof. Apply the map $(x,y)\mapsto(x,w)=(x,y-x)$, which has Jacobian $1$, to the joint density of $(X_{(1)},X_{(n)})$ in Theorem 4.8(2), and integrate out $x$. For the cdf, argue directly. Ties have probability $0$, so $\{R\le w\}$ is the disjoint union over $i$ of the events "$X_i$ is the minimum and every other $X_j$ lies in $[X_i,X_i+w]$". By symmetry each has the same probability, and conditioning on $X_i=x$ gives $[F(x+w)-F(x)]^{n-1}$. ∎

📐 Theorem 4.11 — Exponential order statistics: Rényi's representation

Let $X_1,\dots,X_n$ be iid $\operatorname{Exp}(\lambda)$, and put $X_{(0)}=0$. The normalised spacings $$D_i=(n-i+1)\left(X_{(i)}-X_{(i-1)}\right),\qquad i=1,\dots,n,$$ are iid $\operatorname{Exp}(\lambda)$. Consequently:

  1. $X_{(r)}=\sum_{i=1}^r\dfrac{D_i}{n-i+1}$, so $E[X_{(r)}]=\dfrac1\lambda\sum_{i=1}^r\dfrac{1}{n-i+1}$ and $\operatorname{Var}(X_{(r)})=\dfrac{1}{\lambda^2}\sum_{i=1}^r\dfrac{1}{(n-i+1)^2}$;
  2. for $r\lt s$, $X_{(s)}-X_{(r)}$ is independent of $(X_{(1)},\dots,X_{(r)})$ and has the law of the $(s-r)$-th order statistic of $n-r$ iid $\operatorname{Exp}(\lambda)$ variables; in particular the range $X_{(n)}-X_{(1)}$ has the law of the maximum of $n-1$ iid $\operatorname{Exp}(\lambda)$ variables;
  3. the total time on test up to the $r$-th failure, $\sum_{i=1}^rX_{(i)}+(n-r)X_{(r)}=\sum_{i=1}^rD_i$, has the $\operatorname{Gamma}(r,\lambda)$ law, so $2\lambda\left[\sum_{i\le r}X_{(i)}+(n-r)X_{(r)}\right]\sim\chi^2_{2r}$.

Proof. By Theorem 4.8(1) the order statistics have density $n!\,\lambda^ne^{-\lambda\sum y_i}$ on $0\lt y_1\lt\dots\lt y_n$. The map $y\mapsto d$, $d_i=(n-i+1)(y_i-y_{i-1})$, is linear and lower triangular with diagonal $n,n-1,\dots,1$, so its determinant is $n!$. It maps the ordered region bijectively onto $(0,\infty)^n$, with inverse $y_r=\sum_{i\le r}d_i/(n-i+1)$. Summation by parts gives $$\sum_{i=1}^r(n-i+1)(y_i-y_{i-1})=\sum_{i=1}^ry_i+(n-r)y_r,$$ and for $r=n$ this says $\sum_id_i=\sum_iy_i$. Hence $D$ has density $\frac{1}{n!}\cdot n!\,\lambda^ne^{-\lambda\sum d_i}=\prod_i\lambda e^{-\lambda d_i}$, so the $D_i$ are iid $\operatorname{Exp}(\lambda)$. (1) follows from the inverse map. (2) $X_{(s)}-X_{(r)}=\sum_{i=r+1}^sD_i/(n-i+1)$ involves only $D_{r+1},\dots,D_s$, which are independent of $D_1,\dots,D_r$. Re-indexing with $j=i-r$ gives the representation for a sample of size $n-r$. (3) is the summation-by-parts identity together with the reproductive property of the gamma law. ∎

The representation has a memorable interpretation. Until the first failure, $n$ components are running, so the first failure time is the minimum of $n$ exponentials, which is $\operatorname{Exp}(n\lambda)$. By the memoryless property (§2.1), the $n-1$ survivors are then "as good as new", and the wait for the next failure is $\operatorname{Exp}((n-1)\lambda)$, independently of the past, and so on.

✏️ Example 5 — Uniform order statistics
Let $U_1,\dots,U_5$ be iid $U(0,1)$. (a) Find the density of the sample median and $P(\text{median}\le\frac14)$. (b) Find $E[R]$ and $P(R\gt\frac12)$ for the range $R$. (c) For a sample of size $n$ from $U(0,\theta)$, find $E[X_{(n)}]$, $E[X_{(1)}]$ and an unbiased estimator of $\theta$ based on $X_{(n)}$.
(a) The median is $U_{(3)}\sim\operatorname{Beta}(3,3)$, with density $30u^2(1-u)^2$ on $(0,1)$. By Theorem 4.7(1), $P(U_{(3)}\le\frac14)=P(\operatorname{Bin}(5,\tfrac14)\ge3)=\binom53\frac{9}{4^5}+\binom54\frac{3}{4^5}+\frac{1}{4^5}=\frac{90+15+1}{1024}=\frac{106}{1024}=0.1035$.
(b) $R\sim\operatorname{Beta}(4,2)$: $E[R]=\frac46=\frac23$, and $P(R\le\frac12)=5\left(\frac12\right)^4-4\left(\frac12\right)^5=\frac{5}{16}-\frac{2}{16}=\frac3{16}$, so $P(R\gt\frac12)=\frac{13}{16}$.
(c) $X_i=\theta U_i$, so $X_{(r)}=\theta U_{(r)}$ and $E[X_{(n)}]=\frac{n\theta}{n+1}$, $E[X_{(1)}]=\frac{\theta}{n+1}$. Hence $\frac{n+1}{n}X_{(n)}$ is unbiased for $\theta$ (so is $(n+1)X_{(1)}$, but its variance is much larger). In §4.2 we prove that $\frac{n+1}{n}X_{(n)}$ is the best unbiased estimator; the teaser widget of the chapter introduction compares it with $2\bar X$.
✏️ Example 6 — A life test with exponential lifetimes
Five components with iid exponential lifetimes of mean $1000$ hours are put on test together, and the test stops at the third failure. (a) Find the mean and standard deviation of the duration of the test. (b) Find the density of the median lifetime of a sample of $3$ components with mean life $1$, and its mean. (c) What is the law of the total time on test, and why does it matter?
(a) The duration is $X_{(3)}$ with $n=5$ and $1/\lambda=1000$. By Theorem 4.11(1), $E[X_{(3)}]=1000\left(\frac15+\frac14+\frac13\right)=1000\cdot\frac{47}{60}=783.3$ hours, and $\operatorname{Var}(X_{(3)})=10^6\left(\frac1{25}+\frac1{16}+\frac19\right)=213\,611$, so the SD is $462.2$ hours. Stopping at the third failure saves a great deal of time, since $E[X_{(5)}]=1000\cdot\frac{137}{60}=2283$ hours.
(b) By Theorem 4.7(2) with $n=3$, $r=2$, $F(x)=1-e^{-x}$: $f_{(2)}(x)=6(1-e^{-x})e^{-x}\cdot e^{-x}=6(e^{-2x}-e^{-3x})$, $x\gt0$. By Theorem 4.11, $E[X_{(2)}]=\frac13+\frac12=\frac56$. Check: $\int_0^\infty6x(e^{-2x}-e^{-3x})dx=6\left(\frac14-\frac19\right)=\frac56$. ✓
(c) The total time on test is $T=X_{(1)}+X_{(2)}+X_{(3)}+2X_{(3)}$, the sum of the hours run by all components until the test stops. By Theorem 4.11(3), $T\sim\operatorname{Gamma}(3,\lambda)$ and $2\lambda T\sim\chi^2_6$. This is a pivot. It gives an exact confidence interval for the mean life from a censored test (§4.3 and §9.1), and $T/3$ is the MLE of $1/\lambda$.
Interactive 2 — Order Statistics and the Range: Simulation Against the Exact Density

Repeated samples of size $n$ are drawn (seeded generator), and the blue histogram collects $X_{(r)}$ or the range $R$. The purple curve is the exact density: Theorem 4.7 for $X_{(r)}$ and Theorem 4.10 for $R$, computed by numerical integration for the normal law. Try $r=1$ and $r=n$ for the exponential (an $\operatorname{Exp}(n)$ minimum, and a maximum that drifts to the right like $\log n$). Try the middle $r$ for the uniform (a symmetric beta law), and the range of the normal as $n$ grows, which is the basis of the quality-control estimate $\hat\sigma=\bar R/d_2$, where the constant $d_2=E[R]/\sigma$ depends on $n$.

53
samples drawn—
simulated mean, SD—
exact mean, SD—
exact law—

Sample Quantiles and the Efficiency of the Median

For large $n$, a sample quantile is approximately normal. Its variance depends on the population density at the quantile: the more probability there is near $\xi_p$, the more precisely $\xi_p$ is located.

📐 Theorem 4.12 — Asymptotic normality of sample quantiles

Let $0\lt p\lt1$, and let $F$ be differentiable at its $p$-quantile $\xi_p$ with $f(\xi_p)=F'(\xi_p)\gt0$. If $r_n/n=p+o(n^{-1/2})$ (for example $r_n=\lceil np\rceil$), then $$\sqrt n\left(X_{(r_n)}-\xi_p\right)\xrightarrow{d}N\left(0,\ \frac{p(1-p)}{f(\xi_p)^2}\right).$$ In particular the sample median $\tilde X_n$ of a population with median $m$ and $f(m)\gt0$ satisfies $\sqrt n(\tilde X_n-m)\xrightarrow{d}N\left(0,\frac{1}{4f(m)^2}\right)$.

Proof (sketch). Fix $t$ and put $x_n=\xi_p+t/\sqrt n$ and $p_n=F(x_n)$. By differentiability, $p_n=p+\frac{t f(\xi_p)}{\sqrt n}+o(n^{-1/2})$. By the counting identity, $$P\left(\sqrt n(X_{(r_n)}-\xi_p)\le t\right)=P\left(N_n\ge r_n\right),\qquad N_n\sim\operatorname{Bin}(n,p_n).$$ Standardise: $\frac{r_n-np_n}{\sqrt{np_n(1-p_n)}}\to\frac{-tf(\xi_p)}{\sqrt{p(1-p)}}$. The CLT for binomial counts holds uniformly in the success probability on compact subsets of $(0,1)$ (by the Berry–Esseen bound of §2.3), so $$P(N_n\ge r_n)\to1-\Phi\left(\frac{-tf(\xi_p)}{\sqrt{p(1-p)}}\right)=\Phi\left(\frac{tf(\xi_p)}{\sqrt{p(1-p)}}\right),$$ which is the cdf of $N\left(0,p(1-p)/f(\xi_p)^2\right)$ at $t$. The even-$n$ median is an average of two such order statistics whose difference is $o_p(n^{-1/2})$. ∎

Comparing two estimators that are both asymptotically normal and centred at the same parameter is done by the ratio of their asymptotic variances, the asymptotic relative efficiency (ARE). If $\operatorname{ARE}(T_1,T_2)=v_2/v_1$, then $T_2$ needs about $1/\operatorname{ARE}$ times as many observations as $T_1$ to match its precision. Section 4.2 develops efficiency systematically.

✏️ Example 7 — Prove or disprove: mean, median or midrange?
Decide which statements are true. (a) For normal samples, the sample median estimates $\mu$ more precisely than $\bar X$ when $n$ is large. (b) For samples from the Laplace law with density $\frac{1}{2b}e^{-|x-\theta|/b}$, the sample median is asymptotically twice as efficient as $\bar X$. (c) For Cauchy samples $C(\theta,\gamma)$, $\bar X$ is a consistent estimator of $\theta$. (d) For samples from $U(\theta-\frac12,\theta+\frac12)$, the midrange has variance of order $1/n^2$, while $\operatorname{Var}(\bar X)$ is of order $1/n$.
(a) False $f(\mu)=\frac{1}{\sigma\sqrt{2\pi}}$, so Theorem 4.12 gives the asymptotic variance $\frac{1}{4f(\mu)^2n}=\frac{\pi\sigma^2}{2n}$ for the median, against $\frac{\sigma^2}{n}$ for $\bar X$. The ARE of the median relative to the mean is $2/\pi\approx0.637$: the median needs about $57\%$ more observations.
(b) True $\operatorname{Var}(X)=2b^2$, so $\operatorname{Var}(\bar X)=2b^2/n$. Also $f(\theta)=\frac1{2b}$, so the median has asymptotic variance $\frac{1}{4n/(4b^2)}=\frac{b^2}{n}$. The ratio is $2$. Heavier tails favour the median.
(c) False By the reproductive property of the Cauchy law (§2.1, proved with characteristic functions in §2.2), $\bar X\sim C(\theta,\gamma)$ for every $n$, so averaging does not help at all. The median is consistent: $f(\theta)=\frac{1}{\pi\gamma}$, so it is asymptotically $N\left(\theta,\frac{\pi^2\gamma^2}{4n}\right)$.
(d) True Write $X_i=\theta-\frac12+U_i$. The midrange is $\theta-\frac12+\frac12(U_{(1)}+U_{(n)})$. By Theorem 4.9, $\operatorname{Var}(U_{(1)})=\operatorname{Var}(U_{(n)})=\frac{n}{(n+1)^2(n+2)}$ and $\operatorname{Cov}=\frac{1}{(n+1)^2(n+2)}$, so $$\operatorname{Var}(\text{midrange})=\frac14\cdot\frac{2n+2}{(n+1)^2(n+2)}=\frac{1}{2(n+1)(n+2)},$$ while $\operatorname{Var}(\bar X)=\frac{1}{12n}$. When the support has sharp edges, the extremes carry the information. Theorem 4.12 does not apply here, because the midrange is not a sample quantile.
⚠️ Counterexamples and pitfalls
  1. $\bar X$ and $S^2$ are independent only for normal samples (Geary). For $\operatorname{Exp}(1)$ samples $\operatorname{Cov}(\bar X,S^2)=2/n\ne0$. For symmetric non-normal laws they are uncorrelated but still dependent (Example 3).
  2. $(n-1)S^2/\sigma^2\sim\chi^2_{n-1}$ is not robust: $\operatorname{Var}(S^2)\approx(\kappa-1)\sigma^4/n$, not $2\sigma^4/n$, and this error does not go away as $n\to\infty$. The $t$ statistic, by contrast, is asymptotically $N(0,1)$ for every finite-variance population.
  3. Degrees of freedom: $\sum(X_i-\bar X)^2/\sigma^2\sim\chi^2_{n-1}$, but $\sum(X_i-\mu)^2/\sigma^2\sim\chi^2_n$. Mixing these up is the most common Part B error.
  4. $t$ and $F$ need independence of numerator and denominator. $(X_1-\bar X)/S$ is not $t$-distributed: it is bounded by $(n-1)/\sqrt n$ in absolute value, because $(n-1)S^2\ge\frac{n}{n-1}(X_1-\bar X)^2$.
  5. Moments of $t$ and $F$ may not exist: $t_1$ (Cauchy) has no mean, $t_2$ has infinite variance, and $E[F_{m,n}]=\frac{n}{n-2}$ depends on the denominator degrees of freedom and needs $n\gt2$.
  6. $F$ quantiles do not flip by themselves: $F_{m,n;1-\alpha}=1/F_{n,m;\alpha}$, with the degrees of freedom exchanged, and in general $F_{m,n;\alpha}\ne1/F_{m,n;1-\alpha}$ unless $m=n$.
  7. Order statistics are neither independent nor identically distributed, even though the $X_i$ are iid: $\operatorname{Cov}(U_{(r)},U_{(s)})\gt0$. The spacings of an exponential sample, once normalised, are independent (Rényi); uniform spacings are exchangeable but not independent (their sum is fixed).
  8. $E[X_{(r)}]\ne F^{-1}\left(\frac{r}{n+1}\right)$ in general: it is true for the uniform law, but for $\operatorname{Exp}(1)$ with $n=2$, $E[X_{(2)}]=\frac32$ while $F^{-1}(\frac23)=\log3=1.10$.
  9. The median is not always worse than the mean (Laplace, Cauchy), and the mean is not always better than everything else: the midrange beats it for uniform samples. Efficiency depends on the population.
  10. Asymptotic quantile results need $f(\xi_p)\gt0$: if the density vanishes at the median, the sample median converges more slowly than $n^{-1/2}$.
  11. The sample median of a discrete law need not be close to normal or even consistent for a unique value: for $\operatorname{Ber}(\frac12)$ the population median is not unique, and the sample median keeps jumping between $0$ and $1$.

Exercises

I
Sampling distributions and standard errors.
  • 1 A population has mean $50$ and SD $12$. For $n=36$, find $\operatorname{SE}(\bar X)$ and approximate $P(\bar X\gt53)$.
  • 2 Show that $E[S^2]=\sigma^2$ without assuming normality, and find the bias of $\hat\sigma^2=\frac1n\sum(X_i-\bar X)^2$.
  • 3 For a sample of size $n=4$ from $\operatorname{Ber}(\frac12)$, compute $\operatorname{Var}(S^2)$ from Theorem 4.1(4), and check your answer by listing the $16$ equally likely samples.
  • 4 Prove $\operatorname{Cov}(\bar X,S^2)=\mu_3/n$ and evaluate it for samples from $\operatorname{Exp}(1)$.
  • 5 Two independent samples of sizes $50$ and $40$ come from populations with SDs $6$ and $8$. Find the standard error of $\bar X_1-\bar X_2$.
  • 6 A pollster expects $p\approx0.3$. How large a sample makes $\operatorname{SE}(\hat p)\le0.01$?
  • 7 Show that $\operatorname{SE}(S^2)\approx\sigma^2\sqrt{(\kappa-1)/n}$ for large $n$, where $\kappa=\mu_4/\sigma^4$. Evaluate it for $\sigma^2=1$, $n=100$ and normal, uniform and exponential populations.
  • 8 Applied. A laboratory records $n=200$ repeated readings of a reference sample, with $s=2.5$ units. Assuming normal errors, give an approximate $95\%$ interval for the true measurement SD $\sigma$ using Theorem 4.6(3).
1 $\operatorname{SE}=12/6=2$. By the CLT, $P(\bar X\gt53)\approx P(Z\gt1.5)=0.0668$.
2 As in Theorem 4.1(2): $(n-1)S^2=\sum(X_i-\mu)^2-n(\bar X-\mu)^2$ has mean $n\sigma^2-\sigma^2$. So $E[\hat\sigma^2]=\frac{n-1}{n}\sigma^2$ and the bias is $-\sigma^2/n$.
3 Here $\sigma^2=\frac14$, and $X-\frac12=\pm\frac12$, so $\mu_4=\frac1{16}=\sigma^4$. Then $\operatorname{Var}(S^2)=\frac{1}{64}-\frac{1}{4\cdot3\cdot16}=\frac{1}{64}-\frac{1}{192}=\frac{1}{96}$. Check: with $k$ ones among $4$, $S^2=\frac{k(4-k)}{12}$. The counts $k=0,\dots,4$ occur with probabilities $\frac{1,4,6,4,1}{16}$, giving $S^2=0,\frac14,\frac13,\frac14,0$. So $E[S^2]=\frac{8\cdot\frac14+6\cdot\frac13}{16}=\frac14$ and $E[S^4]=\frac{8\cdot\frac1{16}+6\cdot\frac19}{16}=\frac{1}{16}\left(\frac12+\frac23\right)=\frac{7}{96}$. Hence $\operatorname{Var}=\frac{7}{96}-\frac{6}{96}=\frac1{96}$. ✓
4 See the proof of Theorem 4.1(3). For $\operatorname{Exp}(1)$, $\mu_3=E[(X-1)^3]=E[X^3]-3E[X^2]+3E[X]-1=6-6+3-1=2$, so $\operatorname{Cov}=2/n$.
5 $\sqrt{36/50+64/40}=\sqrt{0.72+1.6}=\sqrt{2.32}=1.523$.
6 $\sqrt{0.21/n}\le0.01\iff n\ge0.21/0.0001=2100$.
7 In Theorem 4.1(4), $\frac{\mu_4}{n}-\frac{(n-3)\sigma^4}{n(n-1)}=\frac{\sigma^4}{n}\left(\kappa-\frac{n-3}{n-1}\right)\approx\frac{(\kappa-1)\sigma^4}{n}$. With $\kappa=3$, $1.8$, $9$ this gives $\sqrt{2/100}=0.141$, $\sqrt{0.8/100}=0.089$ and $\sqrt{8/100}=0.283$.
8 $S$ is approximately $N(\sigma,\sigma^2/(2n))$, so with $\sigma$ estimated by $s$, $s\pm1.96\,s/\sqrt{2n}=2.5\pm1.96\times2.5/20=2.5\pm0.245$, i.e. $(2.255,\,2.745)$. The exact $\chi^2$ interval of §4.3 is almost the same at this sample size.
II
Normal samples: $\chi^2$, $t$ and $F$.
  • 1 $X_1,\dots,X_4$ are iid $N(0,1)$. Name the laws of (a) $X_1^2+X_2^2$; (b) $X_1/|X_2|$; (c) $\dfrac{X_1^2+X_2^2}{X_3^2+X_4^2}$; (d) $\dfrac{\sqrt3\,X_1}{\sqrt{X_2^2+X_3^2+X_4^2}}$.
  • 2 For a normal sample of size $15$, find $P(S^2\le1.5\sigma^2)$.
  • 3 Use the mgf to find the mean and variance of $\chi^2_k$, and show that $(\chi^2_k-k)/\sqrt{2k}\xrightarrow{d}N(0,1)$.
  • 4 Derive $\operatorname{Var}(t_k)=\frac{k}{k-2}$ from $E[1/V]$ for $V\sim\chi^2_k$, and explain why $t_2$ has infinite variance.
  • 5 Prove $F_{m,n;1-\alpha}=1/F_{n,m;\alpha}$. Given $F_{10,5;0.05}=4.735$, find the lower $5\%$ point of $F_{5,10}$.
  • 6 Show that $F_{2,2}$ has density $(1+x)^{-2}$ on $(0,\infty)$, and find $P(F_{2,2}\gt3)$.
  • 7 Prove the two-sample result of Theorem 4.5(2), stating exactly where independence is used.
  • 8 Let $X_1,X_2$ be iid. Show that $X_1+X_2$ and $X_1-X_2$ are always uncorrelated (when variances exist), that they are independent if the law is normal, and that they are dependent if the law is $U(0,1)$.
1 (a) $\chi^2_2=\operatorname{Exp}(\frac12)$, i.e. mean $2$. (b) $X_1/\sqrt{X_2^2/1}\sim t_1$, the standard Cauchy law. (c) $\frac{(X_1^2+X_2^2)/2}{(X_3^2+X_4^2)/2}\sim F_{2,2}$. (d) $X_1/\sqrt{\chi^2_3/3}\sim t_3$.
2 $P\left(\frac{14S^2}{\sigma^2}\le21\right)=P(\chi^2_{14}\le21)=0.898$.
3 $M(t)=(1-2t)^{-k/2}$, $M'(0)=k$, $M''(0)=k(k+2)$, so the variance is $2k$. A $\chi^2_k$ variable is a sum of $k$ iid $\chi^2_1$ variables with mean $1$ and variance $2$; apply the CLT.
4 $E[T^2]=k\,E[Z^2]\,E[1/V]$ and $E[1/V]=\frac{\Gamma(k/2-1)}{2\Gamma(k/2)}=\frac{1}{k-2}$ for $k\gt2$. For $k=2$, $V\sim\operatorname{Exp}(\frac12)$ has density $\frac12e^{-v/2}$, bounded away from $0$ near $v=0$, so $E[1/V]=\int_0^\infty\frac{1}{2v}e^{-v/2}dv=\infty$.
5 $1/F\sim F_{n,m}$, so $\alpha=P(1/F\gt F_{n,m;\alpha})=P(F\lt1/F_{n,m;\alpha})$; hence $1/F_{n,m;\alpha}$ is the lower $\alpha$ point of $F_{m,n}$. The lower $5\%$ point of $F_{5,10}$ is $1/4.735=0.2112$.
6 With $m=n=2$, Theorem 4.3(2) gives $\frac{1}{B(1,1)(1+x)^2}=(1+x)^{-2}$. The cdf is $\frac{x}{1+x}$, so $P(F_{2,2}\gt3)=\frac14$.
7 Independence is used three times. Within each sample, $\bar X\perp S_1^2$ and $\bar Y\perp S_2^2$ (Theorem 4.2). Between samples, every function of the $X$'s is independent of every function of the $Y$'s. So $\bar X-\bar Y$ is independent of $(S_1^2,S_2^2)$, and $(n_1-1)S_1^2/\sigma^2$ and $(n_2-1)S_2^2/\sigma^2$ are independent chi-squares, whose sum is $\chi^2_{n_1+n_2-2}$. The common $\sigma^2$ is what lets the $\sigma$'s cancel.
8 $\operatorname{Cov}(X_1+X_2,X_1-X_2)=\operatorname{Var}X_1-\operatorname{Var}X_2=0$. Normal: the pair is jointly normal and uncorrelated, hence independent. $U(0,1)$: if $X_1+X_2\gt1.8$, then both values exceed $0.8$, so $|X_1-X_2|\lt0.2$. Thus $P(X_1+X_2\gt1.8,\ |X_1-X_2|\gt0.5)=0$, although each event has positive probability.
III
Order statistics.
  • 1 For a $U(0,1)$ sample of size $7$, find the mean and variance of the sample median.
  • 2 For iid $\operatorname{Exp}(\lambda)$, find the densities of $X_{(1)}$ and $X_{(n)}$, and show that $X_{(1)}\sim\operatorname{Exp}(n\lambda)$.
  • 3 A sample of size $3$ is drawn from the density $2x$ on $(0,1)$. Find the density and the mean of the sample median.
  • 4 Write down the joint density of $(U_{(1)},U_{(n)})$ for a uniform sample and find $\operatorname{Cov}(U_{(1)},U_{(n)})$ by direct integration.
  • 5 Show that the range of $n$ iid $\operatorname{Exp}(1)$ variables has density $(n-1)e^{-w}(1-e^{-w})^{n-2}$ and mean $1+\frac12+\dots+\frac{1}{n-1}$. Evaluate the mean for $n=4$.
  • 6 For a continuous population, find (a) the probability that the maximum of $10$ observations exceeds the population $90$th percentile; (b) $P(X_{(1)}\lt m\lt X_{(6)})$ for a sample of size $6$, where $m$ is the population median.
  • 7 Applied. A 2-out-of-3 system (it works while at least two of its three components work) uses components with iid exponential lifetimes of mean $6$ years. Find the mean system life and its reliability $P(\text{life}\gt t)$.
  • 8 Find the asymptotic law of the sample median of an $\operatorname{Exp}(1)$ sample.
  • 9 Show that $C=F(X_{(n)})-F(X_{(1)})$ is $\operatorname{Beta}(n-1,2)$ for every continuous $F$, and find the least $n$ with $E[C]\ge0.95$.
1 $U_{(4)}\sim\operatorname{Beta}(4,4)$: mean $\frac12$, variance $\frac{4\cdot4}{8^2\cdot9}=\frac1{36}$.
2 $f_{(1)}=n(e^{-\lambda x})^{n-1}\lambda e^{-\lambda x}=n\lambda e^{-n\lambda x}$, which is $\operatorname{Exp}(n\lambda)$. Also $f_{(n)}=n(1-e^{-\lambda x})^{n-1}\lambda e^{-\lambda x}$.
3 $F=x^2$, so $f_{(2)}=6F(1-F)f=6x^2(1-x^2)\cdot2x=12x^3(1-x^2)$. The mean is $12\left(\frac15-\frac17\right)=\frac{24}{35}$.
4 $f(x,y)=n(n-1)(y-x)^{n-2}$ on $0\lt x\lt y\lt1$. Then $E[U_{(1)}U_{(n)}]=\frac{1}{n+2}$; for instance integrate $y$ over $(x,1)$ by parts, or use Theorem 4.9(2) with $r=1$, $s=n$. So $\operatorname{Cov}=\frac{1}{n+2}-\frac{n}{(n+1)^2}=\frac{1}{(n+1)^2(n+2)}$.
5 By Theorem 4.11(2), $R$ is the maximum of $n-1$ iid $\operatorname{Exp}(1)$ variables, with density $(n-1)(1-e^{-w})^{n-2}e^{-w}$ and mean $\sum_{i=1}^{n-1}\frac{1}{n-1-i+1}=\sum_{k=1}^{n-1}\frac1k$. For $n=4$ this is $\frac{11}{6}$.
6 (a) $1-0.9^{10}=0.651$. (b) The event fails exactly when all six observations lie on one side of $m$: $1-2\cdot2^{-6}=\frac{31}{32}=0.969$. Both answers hold for every continuous $F$.
7 The system life is $X_{(2)}$ with $n=3$ and $1/\lambda=6$: $E=6\left(\frac13+\frac12\right)=5$ years. With component reliability $q=e^{-t/6}$, the system works iff at least two components work: $P=3q^2(1-q)+q^3=3e^{-t/3}-2e^{-t/2}$.
8 $m=\log2$ and $f(m)=e^{-\log 2}=\frac12$, so $\frac{1}{4f(m)^2}=1$ and $\sqrt n(\tilde X_n-\log2)\xrightarrow{d}N(0,1)$.
9 By Theorem 4.9(4), $(F(X_{(1)}),F(X_{(n)}))$ has the law of $(U_{(1)},U_{(n)})$, and Theorem 4.9(3) applies. $E[C]=\frac{n-1}{n+1}\ge0.95\iff n\ge39$. These are Wilks' tolerance limits.
IV
Asymptotics and proofs.
  • 1 For a Poisson sample, find the limit law of $\sqrt n(S^2-\lambda)$ and compare it with that of $\sqrt n(\bar X-\lambda)$. (The central fourth moment of $\operatorname{Poi}(\lambda)$ is $\lambda+3\lambda^2$.)
  • 2 For the logistic density $f(x)=\frac{e^{-x}}{(1+e^{-x})^2}$ (variance $\pi^2/3$), find the ARE of the sample median relative to $\bar X$ as an estimator of the centre.
  • 3 Prove the Rényi representation directly for $n=2$: show that $X_{(1)}$ and $X_{(2)}-X_{(1)}$ are independent, with laws $\operatorname{Exp}(2\lambda)$ and $\operatorname{Exp}(\lambda)$.
  • 4 For a $U(0,\theta)$ sample, show that $n(\theta-X_{(n)})/\theta\xrightarrow{d}\operatorname{Exp}(1)$. What does this say about the rate at which $X_{(n)}$ approaches $\theta$?
  • 5 Show that for every continuous population the probability that the next observation $X_{n+1}$ exceeds the current maximum $X_{(n)}$ is $\frac{1}{n+1}$, and that $E[F(X_{(r)})]=\frac{r}{n+1}$.
  • 6 Prove that $T=\sqrt n(\bar X-\mu)/S\xrightarrow{d}N(0,1)$ for every population with $0\lt\sigma^2\lt\infty$, and deduce that $t_{n-1,\alpha}\to z_\alpha$.
1 $\mu_4-\sigma^4=\lambda+3\lambda^2-\lambda^2=\lambda+2\lambda^2$, so $\sqrt n(S^2-\lambda)\xrightarrow{d}N(0,\lambda+2\lambda^2)$, while $\sqrt n(\bar X-\lambda)\xrightarrow{d}N(0,\lambda)$. Both estimate $\lambda$, but $\bar X$ is better by the factor $1+2\lambda$.
2 The median is $0$ and $f(0)=\frac14$, so the median has asymptotic variance $\frac{1}{4\cdot\frac1{16}\,n}=\frac4n$. The mean has variance $\frac{\pi^2}{3n}$. $\operatorname{ARE}(\text{median},\bar X)=\frac{\pi^2/3}{4}=\frac{\pi^2}{12}=0.822$.
3 $(X_{(1)},X_{(2)})$ has density $2\lambda^2e^{-\lambda(x+y)}$ on $0\lt x\lt y$. With $u=x$, $v=y-x$ (Jacobian $1$) this becomes $2\lambda^2e^{-\lambda(2u+v)}=\left(2\lambda e^{-2\lambda u}\right)\left(\lambda e^{-\lambda v}\right)$ on $(0,\infty)^2$, a product.
4 $P\left(n(\theta-X_{(n)})/\theta\gt x\right)=P\left(X_{(n)}\lt\theta(1-\frac xn)\right)=\left(1-\frac xn\right)^n\to e^{-x}$. The error $\theta-X_{(n)}$ is of order $1/n$, not $1/\sqrt n$: the maximum is "super-efficient" for an endpoint.
5 By symmetry, each of $X_1,\dots,X_{n+1}$ is equally likely to be the largest, so the probability is $\frac{1}{n+1}$. Also $F(X_{(r)})\sim\operatorname{Beta}(r,n-r+1)$ has mean $\frac{r}{n+1}$: on average the order statistics split the probability into $n+1$ equal parts.
6 This is Theorem 4.6(4): $\sqrt n(\bar X-\mu)/\sigma\xrightarrow{d}N(0,1)$ and $S/\sigma\xrightarrow{P}1$; apply Slutsky. For normal samples $T\sim t_{n-1}$, so $t_{n-1}\xrightarrow{d}N(0,1)$. Since the limit cdf is continuous and strictly increasing, the quantiles converge as well: $t_{n-1,\alpha}\to z_\alpha$.
🎯 Exam corner CSIR Part BCSIR Part CGATE ST

E1. CSIR Part B GATE ST Let $X_1,X_2,X_3,X_4$ be iid $N(0,1)$. The distribution of $\dfrac{X_1+X_2}{\sqrt{X_3^2+X_4^2}}$ is (A) $t_2$; (B) $t_4$; (C) $F_{1,2}$; (D) standard Cauchy.

Idea Put the ratio into the form $Z/\sqrt{V/k}$ with $Z\perp V$.
Work $Z=(X_1+X_2)/\sqrt2\sim N(0,1)$ and $V=X_3^2+X_4^2\sim\chi^2_2$ are independent, and $\frac{X_1+X_2}{\sqrt{X_3^2+X_4^2}}=\frac{\sqrt2\,Z}{\sqrt V}=\frac{Z}{\sqrt{V/2}}\sim t_2$. Its square would be $F_{1,2}$, and $t_1$ (not $t_2$) is Cauchy.
Answer (A).

E2. CSIR Part B Let $U_1,\dots,U_4$ be iid $U(0,1)$. Then $P\left(\max U_i-\min U_i\le\frac12\right)$ equals (A) $\frac1{16}$; (B) $\frac{3}{16}$; (C) $\frac{5}{16}$; (D) $\frac{11}{16}$.

Idea The range of a uniform sample is $\operatorname{Beta}(n-1,2)$ (Theorem 4.9(3)).
Work $P(R\le w)=nw^{n-1}-(n-1)w^n=4\cdot\frac18-3\cdot\frac1{16}=\frac{5}{16}$. Option (B) is the value for $n=5$.
Answer (C).

E3. CSIR Part B GATE ST Four components have iid exponential lifetimes with rate $\lambda=2$ per year. The expected time until the second failure is (A) $\frac18$ year; (B) $\frac{7}{24}$ year; (C) $\frac{7}{12}$ year; (D) $\frac14$ year.

Idea Rényi: $E[X_{(r)}]=\frac1\lambda\sum_{i=1}^r\frac{1}{n-i+1}$.
Work $\frac12\left(\frac14+\frac13\right)=\frac12\cdot\frac{7}{12}=\frac{7}{24}$. (A) is $E[X_{(1)}]$, and (C) forgets the factor $1/\lambda$.
Answer (B).

E4. CSIR Part C Let $X_1,\dots,X_n$ ($n\ge3$) be iid $N(\mu,\sigma^2)$ with sample mean $\bar X$ and sample variance $S^2$. Which of the following are true? (One or more options may be correct.) (A) $\bar X$ and $\sum(X_i-\bar X)^2$ are independent. (B) $\bar X$ and the sample range $X_{(n)}-X_{(1)}$ are independent. (C) $(X_1-\bar X)/S$ has the $t_{n-1}$ law. (D) $n(\bar X-\mu)^2/S^2$ has the $F_{1,n-1}$ law.

(A) True Theorem 4.2.
(B) True $X_{(n)}-X_{(1)}=\max_i(X_i-\bar X)-\min_i(X_i-\bar X)$ is a function of the deviations, which are independent of $\bar X$ by Theorem 4.2(3).
(C) False $(n-1)S^2\ge(X_1-\bar X)^2+\frac{(X_1-\bar X)^2}{n-1}$, by Cauchy–Schwarz applied to the other deviations, which sum to $-(X_1-\bar X)$. So $|X_1-\bar X|/S\le(n-1)/\sqrt n$ is bounded, while $t_{n-1}$ is unbounded. The numerator and denominator are not independent.
(D) True It is the square of the $t_{n-1}$ pivot of Theorem 4.5(1), and $t_{n-1}^2=F_{1,n-1}$ (Theorem 4.4(2)).
Answer (A), (B) and (D).

E5. CSIR Part C Let $T\sim t_k$ and $F\sim F_{m,n}$. Which of the following are true? (A) $T^2\sim F_{1,k}$. (B) $1/F\sim F_{n,m}$. (C) $E[F]=\frac{n}{n-2}$ for every $m$ when $n\gt2$. (D) $\operatorname{Var}(T)=\frac{k}{k-2}$ for every $k\ge2$.

(A) True Theorem 4.4(2).
(B) True Theorem 4.4(3).
(C) True $E[F]=\frac nm E[U]E[1/V]=\frac{n}{m}\cdot m\cdot\frac{1}{n-2}$, free of $m$.
(D) False For $k=2$, $E[T^2]=2E[1/V]=\infty$ with $V\sim\chi^2_2$ (Exercise II.4). The formula needs $k\ge3$.
Answer (A), (B) and (C).

E6. CSIR Part C GATE ST Let $X_1,\dots,X_n$ ($n\ge2$) be iid $U(0,\theta)$. Which of the following are true? (A) $X_{(n)}/\theta\sim\operatorname{Beta}(n,1)$. (B) $X_{(1)}/X_{(n)}$ is independent of $X_{(n)}$. (C) $\sqrt n(\theta-X_{(n)})$ converges in distribution to a non-degenerate normal law. (D) $\operatorname{Var}(X_{(n)})=\frac{\theta^2}{n+2}$.

(A) True $X_{(n)}/\theta=U_{(n)}\sim\operatorname{Beta}(n,1)$ (Theorem 4.9(1)), with density $nu^{n-1}$.
(B) True $(X_{(1)},X_{(n)})$ has density $\frac{n(n-1)(y-x)^{n-2}}{\theta^n}$ on $0\lt x\lt y\lt\theta$. With $v=x/y$ and $y$ kept, the Jacobian is $y$, and the density becomes $\frac{ny^{n-1}}{\theta^n}\cdot(n-1)(1-v)^{n-2}$, a product of a function of $y$ and a function of $v$. (Basu's theorem in §4.2 gives a second proof.)
(C) False $n(\theta-X_{(n)})/\theta\xrightarrow{d}\operatorname{Exp}(1)$ (Exercise IV.4). Hence $\sqrt n(\theta-X_{(n)})=\frac{\theta}{\sqrt n}\cdot\frac{n(\theta-X_{(n)})}{\theta}\xrightarrow{P}0$, a degenerate limit.
(D) False $\operatorname{Var}(X_{(n)})=\theta^2\operatorname{Var}(U_{(n)})=\frac{n\theta^2}{(n+1)^2(n+2)}$.
Answer (A) and (B).
🧑‍🏫 For the teacher

Suggested plan — 3 lectures × 60 min.

  1. Statistics and normal sampling theory. Definition 4.1 and Theorem 4.1 (prove (1)–(3) on the board and leave (4) as a computation). Example 1. Then the Helmert proof of Theorem 4.2 in full; draw the picture of the deviation vector in the hyperplane $\sum x_i=0$. State Definition 4.2 and derive the $t$ density (Theorem 4.3(1)). Finish with Example 3 as a class discussion.
  2. $t$, $F$ and asymptotics. Theorem 4.3(2) via the beta law and Theorems 4.4–4.5, then Example 2 with a quantile table or scipy. Project Interactive 1: first the normal population with both statistics, then the exponential population with the variance statistic at $n=30$ (the tail rate stays far above $5\%$), then the $t$ statistic. Theorem 4.6 and the standard-error table; Example 4.
  3. Order statistics. The counting identity and Theorem 4.7 with the "boxes" heuristic, Theorem 4.8, the uniform beta laws (Theorem 4.9) and the range (Theorem 4.10). Rényi's representation via memorylessness before the Jacobian proof. Examples 5 and 6. Project Interactive 2 (range of the normal, minimum of the exponential), then Theorem 4.12 and Example 7. End with E4 and E6.

Discussion questions. (1) Why does the sample variance use the divisor $n-1$? Give two answers, one from unbiasedness and one from degrees of freedom. (2) For which populations would you estimate the centre by the mean, the median or the midrange, and why? (3) Order statistics of an iid sample are dependent. Which pairs are most strongly correlated, and why are normalised exponential spacings nevertheless independent?

Misconceptions to watch for. Believing that $\bar X\perp S^2$ for all populations; using $n$ instead of $n-1$ degrees of freedom; forming a "$t$ statistic" from dependent pieces; flipping $F$ quantiles without exchanging the degrees of freedom; treating order statistics as independent; assuming every estimator has error of order $1/\sqrt n$ (the maximum of a uniform sample has error of order $1/n$).

Suggested homework. I(2, 3, 7, 8), II(1, 4, 5, 8), III(1, 3, 5, 7, 9), IV(2, 4, 5); Exam corner E1, E4, E6.

🔬 Research–Teaching NexusResearch-tutored · optional, about 45 minutes

Research question. How many computer simulations does a safety analyst need before the largest simulated value can be trusted as a "95/95" upper bound, meaning one that exceeds the 95th percentile of the true output distribution with 95% confidence, whatever that distribution is?

Why it matters. Nuclear regulators in several countries accept "best estimate plus uncertainty" safety analyses. A reactor code is run many times with randomly perturbed inputs, and the peak cladding temperature is reported as an upper tolerance limit computed from the order statistics of the runs. Because the output distribution is unknown, the guarantee must be distribution-free. That is exactly what Theorem 4.9(4) provides. The same reasoning is used for tolerance limits in pharmaceutical manufacturing and in aircraft-component testing.

What to do.

  1. Show that $P\left(X_{(n)}\gt\xi_{0.95}\right)=1-0.95^n$ for every continuous $F$, and find the least $n$ for which this is at least $0.95$.
  2. For a two-sided limit, the proportion covered is $C=F(X_{(n)})-F(X_{(1)})\sim\operatorname{Beta}(n-1,2)$. Find the least $n$ with $P(C\ge0.95)\ge0.95$, using scipy's beta.sf.
  3. If the analyst reports the second largest value instead, so as to be robust to one wild run, how many runs are needed for the one-sided 95/95 guarantee? (Use Theorem 4.7(1).)
  4. Check your answers by simulation with three very different populations (normal, exponential, Cauchy), using a seeded generator.

Tools: Python with numpy and scipy, or a spreadsheet with a beta-distribution function.

Hand in: A one-page note giving the three sample sizes with their derivations, a table of simulated coverage frequencies for the three populations, and one paragraph on why the answers do not depend on the population.

Going further. Read about Wilks' 1941 paper on tolerance limits and its modern use in reactor safety codes. Then investigate how the required number of runs grows if the guarantee must hold simultaneously for two correlated outputs.

Instructor note: (1) $1-0.95^{59}=0.9515\ge0.95$ but $1-0.95^{58}=0.9490$, so $n=59$, the well-known "59 runs" rule. (2) $P(C\ge0.95)=0.95002$ at $n=93$ and $0.94786$ at $n=92$, so $n=93$. (3) The second largest exceeds $\xi_{0.95}$ iff at least two observations do: $1-0.95^n-0.05n\cdot0.95^{n-1}\ge0.95$ first holds at $n=93$, which happens to coincide with (2). Simulated coverages should agree with these values up to Monte Carlo error for all three populations, including the Cauchy.

🧪 Self-check quiz

0/5 correct

Q1. For a sample of size $n$ from $N(\mu,\sigma^2)$, the law of $\sum_{i=1}^n(X_i-\bar X)^2/\sigma^2$ is

Q2. If $T\sim t_k$, then $T^2$ has the law

Q3. For an iid $U(0,1)$ sample of size $n$, the $r$-th order statistic $U_{(r)}$ has the law

Q4. A population has SD $5$. The standard error of $\bar X$ for $n=100$ is

Q5. For large normal samples, the variance of the sample median as an estimator of $\mu$ is approximately