4.1 Sampling Distributions and Order Statistics
Every statistic is a random variable with a law of its own. For normal samples the laws are exact, namely $\chi^2$, $t$ and $F$. For large samples the CLT and the delta method give normal approximations. For order statistics a counting argument gives everything, from the density of the median to the law of the range.
A quality engineer weighs $n=25$ tablets and finds an average of $\bar x=502.3$ mg and a standard deviation of $s=4.1$ mg. The question that matters is not what these numbers are but how far they could be from the truth. If the production line were sampled again, $\bar x$ and $s$ would change. They are values of the statistics $\bar X$ and $S$, which are functions of random variables and therefore random variables themselves. Their distributions, the sampling distributions, are the bridge between probability and inference. Every confidence interval of §4.3, every test of Chapter 5 and every comparison of estimators in §4.2 is a statement about a sampling distribution.
This section computes the sampling distributions that statistics uses most. Means and variances of $\bar X$ and $S^2$ hold for every population. For normal populations there is an exact theory: $\bar X$ and $S^2$ are independent, $(n-1)S^2/\sigma^2$ is $\chi^2_{n-1}$, and the studentised mean has Student's $t$ law. For other populations, the central limit theorem and the delta method of §2.3 supply asymptotic laws and standard errors. Finally, order statistics, the sorted sample values, have laws that follow from a single counting argument, and uniform and exponential samples give especially clean answers. CSIR Part B questions often ask you to identify a $\chi^2$, $t$ or $F$ law or to compute the mean of an order statistic. Part C questions ask which statements about independence and distribution are true, and many of the false options are statements that hold only for normal samples.
Statistics, Sampling Distributions and Standard Errors
A random sample of size $n$ from a distribution $F$ is a collection $X_1,\dots,X_n$ of iid random variables with common cdf $F$. A statistic is a random variable $T=t(X_1,\dots,X_n)$, where $t$ is a measurable function that does not involve unknown parameters. The distribution of $T$ is its sampling distribution, and the standard deviation $\sqrt{\operatorname{Var}(T)}$ is the standard error of $T$, written $\operatorname{SE}(T)$. When the standard error depends on unknown parameters, replacing them by estimates gives the estimated standard error $\widehat{\operatorname{SE}}(T)$.
The basic statistics are the sample mean $\bar X=\frac1n\sum_{i=1}^nX_i$, the sample variance $S^2=\frac{1}{n-1}\sum_{i=1}^n(X_i-\bar X)^2$ (note the divisor $n-1$), the sample standard deviation $S=\sqrt{S^2}$, and the sample moments $m_r'=\frac1n\sum X_i^r$ and $m_r=\frac1n\sum(X_i-\bar X)^r$. We write $\mu=E[X_1]$, $\sigma^2=\operatorname{Var}(X_1)$, and $\mu_r=E[(X_1-\mu)^r]$ for the central moments of the population.
The algebraic identity $$\sum_{i=1}^n(X_i-\bar X)^2=\sum_{i=1}^n(X_i-c)^2-n(\bar X-c)^2\qquad(c\in\mathbb{R})$$ (expand $X_i-c=(X_i-\bar X)+(\bar X-c)$ and use $\sum(X_i-\bar X)=0$) is used constantly below. With $c=\mu$ it explains the divisor $n-1$.
Let $X_1,\dots,X_n$ be a random sample with $\sigma^2\lt\infty$ and $n\ge2$. Then
- $E[\bar X]=\mu$ and $\operatorname{Var}(\bar X)=\dfrac{\sigma^2}{n}$, so $\operatorname{SE}(\bar X)=\dfrac{\sigma}{\sqrt n}$;
- $E[S^2]=\sigma^2$, while $\hat\sigma^2=\frac1n\sum(X_i-\bar X)^2$ has $E[\hat\sigma^2]=\frac{n-1}{n}\sigma^2$;
- if $E[X_1^3]$ exists, then $\operatorname{Cov}(\bar X,S^2)=\dfrac{\mu_3}{n}$; in particular $\bar X$ and $S^2$ are uncorrelated for every symmetric population;
- if $E[X_1^4]$ exists, then $\operatorname{Var}(S^2)=\dfrac{\mu_4}{n}-\dfrac{(n-3)\,\sigma^4}{n(n-1)}$.
Proof. Since $S^2$ is unchanged when every $X_i$ is replaced by $X_i-\mu$, we may assume $\mu=0$ in (2)–(4). (1) By linearity $E[\bar X]=\mu$, and by independence $\operatorname{Var}(\bar X)=\frac{1}{n^2}\sum\operatorname{Var}(X_i)=\frac{\sigma^2}{n}$.
(2) The identity with $c=0$ gives $(n-1)S^2=\sum X_i^2-n\bar X^2$. Taking expectations, $E[\sum X_i^2]=n\sigma^2$ and $E[n\bar X^2]=n\operatorname{Var}(\bar X)=\sigma^2$, so $(n-1)E[S^2]=(n-1)\sigma^2$. Also $\hat\sigma^2=\frac{n-1}{n}S^2$.
(3) With $\mu=0$, $\operatorname{Cov}(\bar X,S^2)=E[\bar XS^2]$. Now $E[\bar X\sum_jX_j^2]=\frac1n\sum_i\sum_jE[X_iX_j^2]$. By independence and $E[X_i]=0$, only the $n$ terms with $i=j$ survive, so this equals $\mu_3$. Similarly $E[\bar X^3]=\frac{1}{n^3}\sum_{i,j,k}E[X_iX_jX_k]=\frac{n\mu_3}{n^3}$, since only the $n$ terms with $i=j=k$ survive. Hence $(n-1)E[\bar XS^2]=\mu_3-n\cdot\frac{\mu_3}{n^2}=\frac{(n-1)\mu_3}{n}$.
(4) (Sketch.) Write $(n-1)S^2=\frac{n-1}{n}\sum_iX_i^2-\frac1n\sum_{i\ne j}X_iX_j$. Square this and take expectations term by term. A product of four $X$'s has nonzero mean only when the indices pair up, giving $\mu_4$ (all four equal) or $\sigma^4$ (two distinct pairs). Collecting the terms gives $(n-1)^2E[S^4]=\frac{(n-1)^2}{n}\mu_4+\frac{(n-1)(n^2-2n+3)}{n}\sigma^4$. Subtracting $(n-1)^2\sigma^4$ yields (4). (Exercise I.3 checks the formula on a two-point population, and the widget below checks it by simulation.) ∎
For a normal population $\mu_4=3\sigma^4$, and (4) becomes $\operatorname{Var}(S^2)=\frac{2\sigma^4}{n-1}$. Theorem 4.2 below explains why. Writing $\kappa=\mu_4/\sigma^4$ for the kurtosis, (4) shows that $\operatorname{Var}(S^2)\approx(\kappa-1)\sigma^4/n$ for large $n$. Heavy-tailed populations therefore make $S^2$ far less precise than the normal theory suggests: the factor $\kappa-1$ is $2$ for normal, $0.8$ for uniform and $8$ for exponential populations.
Sampling from a Normal Population: $\chi^2$, $t$ and $F$
For normal samples the sampling distributions are known exactly, and three families of laws appear. The chi-square law was introduced in §2.1 as a gamma law. The $t$ and $F$ laws are new.
- For $k\in\mathbb{N}$, the chi-square law with $k$ degrees of freedom is $\chi^2_k=\operatorname{Gamma}(\tfrac k2,\tfrac12)$, with density $\dfrac{x^{k/2-1}e^{-x/2}}{2^{k/2}\Gamma(k/2)}$, $x\gt0$. It is the law of $Z_1^2+\dots+Z_k^2$ for iid $N(0,1)$ variables $Z_j$ (§2.1).
- If $Z\sim N(0,1)$ and $V\sim\chi^2_k$ are independent, then $T=\dfrac{Z}{\sqrt{V/k}}$ has Student's $t$ law with $k$ degrees of freedom, $t_k$.
- If $U\sim\chi^2_m$ and $V\sim\chi^2_n$ are independent, then $F=\dfrac{U/m}{V/n}$ has Snedecor's $F$ law with $(m,n)$ degrees of freedom, $F_{m,n}$. The order matters: $m$ belongs to the numerator.
Upper quantiles are written $\chi^2_{k,\alpha}$, $t_{k,\alpha}$ and $F_{m,n;\alpha}$, so that, for example, $P(T\gt t_{k,\alpha})=\alpha$.
Let $X_1,\dots,X_n$ be iid $N(\mu,\sigma^2)$ with $n\ge2$. Then
- $\bar X\sim N(\mu,\sigma^2/n)$;
- $\dfrac{(n-1)S^2}{\sigma^2}=\dfrac{\sum(X_i-\bar X)^2}{\sigma^2}\sim\chi^2_{n-1}$;
- $\bar X$ is independent of the vector of deviations $(X_1-\bar X,\dots,X_n-\bar X)$, and hence of $S^2$, of the range $X_{(n)}-X_{(1)}$, of the mean deviation $\frac1n\sum|X_i-\bar X|$, and of every other function of the deviations.
Conversely (Geary's theorem, statement): if $X_1,\dots,X_n$ ($n\ge2$) are iid and $\bar X$ and $S^2$ are independent, then the common law is normal (or degenerate).
Proof. Put $Z_i=(X_i-\mu)/\sigma$, which are iid $N(0,1)$, and $Z=(Z_1,\dots,Z_n)^{\mathsf T}$. The joint density of $Z$ is $(2\pi)^{-n/2}\exp(-\tfrac12\|z\|^2)$, which depends on $z$ only through its length. Let $A$ be an orthogonal $n\times n$ matrix whose first row is $a_1=\frac{1}{\sqrt n}(1,\dots,1)$. One explicit choice is the Helmert matrix, whose $k$-th row for $k=2,\dots,n$ is $$a_k=\tfrac{1}{\sqrt{k(k-1)}}\,(\underbrace{1,\dots,1}_{k-1},\,-(k-1),\,0,\dots,0).$$ Each row has length $1$ because $\frac{(k-1)+(k-1)^2}{k(k-1)}=1$. Each $a_k$ with $k\ge2$ has entries summing to $0$, so it is orthogonal to $a_1$. For $2\le j\lt k$, $a_k$ is constant on the first $j$ coordinates, which are the only nonzero ones of $a_j$, so $a_j\cdot a_k$ is a multiple of the sum of the entries of $a_j$, which is $0$.
Let $Y=AZ$. The map $z\mapsto Az$ is a bijection with $|\det A|=1$ and $\|Az\|=\|z\|$. By the Jacobian formula (§1.4) $Y$ has density $(2\pi)^{-n/2}\exp(-\tfrac12\|y\|^2)$, so $Y_1,\dots,Y_n$ are iid $N(0,1)$. Now $Y_1=a_1\cdot Z=\sqrt n\,\bar Z$. Since $A^{\mathsf T}A=I$, we have $Z=\sum_kY_ka_k^{\mathsf T}$, and the $k=1$ term is $\bar Z\,(1,\dots,1)^{\mathsf T}$. Hence $$(Z_1-\bar Z,\dots,Z_n-\bar Z)^{\mathsf T}=\sum_{k=2}^nY_ka_k^{\mathsf T},$$ which is a function of $(Y_2,\dots,Y_n)$ alone. Since $Y_1$ is independent of $(Y_2,\dots,Y_n)$, $\bar Z$ is independent of the deviations, which proves (3) because $X_i-\bar X=\sigma(Z_i-\bar Z)$ and $\bar X=\mu+\sigma\bar Z$. By orthonormality of the $a_k$, $$\sum_i(Z_i-\bar Z)^2=\Big\|\sum_{k\ge2}Y_ka_k\Big\|^2=\sum_{k=2}^nY_k^2\sim\chi^2_{n-1},$$ which is (2). Finally $\bar X=\mu+\sigma Y_1/\sqrt n\sim N(\mu,\sigma^2/n)$, which is (1). ∎
The proof shows where the "lost" degree of freedom goes. The $n$ deviations satisfy one linear constraint, $\sum(X_i-\bar X)=0$, so they live in an $(n-1)$-dimensional subspace, and their squared length is a sum of $n-1$ independent squared standard normals.
- $t_k$ has density $$f(t)=\frac{\Gamma\left(\frac{k+1}{2}\right)}{\sqrt{k\pi}\,\Gamma\left(\frac k2\right)}\left(1+\frac{t^2}{k}\right)^{-(k+1)/2},\qquad t\in\mathbb{R}.$$ It is symmetric about $0$, and $E|T|^r\lt\infty$ if and only if $r\lt k$. Hence $E[T]=0$ for $k\ge2$, and $\operatorname{Var}(T)=\dfrac{k}{k-2}$ for $k\ge3$; for $k=2$ the variance is infinite.
- $F_{m,n}$ has density $$f(x)=\frac{(m/n)^{m/2}\,x^{m/2-1}}{B\left(\frac m2,\frac n2\right)\left(1+\frac{m}{n}x\right)^{(m+n)/2}},\qquad x\gt0.$$ Its mean is $E[F]=\dfrac{n}{n-2}$ for $n\gt2$ (it does not depend on $m$), and $\operatorname{Var}(F)=\dfrac{2n^2(m+n-2)}{m(n-2)^2(n-4)}$ for $n\gt4$.
Proof. (1) Conditionally on $V=v$, $T=Z\sqrt{k/v}$ is $N(0,k/v)$, with density $\sqrt{v/k}\,\varphi(t\sqrt{v/k})$. Averaging over $V$, $$f(t)=\int_0^\infty\sqrt{\tfrac{v}{2\pi k}}\,e^{-t^2v/(2k)}\,\frac{v^{k/2-1}e^{-v/2}}{2^{k/2}\Gamma(k/2)}\,dv=\frac{1}{\sqrt{2\pi k}\,2^{k/2}\Gamma(k/2)}\int_0^\infty v^{\frac{k+1}{2}-1}e^{-\frac v2\left(1+\frac{t^2}{k}\right)}dv.$$ The last integral is a gamma integral, equal to $\Gamma(\frac{k+1}2)\left(\frac{2}{1+t^2/k}\right)^{(k+1)/2}$, and $2^{(k+1)/2}/(\sqrt{2\pi k}\,2^{k/2})=1/\sqrt{k\pi}$. Moments: by independence, $E|T|^r=k^{r/2}E|Z|^r\,E[V^{-r/2}]$. By the gamma moment formula of §2.1, $E[V^{-r/2}]\lt\infty$ iff $-r/2\gt-k/2$, i.e. $r\lt k$. Finally $E[T^2]=k\,E[Z^2]\,E[V^{-1}]=k\cdot\frac{1}{k-2}$.
(2) Since $\chi^2_j=\operatorname{Gamma}(\frac j2,\frac12)$, the gamma-ratio theorem of §2.1 shows that $B=U/(U+V)\sim\operatorname{Beta}(\frac m2,\frac n2)$. Now $F=\frac nm\cdot\frac{B}{1-B}$, an increasing bijection of $(0,1)$ onto $(0,\infty)$ with inverse $b=\frac{mx}{n+mx}$ and $\frac{db}{dx}=\frac{mn}{(n+mx)^2}$. Substituting $b$ and $1-b=\frac{n}{n+mx}$ into the beta density and multiplying by $\frac{db}{dx}$ gives $$\frac{(mx)^{m/2-1}n^{n/2-1}}{B(\frac m2,\frac n2)(n+mx)^{(m+n)/2-2}}\cdot\frac{mn}{(n+mx)^2},$$ which simplifies to the stated density. For the moments, $E[F]=\frac nm E[U]E[V^{-1}]=\frac nm\cdot m\cdot\frac{1}{n-2}$ and $E[F^2]=\frac{n^2}{m^2}E[U^2]E[V^{-2}]=\frac{n^2}{m^2}\cdot\frac{m(m+2)}{(n-2)(n-4)}$. Then $$\operatorname{Var}(F)=\frac{n^2\left[(m+2)(n-2)-m(n-4)\right]}{m(n-2)^2(n-4)}=\frac{2n^2(m+n-2)}{m(n-2)^2(n-4)}.\ ∎$$
- $\chi^2_k$ has mean $k$, variance $2k$ and mgf $(1-2t)^{-k/2}$ for $t\lt\frac12$. Independent chi-squares add: $\chi^2_a+\chi^2_b=\chi^2_{a+b}$. Also $\chi^2_2=\operatorname{Exp}(\frac12)$, and $(\chi^2_k-k)/\sqrt{2k}\xrightarrow{d}N(0,1)$ as $k\to\infty$.
- If $T\sim t_k$, then $T^2\sim F_{1,k}$. Also $t_1$ is the standard Cauchy law, and $t_k\xrightarrow{d}N(0,1)$ as $k\to\infty$.
- If $F\sim F_{m,n}$, then $1/F\sim F_{n,m}$. Consequently the lower quantile is $F_{m,n;1-\alpha}=1/F_{n,m;\alpha}$.
- If $F\sim F_{m,n}$, then $\dfrac{mF}{n+mF}\sim\operatorname{Beta}\left(\frac m2,\frac n2\right)$; in particular $F_{2,2}$ has cdf $\frac{x}{1+x}$.
- If $F_n\sim F_{m,n}$ with $m$ fixed, then $mF_n\xrightarrow{d}\chi^2_m$ as $n\to\infty$.
Proof. (1) is §2.1 (gamma moments, the gamma mgf and the reproductive property) together with the CLT applied to a sum of $k$ iid $\chi^2_1$ variables. (2) $T^2=\frac{Z^2/1}{V/k}$ and $Z^2\sim\chi^2_1$ is independent of $V$. For $k=1$ the density in Theorem 4.3 is $\frac{\Gamma(1)}{\sqrt\pi\,\Gamma(1/2)}(1+t^2)^{-1}=\frac{1}{\pi(1+t^2)}$. For the limit, write $T_k=Z/\sqrt{V_k/k}$ where $V_k$ is a sum of $k$ iid $\chi^2_1$ variables. By the weak law, $V_k/k\xrightarrow{P}1$, and Slutsky's theorem (§2.3) gives $T_k\xrightarrow{d}Z$. (3) $1/F=\frac{V/n}{U/m}$ is of the defining form with the roles exchanged. Hence $P(F\le x)=P(1/F\ge1/x)$; putting $x=1/F_{n,m;\alpha}$ makes the right side $\alpha$. (4) This is the variable $B$ of the proof of Theorem 4.3. For $m=n=2$, $B$ is uniform, so $P(F\le x)=P\left(B\le\frac{x}{1+x}\right)=\frac{x}{1+x}$. (5) $mF_n=U/(V_n/n)$ and $V_n/n\xrightarrow{P}1$; apply Slutsky. ∎
- (One sample) If $X_1,\dots,X_n$ are iid $N(\mu,\sigma^2)$, then $\dfrac{\sqrt n(\bar X-\mu)}{S}\sim t_{n-1}$. If $\mu$ is known, $\dfrac{1}{\sigma^2}\sum(X_i-\mu)^2\sim\chi^2_n$.
- (Two samples, common variance) Let $X_1,\dots,X_{n_1}$ be iid $N(\mu_1,\sigma^2)$ and $Y_1,\dots,Y_{n_2}$ iid $N(\mu_2,\sigma^2)$, independent of each other. Define the pooled variance $S_p^2=\dfrac{(n_1-1)S_1^2+(n_2-1)S_2^2}{n_1+n_2-2}$. Then $$\frac{(\bar X-\bar Y)-(\mu_1-\mu_2)}{S_p\sqrt{\frac{1}{n_1}+\frac{1}{n_2}}}\sim t_{n_1+n_2-2}.$$
- (Variance ratio) With the same two samples but variances $\sigma_1^2,\sigma_2^2$: $\dfrac{S_1^2/\sigma_1^2}{S_2^2/\sigma_2^2}\sim F_{n_1-1,\,n_2-1}$.
Proof. (1) $Z=\sqrt n(\bar X-\mu)/\sigma\sim N(0,1)$ and $V=(n-1)S^2/\sigma^2\sim\chi^2_{n-1}$ are independent by Theorem 4.2, and $Z/\sqrt{V/(n-1)}=\sqrt n(\bar X-\mu)/S$. The second statement is Definition 4.2(1) applied to $(X_i-\mu)/\sigma$. (2) $\bar X-\bar Y\sim N\left(\mu_1-\mu_2,\sigma^2(\frac1{n_1}+\frac1{n_2})\right)$. By Theorem 4.2 applied to each sample and by the independence of the samples, the four statistics $\bar X,S_1^2,\bar Y,S_2^2$ are mutually independent, and $(n_1+n_2-2)S_p^2/\sigma^2$ is a sum of independent $\chi^2_{n_1-1}$ and $\chi^2_{n_2-1}$ variables, hence $\chi^2_{n_1+n_2-2}$. Divide the standardised difference by $\sqrt{S_p^2/\sigma^2}$. (3) The ratio is $\frac{\chi^2_{n_1-1}/(n_1-1)}{\chi^2_{n_2-1}/(n_2-1)}$ with independent numerator and denominator. ∎
These three statistics are pivots: they depend on the unknown parameters, but their laws do not. That is exactly what is needed to build the confidence intervals of §4.3 and the $t$- and $F$-tests of Chapter 5.
chi2.sf(18, 9)). For comparison, $\chi^2_{9,0.05}=16.92$.Asymptotic Distributions and Standard Errors
For most statistics from non-normal populations, exact sampling distributions are out of reach. For large $n$, however, the CLT, Slutsky's theorem and the delta method of §2.3 give normal approximations whose variances provide the standard errors.
Let $X_1,X_2,\dots$ be iid.
- If $E[X_1^{2r}]\lt\infty$, then $\sqrt n\,(m_r'-\mu_r')\xrightarrow{d}N\left(0,\ \mu_{2r}'-\mu_r'^2\right)$, where $\mu_r'=E[X_1^r]$.
- If $\mu_4\lt\infty$, then $\sqrt n\,(S^2-\sigma^2)\xrightarrow{d}N(0,\ \mu_4-\sigma^4)$, and the same holds for $\hat\sigma^2$.
- If also $\sigma\gt0$, then $\sqrt n\,(S-\sigma)\xrightarrow{d}N\left(0,\ \dfrac{\mu_4-\sigma^4}{4\sigma^2}\right)$. For a normal population the limits in (2) and (3) are $N(0,2\sigma^4)$ and $N(0,\sigma^2/2)$.
- If $0\lt\sigma^2\lt\infty$, the studentised mean satisfies $\sqrt n(\bar X-\mu)/S\xrightarrow{d}N(0,1)$ for every population.
Proof. (1) The variables $X_i^r$ are iid with mean $\mu_r'$ and variance $\mu_{2r}'-\mu_r'^2$; apply the CLT. (2) By the identity with $c=\mu$, $$\hat\sigma^2=\frac1n\sum(X_i-\mu)^2-(\bar X-\mu)^2.$$ The $W_i=(X_i-\mu)^2$ are iid with mean $\sigma^2$ and variance $\mu_4-\sigma^4$, so $\sqrt n(\bar W-\sigma^2)\xrightarrow{d}N(0,\mu_4-\sigma^4)$. Next, $\sqrt n(\bar X-\mu)^2=\left[\sqrt n(\bar X-\mu)\right](\bar X-\mu)$ is a product of a term bounded in probability and a term tending to $0$ in probability, so it tends to $0$ in probability. By Slutsky, $\sqrt n(\hat\sigma^2-\sigma^2)$ has the same limit. Finally $\sqrt n(S^2-\hat\sigma^2)=\frac{\sqrt n}{n-1}\hat\sigma^2\xrightarrow{P}0$. (3) Apply the delta method with $g(x)=\sqrt x$ and $g'(\sigma^2)=\frac1{2\sigma}$; for the normal case use $\mu_4=3\sigma^4$. (4) $S\xrightarrow{P}\sigma$ by (2) and the continuous mapping theorem; then apply the CLT and Slutsky. ∎
| statistic | estimates | standard error (exact or large-sample) |
|---|---|---|
| $\bar X$ | $\mu$ | $\sigma/\sqrt n$ (exact) |
| $\hat p$ (sample proportion) | $p$ | $\sqrt{p(1-p)/n}$ (exact) |
| $\bar X_1-\bar X_2$ (independent samples) | $\mu_1-\mu_2$ | $\sqrt{\sigma_1^2/n_1+\sigma_2^2/n_2}$ (exact) |
| $S^2$ | $\sigma^2$ | $\sqrt{\operatorname{Var}(S^2)}$ from Theorem 4.1(4); $\sigma^2\sqrt{2/(n-1)}$ if normal |
| $S$ | $\sigma$ | $\approx\sqrt{(\mu_4-\sigma^4)/(4\sigma^2n)}$; $\approx\sigma/\sqrt{2n}$ if normal |
| sample $p$-quantile | $\xi_p$ | $\approx\sqrt{p(1-p)}\,/\,(f(\xi_p)\sqrt n)$ (Theorem 4.12) |
| sample median | median $m$ | $\approx1/(2f(m)\sqrt n)$; $\approx\sigma\sqrt{\pi/(2n)}$ if normal |
| $g(\bar X)$ | $g(\mu)$ | $\approx|g'(\mu)|\,\sigma/\sqrt n$ (delta method) |
| sample correlation $r$ | $\rho$ | $\approx(1-\rho^2)/\sqrt n$ (bivariate normal; §7.2) |
Replace unknown quantities by consistent estimates to get estimated standard errors. Slutsky's theorem guarantees that the studentised statistic is still asymptotically $N(0,1)$.
Order Statistics
Sorting a sample $X_1,\dots,X_n$ into increasing order gives the order statistics $X_{(1)}\le X_{(2)}\le\dots\le X_{(n)}$. Thus $X_{(1)}=\min X_i$ and $X_{(n)}=\max X_i$. Derived statistics:
- the sample range $R=X_{(n)}-X_{(1)}$ and the midrange $\frac12(X_{(1)}+X_{(n)})$;
- the sample median: $X_{(k+1)}$ if $n=2k+1$, and $\frac12(X_{(k)}+X_{(k+1)})$ if $n=2k$; more generally the sample $p$-quantile $X_{(\lceil np\rceil)}$;
- the spacings $X_{(i)}-X_{(i-1)}$, $i=2,\dots,n$.
If $F$ is continuous, ties occur with probability $0$, so $X_{(1)}\lt\dots\lt X_{(n)}$ almost surely.
Order statistics are not independent: they are tied together by the ordering, and they are not identically distributed either. Yet one counting idea gives all their laws. The number of observations at or below $x$, $$N(x)=\#\{i:X_i\le x\},$$ is $\operatorname{Bin}(n,F(x))$, and $X_{(r)}\le x$ exactly when $N(x)\ge r$.
Let $X_1,\dots,X_n$ be iid with cdf $F$ and let $1\le r\le n$.
- For every $F$ (discrete, continuous or mixed), $$P(X_{(r)}\le x)=\sum_{j=r}^n\binom nj F(x)^j\left(1-F(x)\right)^{n-j}=\frac{n!}{(r-1)!\,(n-r)!}\int_0^{F(x)}t^{r-1}(1-t)^{n-r}\,dt.$$
- If $F$ has a density $f$, then $X_{(r)}$ has density $$f_{(r)}(x)=\frac{n!}{(r-1)!\,(n-r)!}\,F(x)^{r-1}\left(1-F(x)\right)^{n-r}f(x).$$
For $r=n$ and $r=1$ this gives $F_{(n)}=F^n$ and $1-F_{(1)}=(1-F)^n$, the maximum and minimum laws of §1.4.
Proof. (1) The events $\{X_i\le x\}$ are independent with probability $F(x)$, so $N(x)\sim\operatorname{Bin}(n,F(x))$. Since $\{X_{(r)}\le x\}=\{N(x)\ge r\}$, the first formula follows. The second is the beta–binomial tail identity of §2.1: $$\sum_{j\ge r}\binom nj p^j(1-p)^{n-j}=\frac{n!}{(r-1)!(n-r)!}\int_0^pt^{r-1}(1-t)^{n-r}dt.$$ (2) If $F$ has density $f$, then $F$ is absolutely continuous with $F'=f$ almost everywhere. Differentiating the integral form by the chain rule gives the density. ∎
A heuristic worth remembering. For $X_{(r)}$ to lie in $[x,x+dx]$, the sample must split into $r-1$ values below $x$, one value in $[x,x+dx]$ and $n-r$ values above. The multinomial probability of this is $$\frac{n!}{(r-1)!\,1!\,(n-r)!}\,F(x)^{r-1}\,f(x)\,dx\,\left(1-F(x)\right)^{n-r}.$$ The same "boxes" argument gives every joint density below.
Let $X_1,\dots,X_n$ be iid with density $f$ and cdf $F$.
- The vector $(X_{(1)},\dots,X_{(n)})$ has density $n!\,f(y_1)f(y_2)\cdots f(y_n)$ on $\{y_1\lt y_2\lt\dots\lt y_n\}$, and $0$ elsewhere.
- For $1\le r\lt s\le n$, the pair $(X_{(r)},X_{(s)})$ has density, for $x\lt y$, $$\frac{n!}{(r-1)!\,(s-r-1)!\,(n-s)!}\,F(x)^{r-1}\left[F(y)-F(x)\right]^{s-r-1}\left[1-F(y)\right]^{n-s}f(x)f(y).$$ In particular $(X_{(1)},X_{(n)})$ has density $n(n-1)\left[F(y)-F(x)\right]^{n-2}f(x)f(y)$ for $x\lt y$.
Proof. (1) Up to a null set, $\mathbb{R}^n$ splits into the $n!$ regions $D_\pi=\{x_{\pi(1)}\lt\dots\lt x_{\pi(n)}\}$, one for each permutation $\pi$. On $D_\pi$ the sorting map is the linear map $x\mapsto(x_{\pi(1)},\dots,x_{\pi(n)})$, which has $|\det|=1$ and carries $D_\pi$ onto the ordered region $D=\{y_1\lt\dots\lt y_n\}$. For a Borel set $B\subseteq D$, $$P\left((X_{(1)},\dots,X_{(n)})\in B\right)=\sum_\pi P\left((X_{\pi(1)},\dots,X_{\pi(n)})\in B\right)=n!\int_B\prod_if(y_i)\,dy,$$ because each $(X_{\pi(1)},\dots,X_{\pi(n)})$ has the same density $\prod f(y_i)$ as $(X_1,\dots,X_n)$. (2) Integrate the density in (1) over all coordinates except $y_r=x$ and $y_s=y$. The coordinates split into three blocks. For the $r-1$ ordered coordinates below $x$, $$\int_{y_1\lt\dots\lt y_{r-1}\lt x}\prod_{i\lt r}f(y_i)\,dy=\frac{F(x)^{r-1}}{(r-1)!}.$$ This holds because the unrestricted integral over $(-\infty,x)^{r-1}$ equals $F(x)^{r-1}$, and by symmetry it is shared equally among the $(r-1)!$ orderings. Likewise the $s-r-1$ coordinates in $(x,y)$ contribute $\frac{[F(y)-F(x)]^{s-r-1}}{(s-r-1)!}$, and the $n-s$ coordinates above $y$ contribute $\frac{[1-F(y)]^{n-s}}{(n-s)!}$. Multiply by $n!f(x)f(y)$. ∎
Let $U_1,\dots,U_n$ be iid $U(0,1)$.
- $U_{(r)}\sim\operatorname{Beta}(r,\,n-r+1)$, so $E[U_{(r)}]=\dfrac{r}{n+1}$ and $\operatorname{Var}(U_{(r)})=\dfrac{r(n-r+1)}{(n+1)^2(n+2)}$.
- For $r\lt s$, $U_{(s)}-U_{(r)}\sim\operatorname{Beta}(s-r,\,n-s+r+1)$ and $\operatorname{Cov}(U_{(r)},U_{(s)})=\dfrac{r(n-s+1)}{(n+1)^2(n+2)}$.
- The range $R=U_{(n)}-U_{(1)}\sim\operatorname{Beta}(n-1,2)$, with density $n(n-1)w^{n-2}(1-w)$ on $(0,1)$, cdf $nw^{n-1}-(n-1)w^n$ and mean $\frac{n-1}{n+1}$.
- (Distribution-free) If $X_1,\dots,X_n$ are iid with a continuous cdf $F$, then $F(X_{(r)})\sim\operatorname{Beta}(r,n-r+1)$. For any $F$, $(X_{(1)},\dots,X_{(n)})$ has the same law as $(F^{-1}(U_{(1)}),\dots,F^{-1}(U_{(n)}))$, where $F^{-1}$ is the quantile function.
Proof. (1) With $F(x)=x$ on $(0,1)$, Theorem 4.7(2) gives the density $\frac{n!}{(r-1)!(n-r)!}x^{r-1}(1-x)^{n-r}$, which is the $\operatorname{Beta}(r,n-r+1)$ density since $B(r,n-r+1)=\frac{(r-1)!(n-r)!}{n!}$. The moments follow from the beta moments of §2.1 with $a=r$, $b=n-r+1$, $a+b=n+1$.
(2) By Theorem 4.8(2), $(U_{(r)},U_{(s)})$ has density $c\,x^{r-1}(y-x)^{s-r-1}(1-y)^{n-s}$ on $0\lt x\lt y\lt1$. Change variables to $(x,w)$ with $w=y-x$; the Jacobian is $1$. Integrating out $x\in(0,1-w)$ and substituting $x=(1-w)t$, $$\int_0^{1-w}x^{r-1}(1-w-x)^{n-s}dx=(1-w)^{n-s+r}B(r,n-s+1),$$ so $W$ has a density proportional to $w^{s-r-1}(1-w)^{n-s+r}$, which is $\operatorname{Beta}(s-r,n-s+r+1)$. For the covariance, let $D=(n+1)^2(n+2)$. Using $2\operatorname{Cov}=\operatorname{Var}U_{(r)}+\operatorname{Var}U_{(s)}-\operatorname{Var}W$ and the beta variances, $$2D\operatorname{Cov}=r(n-r+1)+s(n-s+1)-(s-r)(n-s+r+1)=2r(n+1-s),$$ after expanding $(s-r)(n-s+r+1)=(s-r)(n+1)-(s-r)^2$.
(3) is (2) with $r=1$, $s=n$; integrate the density for the cdf. (4) For continuous $F$, the $F(X_i)$ are iid $U(0,1)$ (the probability integral transform, §1.3). Since $F$ is nondecreasing, sorting commutes with applying $F$, so $F(X_{(r)})$ is the $r$-th order statistic of a uniform sample. Similarly the $F^{-1}(U_i)$ are iid with cdf $F$ (the quantile transform, §1.3), and $F^{-1}$ is nondecreasing. ∎
Part (4) is the source of distribution-free results. For instance, $F(X_{(n)})-F(X_{(1)})$, the population proportion covered by the sample range, is $\operatorname{Beta}(n-1,2)$ for every continuous $F$. The nonparametric tests of §5.3 rely on this kind of fact.
If $X_1,\dots,X_n$ ($n\ge2$) are iid with density $f$ and cdf $F$, then the range $R=X_{(n)}-X_{(1)}$ has, for $w\gt0$, $$f_R(w)=n(n-1)\int_{-\infty}^{\infty}f(x)f(x+w)\left[F(x+w)-F(x)\right]^{n-2}dx,\qquad P(R\le w)=n\int_{-\infty}^\infty f(x)\left[F(x+w)-F(x)\right]^{n-1}dx.$$
Proof. Apply the map $(x,y)\mapsto(x,w)=(x,y-x)$, which has Jacobian $1$, to the joint density of $(X_{(1)},X_{(n)})$ in Theorem 4.8(2), and integrate out $x$. For the cdf, argue directly. Ties have probability $0$, so $\{R\le w\}$ is the disjoint union over $i$ of the events "$X_i$ is the minimum and every other $X_j$ lies in $[X_i,X_i+w]$". By symmetry each has the same probability, and conditioning on $X_i=x$ gives $[F(x+w)-F(x)]^{n-1}$. ∎
Let $X_1,\dots,X_n$ be iid $\operatorname{Exp}(\lambda)$, and put $X_{(0)}=0$. The normalised spacings $$D_i=(n-i+1)\left(X_{(i)}-X_{(i-1)}\right),\qquad i=1,\dots,n,$$ are iid $\operatorname{Exp}(\lambda)$. Consequently:
- $X_{(r)}=\sum_{i=1}^r\dfrac{D_i}{n-i+1}$, so $E[X_{(r)}]=\dfrac1\lambda\sum_{i=1}^r\dfrac{1}{n-i+1}$ and $\operatorname{Var}(X_{(r)})=\dfrac{1}{\lambda^2}\sum_{i=1}^r\dfrac{1}{(n-i+1)^2}$;
- for $r\lt s$, $X_{(s)}-X_{(r)}$ is independent of $(X_{(1)},\dots,X_{(r)})$ and has the law of the $(s-r)$-th order statistic of $n-r$ iid $\operatorname{Exp}(\lambda)$ variables; in particular the range $X_{(n)}-X_{(1)}$ has the law of the maximum of $n-1$ iid $\operatorname{Exp}(\lambda)$ variables;
- the total time on test up to the $r$-th failure, $\sum_{i=1}^rX_{(i)}+(n-r)X_{(r)}=\sum_{i=1}^rD_i$, has the $\operatorname{Gamma}(r,\lambda)$ law, so $2\lambda\left[\sum_{i\le r}X_{(i)}+(n-r)X_{(r)}\right]\sim\chi^2_{2r}$.
Proof. By Theorem 4.8(1) the order statistics have density $n!\,\lambda^ne^{-\lambda\sum y_i}$ on $0\lt y_1\lt\dots\lt y_n$. The map $y\mapsto d$, $d_i=(n-i+1)(y_i-y_{i-1})$, is linear and lower triangular with diagonal $n,n-1,\dots,1$, so its determinant is $n!$. It maps the ordered region bijectively onto $(0,\infty)^n$, with inverse $y_r=\sum_{i\le r}d_i/(n-i+1)$. Summation by parts gives $$\sum_{i=1}^r(n-i+1)(y_i-y_{i-1})=\sum_{i=1}^ry_i+(n-r)y_r,$$ and for $r=n$ this says $\sum_id_i=\sum_iy_i$. Hence $D$ has density $\frac{1}{n!}\cdot n!\,\lambda^ne^{-\lambda\sum d_i}=\prod_i\lambda e^{-\lambda d_i}$, so the $D_i$ are iid $\operatorname{Exp}(\lambda)$. (1) follows from the inverse map. (2) $X_{(s)}-X_{(r)}=\sum_{i=r+1}^sD_i/(n-i+1)$ involves only $D_{r+1},\dots,D_s$, which are independent of $D_1,\dots,D_r$. Re-indexing with $j=i-r$ gives the representation for a sample of size $n-r$. (3) is the summation-by-parts identity together with the reproductive property of the gamma law. ∎
The representation has a memorable interpretation. Until the first failure, $n$ components are running, so the first failure time is the minimum of $n$ exponentials, which is $\operatorname{Exp}(n\lambda)$. By the memoryless property (§2.1), the $n-1$ survivors are then "as good as new", and the wait for the next failure is $\operatorname{Exp}((n-1)\lambda)$, independently of the past, and so on.
Sample Quantiles and the Efficiency of the Median
For large $n$, a sample quantile is approximately normal. Its variance depends on the population density at the quantile: the more probability there is near $\xi_p$, the more precisely $\xi_p$ is located.
Let $0\lt p\lt1$, and let $F$ be differentiable at its $p$-quantile $\xi_p$ with $f(\xi_p)=F'(\xi_p)\gt0$. If $r_n/n=p+o(n^{-1/2})$ (for example $r_n=\lceil np\rceil$), then $$\sqrt n\left(X_{(r_n)}-\xi_p\right)\xrightarrow{d}N\left(0,\ \frac{p(1-p)}{f(\xi_p)^2}\right).$$ In particular the sample median $\tilde X_n$ of a population with median $m$ and $f(m)\gt0$ satisfies $\sqrt n(\tilde X_n-m)\xrightarrow{d}N\left(0,\frac{1}{4f(m)^2}\right)$.
Proof (sketch). Fix $t$ and put $x_n=\xi_p+t/\sqrt n$ and $p_n=F(x_n)$. By differentiability, $p_n=p+\frac{t f(\xi_p)}{\sqrt n}+o(n^{-1/2})$. By the counting identity, $$P\left(\sqrt n(X_{(r_n)}-\xi_p)\le t\right)=P\left(N_n\ge r_n\right),\qquad N_n\sim\operatorname{Bin}(n,p_n).$$ Standardise: $\frac{r_n-np_n}{\sqrt{np_n(1-p_n)}}\to\frac{-tf(\xi_p)}{\sqrt{p(1-p)}}$. The CLT for binomial counts holds uniformly in the success probability on compact subsets of $(0,1)$ (by the Berry–Esseen bound of §2.3), so $$P(N_n\ge r_n)\to1-\Phi\left(\frac{-tf(\xi_p)}{\sqrt{p(1-p)}}\right)=\Phi\left(\frac{tf(\xi_p)}{\sqrt{p(1-p)}}\right),$$ which is the cdf of $N\left(0,p(1-p)/f(\xi_p)^2\right)$ at $t$. The even-$n$ median is an average of two such order statistics whose difference is $o_p(n^{-1/2})$. ∎
Comparing two estimators that are both asymptotically normal and centred at the same parameter is done by the ratio of their asymptotic variances, the asymptotic relative efficiency (ARE). If $\operatorname{ARE}(T_1,T_2)=v_2/v_1$, then $T_2$ needs about $1/\operatorname{ARE}$ times as many observations as $T_1$ to match its precision. Section 4.2 develops efficiency systematically.
- $\bar X$ and $S^2$ are independent only for normal samples (Geary). For $\operatorname{Exp}(1)$ samples $\operatorname{Cov}(\bar X,S^2)=2/n\ne0$. For symmetric non-normal laws they are uncorrelated but still dependent (Example 3).
- $(n-1)S^2/\sigma^2\sim\chi^2_{n-1}$ is not robust: $\operatorname{Var}(S^2)\approx(\kappa-1)\sigma^4/n$, not $2\sigma^4/n$, and this error does not go away as $n\to\infty$. The $t$ statistic, by contrast, is asymptotically $N(0,1)$ for every finite-variance population.
- Degrees of freedom: $\sum(X_i-\bar X)^2/\sigma^2\sim\chi^2_{n-1}$, but $\sum(X_i-\mu)^2/\sigma^2\sim\chi^2_n$. Mixing these up is the most common Part B error.
- $t$ and $F$ need independence of numerator and denominator. $(X_1-\bar X)/S$ is not $t$-distributed: it is bounded by $(n-1)/\sqrt n$ in absolute value, because $(n-1)S^2\ge\frac{n}{n-1}(X_1-\bar X)^2$.
- Moments of $t$ and $F$ may not exist: $t_1$ (Cauchy) has no mean, $t_2$ has infinite variance, and $E[F_{m,n}]=\frac{n}{n-2}$ depends on the denominator degrees of freedom and needs $n\gt2$.
- $F$ quantiles do not flip by themselves: $F_{m,n;1-\alpha}=1/F_{n,m;\alpha}$, with the degrees of freedom exchanged, and in general $F_{m,n;\alpha}\ne1/F_{m,n;1-\alpha}$ unless $m=n$.
- Order statistics are neither independent nor identically distributed, even though the $X_i$ are iid: $\operatorname{Cov}(U_{(r)},U_{(s)})\gt0$. The spacings of an exponential sample, once normalised, are independent (Rényi); uniform spacings are exchangeable but not independent (their sum is fixed).
- $E[X_{(r)}]\ne F^{-1}\left(\frac{r}{n+1}\right)$ in general: it is true for the uniform law, but for $\operatorname{Exp}(1)$ with $n=2$, $E[X_{(2)}]=\frac32$ while $F^{-1}(\frac23)=\log3=1.10$.
- The median is not always worse than the mean (Laplace, Cauchy), and the mean is not always better than everything else: the midrange beats it for uniform samples. Efficiency depends on the population.
- Asymptotic quantile results need $f(\xi_p)\gt0$: if the density vanishes at the median, the sample median converges more slowly than $n^{-1/2}$.
- The sample median of a discrete law need not be close to normal or even consistent for a unique value: for $\operatorname{Ber}(\frac12)$ the population median is not unique, and the sample median keeps jumping between $0$ and $1$.
Exercises
- 1 A population has mean $50$ and SD $12$. For $n=36$, find $\operatorname{SE}(\bar X)$ and approximate $P(\bar X\gt53)$.
- 2 Show that $E[S^2]=\sigma^2$ without assuming normality, and find the bias of $\hat\sigma^2=\frac1n\sum(X_i-\bar X)^2$.
- 3 For a sample of size $n=4$ from $\operatorname{Ber}(\frac12)$, compute $\operatorname{Var}(S^2)$ from Theorem 4.1(4), and check your answer by listing the $16$ equally likely samples.
- 4 Prove $\operatorname{Cov}(\bar X,S^2)=\mu_3/n$ and evaluate it for samples from $\operatorname{Exp}(1)$.
- 5 Two independent samples of sizes $50$ and $40$ come from populations with SDs $6$ and $8$. Find the standard error of $\bar X_1-\bar X_2$.
- 6 A pollster expects $p\approx0.3$. How large a sample makes $\operatorname{SE}(\hat p)\le0.01$?
- 7 Show that $\operatorname{SE}(S^2)\approx\sigma^2\sqrt{(\kappa-1)/n}$ for large $n$, where $\kappa=\mu_4/\sigma^4$. Evaluate it for $\sigma^2=1$, $n=100$ and normal, uniform and exponential populations.
- 8 Applied. A laboratory records $n=200$ repeated readings of a reference sample, with $s=2.5$ units. Assuming normal errors, give an approximate $95\%$ interval for the true measurement SD $\sigma$ using Theorem 4.6(3).
- 1 $X_1,\dots,X_4$ are iid $N(0,1)$. Name the laws of (a) $X_1^2+X_2^2$; (b) $X_1/|X_2|$; (c) $\dfrac{X_1^2+X_2^2}{X_3^2+X_4^2}$; (d) $\dfrac{\sqrt3\,X_1}{\sqrt{X_2^2+X_3^2+X_4^2}}$.
- 2 For a normal sample of size $15$, find $P(S^2\le1.5\sigma^2)$.
- 3 Use the mgf to find the mean and variance of $\chi^2_k$, and show that $(\chi^2_k-k)/\sqrt{2k}\xrightarrow{d}N(0,1)$.
- 4 Derive $\operatorname{Var}(t_k)=\frac{k}{k-2}$ from $E[1/V]$ for $V\sim\chi^2_k$, and explain why $t_2$ has infinite variance.
- 5 Prove $F_{m,n;1-\alpha}=1/F_{n,m;\alpha}$. Given $F_{10,5;0.05}=4.735$, find the lower $5\%$ point of $F_{5,10}$.
- 6 Show that $F_{2,2}$ has density $(1+x)^{-2}$ on $(0,\infty)$, and find $P(F_{2,2}\gt3)$.
- 7 Prove the two-sample result of Theorem 4.5(2), stating exactly where independence is used.
- 8 Let $X_1,X_2$ be iid. Show that $X_1+X_2$ and $X_1-X_2$ are always uncorrelated (when variances exist), that they are independent if the law is normal, and that they are dependent if the law is $U(0,1)$.
- 1 For a $U(0,1)$ sample of size $7$, find the mean and variance of the sample median.
- 2 For iid $\operatorname{Exp}(\lambda)$, find the densities of $X_{(1)}$ and $X_{(n)}$, and show that $X_{(1)}\sim\operatorname{Exp}(n\lambda)$.
- 3 A sample of size $3$ is drawn from the density $2x$ on $(0,1)$. Find the density and the mean of the sample median.
- 4 Write down the joint density of $(U_{(1)},U_{(n)})$ for a uniform sample and find $\operatorname{Cov}(U_{(1)},U_{(n)})$ by direct integration.
- 5 Show that the range of $n$ iid $\operatorname{Exp}(1)$ variables has density $(n-1)e^{-w}(1-e^{-w})^{n-2}$ and mean $1+\frac12+\dots+\frac{1}{n-1}$. Evaluate the mean for $n=4$.
- 6 For a continuous population, find (a) the probability that the maximum of $10$ observations exceeds the population $90$th percentile; (b) $P(X_{(1)}\lt m\lt X_{(6)})$ for a sample of size $6$, where $m$ is the population median.
- 7 Applied. A 2-out-of-3 system (it works while at least two of its three components work) uses components with iid exponential lifetimes of mean $6$ years. Find the mean system life and its reliability $P(\text{life}\gt t)$.
- 8 Find the asymptotic law of the sample median of an $\operatorname{Exp}(1)$ sample.
- 9 Show that $C=F(X_{(n)})-F(X_{(1)})$ is $\operatorname{Beta}(n-1,2)$ for every continuous $F$, and find the least $n$ with $E[C]\ge0.95$.
- 1 For a Poisson sample, find the limit law of $\sqrt n(S^2-\lambda)$ and compare it with that of $\sqrt n(\bar X-\lambda)$. (The central fourth moment of $\operatorname{Poi}(\lambda)$ is $\lambda+3\lambda^2$.)
- 2 For the logistic density $f(x)=\frac{e^{-x}}{(1+e^{-x})^2}$ (variance $\pi^2/3$), find the ARE of the sample median relative to $\bar X$ as an estimator of the centre.
- 3 Prove the Rényi representation directly for $n=2$: show that $X_{(1)}$ and $X_{(2)}-X_{(1)}$ are independent, with laws $\operatorname{Exp}(2\lambda)$ and $\operatorname{Exp}(\lambda)$.
- 4 For a $U(0,\theta)$ sample, show that $n(\theta-X_{(n)})/\theta\xrightarrow{d}\operatorname{Exp}(1)$. What does this say about the rate at which $X_{(n)}$ approaches $\theta$?
- 5 Show that for every continuous population the probability that the next observation $X_{n+1}$ exceeds the current maximum $X_{(n)}$ is $\frac{1}{n+1}$, and that $E[F(X_{(r)})]=\frac{r}{n+1}$.
- 6 Prove that $T=\sqrt n(\bar X-\mu)/S\xrightarrow{d}N(0,1)$ for every population with $0\lt\sigma^2\lt\infty$, and deduce that $t_{n-1,\alpha}\to z_\alpha$.
E1. CSIR Part B GATE ST Let $X_1,X_2,X_3,X_4$ be iid $N(0,1)$. The distribution of $\dfrac{X_1+X_2}{\sqrt{X_3^2+X_4^2}}$ is (A) $t_2$; (B) $t_4$; (C) $F_{1,2}$; (D) standard Cauchy.
E2. CSIR Part B Let $U_1,\dots,U_4$ be iid $U(0,1)$. Then $P\left(\max U_i-\min U_i\le\frac12\right)$ equals (A) $\frac1{16}$; (B) $\frac{3}{16}$; (C) $\frac{5}{16}$; (D) $\frac{11}{16}$.
E3. CSIR Part B GATE ST Four components have iid exponential lifetimes with rate $\lambda=2$ per year. The expected time until the second failure is (A) $\frac18$ year; (B) $\frac{7}{24}$ year; (C) $\frac{7}{12}$ year; (D) $\frac14$ year.
E4. CSIR Part C Let $X_1,\dots,X_n$ ($n\ge3$) be iid $N(\mu,\sigma^2)$ with sample mean $\bar X$ and sample variance $S^2$. Which of the following are true? (One or more options may be correct.) (A) $\bar X$ and $\sum(X_i-\bar X)^2$ are independent. (B) $\bar X$ and the sample range $X_{(n)}-X_{(1)}$ are independent. (C) $(X_1-\bar X)/S$ has the $t_{n-1}$ law. (D) $n(\bar X-\mu)^2/S^2$ has the $F_{1,n-1}$ law.
E5. CSIR Part C Let $T\sim t_k$ and $F\sim F_{m,n}$. Which of the following are true? (A) $T^2\sim F_{1,k}$. (B) $1/F\sim F_{n,m}$. (C) $E[F]=\frac{n}{n-2}$ for every $m$ when $n\gt2$. (D) $\operatorname{Var}(T)=\frac{k}{k-2}$ for every $k\ge2$.
E6. CSIR Part C GATE ST Let $X_1,\dots,X_n$ ($n\ge2$) be iid $U(0,\theta)$. Which of the following are true? (A) $X_{(n)}/\theta\sim\operatorname{Beta}(n,1)$. (B) $X_{(1)}/X_{(n)}$ is independent of $X_{(n)}$. (C) $\sqrt n(\theta-X_{(n)})$ converges in distribution to a non-degenerate normal law. (D) $\operatorname{Var}(X_{(n)})=\frac{\theta^2}{n+2}$.
Suggested plan — 3 lectures × 60 min.
- Statistics and normal sampling theory. Definition 4.1 and Theorem 4.1 (prove (1)–(3) on the board and leave (4) as a computation). Example 1. Then the Helmert proof of Theorem 4.2 in full; draw the picture of the deviation vector in the hyperplane $\sum x_i=0$. State Definition 4.2 and derive the $t$ density (Theorem 4.3(1)). Finish with Example 3 as a class discussion.
- $t$, $F$ and asymptotics. Theorem 4.3(2) via the beta law and Theorems 4.4–4.5, then Example 2 with a quantile table or scipy. Project Interactive 1: first the normal population with both statistics, then the exponential population with the variance statistic at $n=30$ (the tail rate stays far above $5\%$), then the $t$ statistic. Theorem 4.6 and the standard-error table; Example 4.
- Order statistics. The counting identity and Theorem 4.7 with the "boxes" heuristic, Theorem 4.8, the uniform beta laws (Theorem 4.9) and the range (Theorem 4.10). Rényi's representation via memorylessness before the Jacobian proof. Examples 5 and 6. Project Interactive 2 (range of the normal, minimum of the exponential), then Theorem 4.12 and Example 7. End with E4 and E6.
Discussion questions. (1) Why does the sample variance use the divisor $n-1$? Give two answers, one from unbiasedness and one from degrees of freedom. (2) For which populations would you estimate the centre by the mean, the median or the midrange, and why? (3) Order statistics of an iid sample are dependent. Which pairs are most strongly correlated, and why are normalised exponential spacings nevertheless independent?
Misconceptions to watch for. Believing that $\bar X\perp S^2$ for all populations; using $n$ instead of $n-1$ degrees of freedom; forming a "$t$ statistic" from dependent pieces; flipping $F$ quantiles without exchanging the degrees of freedom; treating order statistics as independent; assuming every estimator has error of order $1/\sqrt n$ (the maximum of a uniform sample has error of order $1/n$).
Suggested homework. I(2, 3, 7, 8), II(1, 4, 5, 8), III(1, 3, 5, 7, 9), IV(2, 4, 5); Exam corner E1, E4, E6.
Research question. How many computer simulations does a safety analyst need before the largest simulated value can be trusted as a "95/95" upper bound, meaning one that exceeds the 95th percentile of the true output distribution with 95% confidence, whatever that distribution is?
Why it matters. Nuclear regulators in several countries accept "best estimate plus uncertainty" safety analyses. A reactor code is run many times with randomly perturbed inputs, and the peak cladding temperature is reported as an upper tolerance limit computed from the order statistics of the runs. Because the output distribution is unknown, the guarantee must be distribution-free. That is exactly what Theorem 4.9(4) provides. The same reasoning is used for tolerance limits in pharmaceutical manufacturing and in aircraft-component testing.
What to do.
- Show that $P\left(X_{(n)}\gt\xi_{0.95}\right)=1-0.95^n$ for every continuous $F$, and find the least $n$ for which this is at least $0.95$.
- For a two-sided limit, the proportion covered is $C=F(X_{(n)})-F(X_{(1)})\sim\operatorname{Beta}(n-1,2)$. Find the least $n$ with $P(C\ge0.95)\ge0.95$, using scipy's
beta.sf. - If the analyst reports the second largest value instead, so as to be robust to one wild run, how many runs are needed for the one-sided 95/95 guarantee? (Use Theorem 4.7(1).)
- Check your answers by simulation with three very different populations (normal, exponential, Cauchy), using a seeded generator.
Tools: Python with numpy and scipy, or a spreadsheet with a beta-distribution function.
Hand in: A one-page note giving the three sample sizes with their derivations, a table of simulated coverage frequencies for the three populations, and one paragraph on why the answers do not depend on the population.
Going further. Read about Wilks' 1941 paper on tolerance limits and its modern use in reactor safety codes. Then investigate how the required number of runs grows if the guarantee must hold simultaneously for two correlated outputs.
🧪 Self-check quiz
Q1. For a sample of size $n$ from $N(\mu,\sigma^2)$, the law of $\sum_{i=1}^n(X_i-\bar X)^2/\sigma^2$ is
Q2. If $T\sim t_k$, then $T^2$ has the law
Q3. For an iid $U(0,1)$ sample of size $n$, the $r$-th order statistic $U_{(r)}$ has the law
Q4. A population has SD $5$. The standard error of $\bar X$ for $n=100$ is
Q5. For large normal samples, the variance of the sample median as an estimator of $\mu$ is approximately