Chapter 2 · Distributions and Limit Theorems

2.3 Modes of Convergence, Laws of Large Numbers and the CLT

What it means for random variables to converge — almost surely, in probability, in mean and in distribution — and the two great limit theorems built on these modes: the laws of large numbers, which say that averages settle down to the mean, and the central limit theorem, which says how the remaining error is distributed; with Slutsky's theorem and the delta method that make them usable in statistics.

Statistics is built on two experimental facts. Averages of many independent measurements settle down to a fixed number, and the scatter of those averages around that number has a bell-shaped histogram, whatever the distribution of a single measurement. The laws of large numbers and the central limit theorem turn these facts into theorems. They justify estimating a probability by a relative frequency, an integral by a Monte Carlo average and a population mean by a sample mean, and they produce the normal-based confidence intervals and tests of Chapters 4 and 5.

To state these theorems precisely we must first decide what "a sequence of random variables converges" means, and there are four standard answers: almost surely, in probability, in $r$-th mean and in distribution. They are genuinely different, and the implications between them, together with the counterexamples showing that no other implications hold, are among the most frequently examined facts in CSIR Part C. The section then proves the weak laws of large numbers of Chebyshev and Khintchine, states Kolmogorov's strong law (with a complete proof under a fourth-moment assumption), proves the Lindeberg–Lévy central limit theorem with the characteristic functions of §2.2, and develops the tools that statisticians use every day: normal approximations with continuity correction, the continuous mapping theorem, Slutsky's theorem and the delta method.

Four Modes of Convergence

Throughout, $X,X_1,X_2,\dots$ are random variables. For the first three modes they must be defined on one probability space $(\Omega,\mathcal F,P)$, because the definitions compare $X_n(\omega)$ with $X(\omega)$; the fourth mode compares only distributions.

📘 Definition 2.11 — Modes of convergence
  1. $X_n\to X$ almost surely (a.s., with probability one), written $X_n\xrightarrow{a.s.}X$, if $P\left(\{\omega:X_n(\omega)\to X(\omega)\}\right)=1$.
  2. $X_n\to X$ in probability, written $X_n\xrightarrow{P}X$, if $P(|X_n-X|\gt\varepsilon)\to0$ for every $\varepsilon\gt0$.
  3. For $r\gt0$, $X_n\to X$ in $r$-th mean (in $L^r$), written $X_n\xrightarrow{L^r}X$, if $E|X_n|^r\lt\infty$, $E|X|^r\lt\infty$ and $E|X_n-X|^r\to0$. For $r=2$ this is convergence in mean square (in quadratic mean).
  4. $X_n\to X$ in distribution, written $X_n\xrightarrow{d}X$, if $F_{X_n}(x)\to F_X(x)$ at every continuity point $x$ of $F_X$ (as defined in §2.2).

The set $\{X_n\to X\}$ is an event: it equals $\bigcap_{k\ge1}\bigcup_{N\ge1}\bigcap_{n\ge N}\{|X_n-X|\le1/k\}$, a countable combination of events. In words, $X_n(\omega)\to X(\omega)$ means that for every $k$ the deviations eventually stay below $1/k$. Almost-sure convergence is a statement about whole sample paths, convergence in probability only about the law of $X_n-X$ at each fixed $n$. The next theorem makes the difference precise.

📐 Theorem 2.30 — Criteria for almost-sure convergence

For $\varepsilon\gt0$ let $A_n(\varepsilon)=\{|X_n-X|\gt\varepsilon\}$.

  1. $X_n\xrightarrow{a.s.}X$ if and only if, for every $\varepsilon\gt0$, $P\left(A_n(\varepsilon)\text{ i.o.}\right)=0$, if and only if, for every $\varepsilon\gt0$, $$P\Big(\sup_{m\ge n}|X_m-X|\gt\varepsilon\Big)\to0\qquad(n\to\infty).$$
  2. (Borel–Cantelli criterion) If $\sum_nP(|X_n-X|\gt\varepsilon)\lt\infty$ for every $\varepsilon\gt0$, then $X_n\xrightarrow{a.s.}X$.
  3. If the $X_n$ are independent and $c$ is a constant, then $X_n\xrightarrow{a.s.}c$ if and only if $\sum_nP(|X_n-c|\gt\varepsilon)\lt\infty$ for every $\varepsilon\gt0$.

Proof. (1) $X_n(\omega)\not\to X(\omega)$ iff there is a $k$ with $|X_n(\omega)-X(\omega)|\gt1/k$ for infinitely many $n$. So $\{X_n\not\to X\}=\bigcup_k\limsup_nA_n(1/k)$. This has probability $0$ iff each $\limsup_nA_n(1/k)$ does, and since $A_n(\varepsilon)$ shrinks as $\varepsilon$ grows, iff $P(\limsup_nA_n(\varepsilon))=0$ for every $\varepsilon\gt0$. Now $\limsup_nA_n(\varepsilon)=\bigcap_nB_n$ with $B_n=\bigcup_{m\ge n}A_m(\varepsilon)=\{\sup_{m\ge n}|X_m-X|\gt\varepsilon\}$, a decreasing sequence, so by continuity from above $P(\limsup A_n(\varepsilon))=\lim_nP(B_n)$.

(2) The first Borel–Cantelli lemma (Theorem 1.22 of §1.2) gives $P(A_n(\varepsilon)\text{ i.o.})=0$; apply (1).

(3) The events $\{|X_n-c|\gt\varepsilon\}$ are independent. If the series diverges for some $\varepsilon$, the second Borel–Cantelli lemma gives $P(|X_n-c|\gt\varepsilon\text{ i.o.})=1$, so $X_n\not\to c$ a.s. The converse is (2). ∎

📐 Theorem 2.31 — Implications between the modes
  1. $X_n\xrightarrow{a.s.}X\ \Rightarrow\ X_n\xrightarrow{P}X$.
  2. $X_n\xrightarrow{L^r}X\ \Rightarrow\ X_n\xrightarrow{P}X$, for every $r\gt0$.
  3. $X_n\xrightarrow{L^s}X\ \Rightarrow\ X_n\xrightarrow{L^r}X$ for $0\lt r\lt s$.
  4. $X_n\xrightarrow{P}X\ \Rightarrow\ X_n\xrightarrow{d}X$.
  5. If $c$ is a constant, $X_n\xrightarrow{d}c\ \Rightarrow\ X_n\xrightarrow{P}c$.
  6. If $X_n\xrightarrow{P}X$, then some subsequence satisfies $X_{n_k}\xrightarrow{a.s.}X$. Consequently $X_n\xrightarrow{P}X$ iff every subsequence has a further subsequence converging to $X$ almost surely.
  7. (Dominated convergence) If $X_n\xrightarrow{P}X$ and $|X_n|\le Y$ with $E[Y^r]\lt\infty$, then $X_n\xrightarrow{L^r}X$. (More generally, $X_n\xrightarrow{L^1}X$ iff $X_n\xrightarrow{P}X$ and the family $\{X_n\}$ is uniformly integrable; statement only.)

No other implication holds in general (Example 1).

Proof. (1) By Theorem 2.30(1), $$P(|X_n-X|\gt\varepsilon)\le P\Big(\sup_{m\ge n}|X_m-X|\gt\varepsilon\Big)\to0.$$

(2) Markov's inequality (§2.2): $P(|X_n-X|\gt\varepsilon)\le E|X_n-X|^r/\varepsilon^r\to0$.

(3) Lyapunov's inequality (§2.2): $\left(E|X_n-X|^r\right)^{1/r}\le\left(E|X_n-X|^s\right)^{1/s}\to0$.

(4) Let $F_n$, $F$ be the cdfs and $\varepsilon\gt0$. If $X_n\le x$, then either $X\le x+\varepsilon$ or $|X_n-X|\gt\varepsilon$; and if $X\le x-\varepsilon$, then either $X_n\le x$ or $|X_n-X|\gt\varepsilon$. Hence $$F(x-\varepsilon)-P(|X_n-X|\gt\varepsilon)\le F_n(x)\le F(x+\varepsilon)+P(|X_n-X|\gt\varepsilon).$$ Letting $n\to\infty$, $$F(x-\varepsilon)\le\liminf F_n(x)\le\limsup F_n(x)\le F(x+\varepsilon).$$ At a continuity point $x$ of $F$, let $\varepsilon\downarrow0$.

(5) The cdf of the constant $c$ is continuous everywhere except at $c$. So $F_n(c-\varepsilon)\to0$ and $F_n(c+\varepsilon/2)\to1$, and $$P(|X_n-c|\gt\varepsilon)\le F_n(c-\varepsilon)+1-F_n(c+\varepsilon/2)\to0.$$

(6) Choose $n_1\lt n_2\lt\dots$ with $P(|X_{n_k}-X|\gt2^{-k})\lt2^{-k}$. The series $\sum_k2^{-k}$ converges, so by the first Borel–Cantelli lemma, a.s. $|X_{n_k}-X|\le2^{-k}$ for all large $k$, and $X_{n_k}\to X$ a.s. For the "iff": every subsequence of a sequence converging in probability converges in probability, so it has an a.s. convergent sub-subsequence. Conversely, if $X_n\not\xrightarrow{P}X$, there are $\varepsilon,\delta\gt0$ and a subsequence with $P(|X_{n_k}-X|\gt\varepsilon)\gt\delta$ for all $k$; no further subsequence can converge a.s., by (1).

(7) By (6), a subsequence converges a.s., so $|X|\le Y$ a.s. Then $|X_n-X|^r\le(2Y)^r$. For $\varepsilon\gt0$, $$E|X_n-X|^r\le\varepsilon^r+E\left[(2Y)^r\mathbf{1}\{|X_n-X|\gt\varepsilon\}\right].$$ Since $P(|X_n-X|\gt\varepsilon)\to0$ and $(2Y)^r$ is integrable, the last expectation tends to $0$ (absolute continuity of the integral: $E[Z\mathbf{1}_{A_n}]\to0$ whenever $E|Z|\lt\infty$ and $P(A_n)\to0$). So $\limsup E|X_n-X|^r\le\varepsilon^r$ for every $\varepsilon\gt0$. ∎

implicationtrue?reason or counterexample
a.s. ⟹ in probabilityyesTheorem 2.31(1)
$L^r$ ⟹ in probabilityyesMarkov's inequality
$L^s$ ⟹ $L^r$ ($r\lt s$)yesLyapunov's inequality
in probability ⟹ in distributionyesTheorem 2.31(4)
in probability ⟹ a.s.notypewriter; independent $\operatorname{Ber}(1/n)$ (Example 1(b), (c))
a.s. ⟹ $L^1$no$n\mathbf{1}\{U\lt1/n\}$ (Example 1(a))
$L^1$ ⟹ a.s.notypewriter (Example 1(b))
in distribution ⟹ in probabilityno (yes to a constant)$X_n=-X$, $X\sim N(0,1)$ (Example 1(d))
$L^r$ ⟹ $L^s$ ($r\lt s$)no$n\mathbf{1}\{U\lt1/n\}\to0$ in $L^{1/2}$, not in $L^1$
✏️ Example 1 — The standard counterexamples
Let $U\sim U(0,1)$ be defined on $\Omega=(0,1)$ with $P$ = length, $U(\omega)=\omega$. Decide in which modes each sequence converges to $0$. (a) $X_n=n\,\mathbf{1}\{U\lt1/n\}$. (b) The typewriter sequence: write $n=2^k+j$ with $0\le j\lt2^k$, and let $X_n=\mathbf{1}\left\{\frac{j}{2^k}\le U\lt\frac{j+1}{2^k}\right\}$. (c) Independent $X_n\sim\operatorname{Ber}(1/n)$, and independent $Y_n\sim\operatorname{Ber}(1/n^2)$. (d) $X\sim N(0,1)$ and $X_n=-X$: does $X_n\to X$?
(a) For each $\omega\in(0,1)$, $X_n(\omega)=0$ as soon as $n\gt1/\omega$, so $X_n\to0$ everywhere, hence a.s. and in probability. But $E[X_n]=n\cdot\frac1n=1$ for all $n$, so $X_n\not\to0$ in $L^1$, and the means do not converge to the mean of the limit. For $0\lt r\lt1$, $E[X_n^r]=n^{r-1}\to0$, so $X_n\to0$ in $L^r$ for $r\lt1$ but not for $r\ge1$.
(b) The intervals sweep across $(0,1)$ once in each block $2^k\le n\lt2^{k+1}$, with length $2^{-k}$. So $E|X_n|^r=P(X_n=1)=2^{-k}\to0$: convergence in every $L^r$ and in probability. But every $\omega$ lies in one interval of every block, so $X_n(\omega)=1$ for infinitely many $n$ and $X_n(\omega)=0$ for infinitely many $n$: $X_n(\omega)$ converges for no $\omega$. The subsequence $X_{2^k}=\mathbf{1}\{U\lt2^{-k}\}$ does converge to $0$ a.s., as Theorem 2.31(6) promises.
(c) $P(X_n\ne0)=\frac1n\to0$ and $E[X_n^r]=\frac1n\to0$: convergence in probability and in every $L^r$. But $\sum\frac1n=\infty$, so by Theorem 2.30(3) $X_n\not\to0$ a.s.: almost every path has infinitely many $1$s. For $Y_n$, $\sum\frac{1}{n^2}\lt\infty$, so $Y_n\to0$ a.s.
(d) $X_n$ and $X$ have the same law $N(0,1)$, so $X_n\xrightarrow{d}X$ trivially. But $P(|X_n-X|\gt\varepsilon)=P(2|X|\gt\varepsilon)$, a positive constant, so $X_n\not\xrightarrow{P}X$. Convergence in distribution says nothing about the joint behaviour of $X_n$ and $X$.
📐 Theorem 2.32 — Characterisations of convergence in distribution
  1. (Helly–Bray) $X_n\xrightarrow{d}X$ if and only if $E[g(X_n)]\to E[g(X)]$ for every bounded continuous $g:\mathbb{R}\to\mathbb{R}$.
  2. (Lévy, §2.2) $X_n\xrightarrow{d}X$ if and only if $\varphi_{X_n}(t)\to\varphi_X(t)$ for every $t$.
  3. (Integer-valued laws) If $X_n$ and $X$ take values in $\mathbb{Z}$, then $X_n\xrightarrow{d}X$ iff $P(X_n=k)\to P(X=k)$ for every $k\in\mathbb{Z}$.
  4. (Scheffé) If $X_n$ and $X$ have densities $f_n\to f$ almost everywhere, then $\int|f_n-f|\to0$; hence $\sup_B|P(X_n\in B)-P(X\in B)|\to0$ over all Borel sets $B$, and in particular $X_n\xrightarrow{d}X$.

Proof. (1) "If": fix a continuity point $x$ of $F$ and $\delta\gt0$. Let $g$ be $1$ on $(-\infty,x]$, $0$ on $[x+\delta,\infty)$ and linear in between; it is bounded and continuous with $\mathbf{1}_{(-\infty,x]}\le g\le\mathbf{1}_{(-\infty,x+\delta]}$. So $$\limsup F_n(x)\le\lim E[g(X_n)]=E[g(X)]\le F(x+\delta).$$ Similarly, with a function squeezed between $\mathbf{1}_{(-\infty,x-\delta]}$ and $\mathbf{1}_{(-\infty,x]}$, $\liminf F_n(x)\ge F(x-\delta)$. Let $\delta\downarrow0$. "Only if" (sketch): let $U\sim U(0,1)$ and put $Y_n=F_n^{-1}(U)$, $Y=F^{-1}(U)$ with the quantile functions of §1.3. Then $Y_n\overset{d}{=}X_n$, $Y\overset{d}{=}X$ (the quantile transform), and one checks that $F_n^{-1}(u)\to F^{-1}(u)$ at every continuity point $u$ of $F^{-1}$, hence for almost every $u$. So $Y_n\to Y$ a.s. (Skorokhod's representation), and dominated convergence gives $E[g(X_n)]=E[g(Y_n)]\to E[g(Y)]=E[g(X)]$.

(3) "Only if": $P(X_n=k)=F_n(k+\frac12)-F_n(k-\frac12)$, and $k\pm\frac12$ are continuity points of $F$. "If": fix $x$ and $M\in\mathbb{N}$. Finite sums converge, so $$\liminf F_n(x)\ge\lim\sum_{-M\le k\le x}P(X_n=k)=P(-M\le X\le x);$$ letting $M\to\infty$, $\liminf F_n(x)\ge F(x)$. In the same way $\liminf P(X_n\gt x)\ge P(x\lt X\le M)\to P(X\gt x)$, i.e. $\limsup F_n(x)\le F(x)$.

(4) $(f-f_n)^+\le f$ and $(f-f_n)^+\to0$ a.e., so $\int(f-f_n)^+\to0$ by dominated convergence. Since $\int(f-f_n)=1-1=0$, $\int|f-f_n|=2\int(f-f_n)^+\to0$. Finally $$|P(X_n\in B)-P(X\in B)|=\left|\int_B(f_n-f)\right|\le\int|f_n-f|.$$ ∎

✏️ Example 2 — Limits of maxima and minima
(a) Let $M_n=\max(U_1,\dots,U_n)$ with $U_j$ iid $U(0,1)$. Show that $M_n\to1$ almost surely and that $n(1-M_n)\xrightarrow{d}\operatorname{Exp}(1)$. (b) Let $M_n$ be the maximum of $n$ iid $\operatorname{Exp}(1)$ variables. Show that $M_n-\log n\xrightarrow{d}G$ with $P(G\le x)=\exp(-e^{-x})$ (the Gumbel law). (c) Let $X_n\sim N(0,\tfrac1n)$. Show that $X_n\xrightarrow{d}0$ although $F_n(0)\not\to F(0)$.
(a) $P(|M_n-1|\gt\varepsilon)=P(M_n\lt1-\varepsilon)=(1-\varepsilon)^n$ for $0\lt\varepsilon\lt1$, which is summable, so $M_n\to1$ a.s. by Theorem 2.30(2). (Also directly: $M_n$ is increasing and bounded by $1$.) For $x\ge0$ and $n\gt x$, $P(n(1-M_n)\gt x)=P(M_n\lt1-\frac xn)=\left(1-\frac xn\right)^n\to e^{-x}$. So $n(1-M_n)\xrightarrow{d}\operatorname{Exp}(1)$: the gap below the endpoint is of order $1/n$, not $1/\sqrt n$.
(b) $P(M_n\le x+\log n)=\left(1-e^{-x-\log n}\right)^n=\left(1-\frac{e^{-x}}{n}\right)^n\to\exp(-e^{-x})$ for every real $x$ (for $n\gt e^{-x}$). The limit is a continuous cdf, so this is convergence in distribution. This is the prototype of extreme-value limits used for floods, wind speeds and insurance losses.
(c) $F_n(x)=\Phi(x\sqrt n)\to0$ for $x\lt0$ and $\to1$ for $x\gt0$, which are exactly the values of the cdf of the constant $0$ at its continuity points. At $x=0$, $F_n(0)=\frac12$ for every $n$ while $F(0)=1$, but $0$ is a jump of $F$ and is exempt. (By Theorem 2.31(5) we even have $X_n\xrightarrow{P}0$; indeed $E[X_n^2]=\frac1n\to0$.)

Laws of Large Numbers

Let $S_n=X_1+\dots+X_n$ and $\bar X_n=S_n/n$. A weak law asserts $\bar X_n-E[\bar X_n]\xrightarrow{P}0$; a strong law asserts almost-sure convergence. The weak law of Chebyshev needs second moments but allows the variables to be merely uncorrelated and not identically distributed; Khintchine's weak law and Kolmogorov's strong law need iid variables but only a finite mean.

📐 Theorem 2.33 — Chebyshev's weak law of large numbers
  1. (Markov's condition) If the $X_j$ have finite variances and $\operatorname{Var}(S_n)/n^2\to0$, then $\dfrac{S_n-E[S_n]}{n}\to0$ in $L^2$ and in probability.
  2. In particular, if the $X_j$ are uncorrelated with $\operatorname{Var}(X_j)\le C$ for all $j$, then $\bar X_n-\frac1n\sum_{j\le n}E[X_j]\xrightarrow{P}0$. If moreover $E[X_j]=\mu$ for all $j$, then $\bar X_n\xrightarrow{P}\mu$.
  3. (Bernoulli's law of large numbers) If $S_n\sim\operatorname{Bin}(n,p)$, then $S_n/n\xrightarrow{P}p$.

Proof. (1) $E\left[\left(\frac{S_n-ES_n}{n}\right)^2\right]=\frac{\operatorname{Var}(S_n)}{n^2}\to0$, which is $L^2$ convergence; Chebyshev's inequality (or Theorem 2.31(2)) gives convergence in probability. (2) For uncorrelated variables $\operatorname{Var}(S_n)=\sum_j\operatorname{Var}(X_j)\le Cn$, so $\operatorname{Var}(S_n)/n^2\le C/n\to0$. (3) is (2) for Bernoulli trials, with $C=p(1-p)$. ∎

📐 Theorem 2.34 — Khintchine's weak law of large numbers
If $X_1,X_2,\dots$ are iid with $E|X_1|\lt\infty$ and $\mu=E[X_1]$, then $\bar X_n\xrightarrow{P}\mu$. No assumption on the variance is needed.

Proof. Let $\varphi$ be the cf of $X_1$. By Theorem 2.19(2) of §2.2 with $n=1$, $\varphi(s)=1+i\mu s+o(|s|)$ as $s\to0$. By Theorem 2.18, $$\varphi_{\bar X_n}(t)=\varphi\!\left(\frac tn\right)^n=\left(1+\frac{i\mu t+\epsilon_n}{n}\right)^n,\qquad\epsilon_n=n\,o(|t|/n)\to0.$$ For complex $z_n\to z$, $(1+z_n/n)^n\to e^z$ (for large $n$, $|z_n/n|\lt\frac12$, and $n\operatorname{Log}(1+z_n/n)=z_n+O(|z_n|^2/n)\to z$). Hence $\varphi_{\bar X_n}(t)\to e^{i\mu t}$, the cf of the constant $\mu$, which is continuous. Lévy's continuity theorem gives $\bar X_n\xrightarrow{d}\mu$, and Theorem 2.31(5) upgrades this to convergence in probability. ∎

📐 Theorem 2.35 — Kolmogorov's strong law of large numbers

Let $X_1,X_2,\dots$ be iid.

  1. If $E|X_1|\lt\infty$, then $\bar X_n\xrightarrow{a.s.}\mu=E[X_1]$. (Statement; we prove it below under the extra assumption $E[X_1^4]\lt\infty$.)
  2. Conversely, if $E|X_1|=\infty$, then $\limsup_n|\bar X_n|=\infty$ a.s.; in particular $\bar X_n$ converges to a finite limit with probability $0$.

So for iid variables, $\bar X_n$ converges a.s. to a finite constant if and only if $E|X_1|\lt\infty$, and the constant is then $E[X_1]$.

Proof of (1) when $E[X_1^4]\lt\infty$. Replacing $X_j$ by $X_j-\mu$ we may assume $\mu=0$. Expand $E[S_n^4]=\sum_{i,j,k,l}E[X_iX_jX_kX_l]$. By independence and $E[X_j]=0$, a term vanishes whenever some index occurs exactly once. The surviving terms have all four indices equal ($n$ terms, each $E[X_1^4]$) or two distinct indices each occurring twice ($3n(n-1)$ ordered quadruples, each $\sigma^4$). Since $\sigma^4\le E[X_1^4]$ (Jensen), $$E[S_n^4]=nE[X_1^4]+3n(n-1)\sigma^4\le3n^2E[X_1^4].$$ Markov's inequality applied to $S_n^4$ gives $P(|\bar X_n|\gt\varepsilon)=P(S_n^4\gt n^4\varepsilon^4)\le\frac{3E[X_1^4]}{n^2\varepsilon^4}$, which is summable in $n$. Theorem 2.30(2) gives $\bar X_n\to0$ a.s. (The general case needs more work: truncation $X_j\mathbf{1}\{|X_j|\le j\}$, Kolmogorov's maximal inequality or Etemadi's subsequence argument.)

Proof of (2). For a non-negative $Y$, the tail formula (Theorem 1.27 of §1.3) gives $E[Y]=\int_0^\infty P(Y\gt t)\,dt\le1+\sum_{n\ge1}P(Y\ge n)$. Fix $c\gt0$ and apply this to $Y=|X_1|/c$: $\sum_nP(|X_n|\ge cn)=\sum_nP(|X_1|/c\ge n)=\infty$, since $E[Y]=\infty$. The events $\{|X_n|\ge cn\}$ are independent, so by the second Borel–Cantelli lemma $|X_n|\ge cn$ infinitely often, a.s. Since $\frac{X_n}{n}=\bar X_n-\frac{n-1}{n}\bar X_{n-1}$, on this event $|\bar X_n|+|\bar X_{n-1}|\ge c$ infinitely often, so $\limsup|\bar X_n|\ge c/2$. Intersect over $c\in\mathbb{N}$. ∎

Consequences used throughout statistics: the relative frequency of an event converges a.s. to its probability (apply the SLLN to indicators; for a fixed $x$ the empirical cdf $\hat F_n(x)\to F(x)$ a.s., and the Glivenko–Cantelli theorem, stated here without proof, gives $\sup_x|\hat F_n(x)-F(x)|\to0$ a.s.); sample moments converge a.s. to population moments; and Monte Carlo averages $\frac1n\sum g(U_j)$ converge a.s. to $\int_0^1g$ whenever $\int_0^1|g|\lt\infty$.

✏️ Example 3 — Which sequences obey a law of large numbers?
(a) $X_k$ are independent with $P(X_k=k^\alpha)=P(X_k=-k^\alpha)=\tfrac12$, where $\alpha\gt0$. For which $\alpha$ does Theorem 2.33 give $\bar X_n\xrightarrow{P}0$? (b) $X_k=Z+\varepsilon_k$, where $Z,\varepsilon_1,\varepsilon_2,\dots$ are iid $N(0,1)$. Does $\bar X_n\to0=E[X_k]$? (c) The $X_k$ have $E[X_k]=\mu$, $\operatorname{Var}(X_k)\le1$ and $|\operatorname{Cov}(X_j,X_k)|\le2^{-|j-k|}$. Show that $\bar X_n\xrightarrow{P}\mu$. (d) $X_k$ are iid standard Cauchy. Does $\bar X_n$ converge in any sense?
(a) $E[X_k]=0$ and $\operatorname{Var}(X_k)=k^{2\alpha}$, so $\frac{\operatorname{Var}(S_n)}{n^2}=\frac{1}{n^2}\sum_{k=1}^nk^{2\alpha}\sim\frac{n^{2\alpha-1}}{2\alpha+1}$, which tends to $0$ iff $\alpha\lt\frac12$. So the weak law holds for $\alpha\lt\frac12$. For $\alpha=\frac12$, $\operatorname{Var}(\bar X_n)\to\frac12$, and the Lyapunov central limit theorem (Theorem 2.37(3)) gives $\bar X_n\xrightarrow{d}N(0,\frac12)$, so the weak law fails: the variables grow too fast.
(b) No. $\bar X_n=Z+\bar\varepsilon_n\to Z$ a.s. by the SLLN applied to the $\varepsilon_k$. The limit is random, not $E[X_k]=0$. Here $\operatorname{Cov}(X_j,X_k)=1$ for all $j\ne k$ and $\operatorname{Var}(S_n)/n^2=\frac{n^2+n}{n^2}\to1$: the common component $Z$ never averages out.
(c) $\operatorname{Var}(S_n)=\sum_{j,k\le n}\operatorname{Cov}(X_j,X_k)\le\sum_{j\le n}\sum_{m\in\mathbb{Z}}2^{-|m|}=3n$, so $\operatorname{Var}(S_n)/n^2\le3/n\to0$ and Theorem 2.33(1) applies. Weak dependence that decays fast enough does not destroy the law of large numbers; this is why time averages of many stationary processes estimate their means.
(d) No. $\bar X_n\sim C(0,1)$ for every $n$ (§2.2), so $\bar X_n$ does not converge in probability to any constant, and by Theorem 2.35(2) it does not converge a.s. to a finite limit: $\limsup|\bar X_n|=\infty$ a.s. It does converge in distribution, trivially, since all the laws are equal.
✏️ Example 4 — The strong law at work
(a) Monte Carlo: estimate $\theta=\int_0^1e^{x^2}dx$ by $\hat\theta_n=\frac1n\sum_{j=1}^ne^{U_j^2}$ with $U_j$ iid $U(0,1)$. Show that $\hat\theta_n\to\theta$ a.s., and find how large $n$ must be for $P(|\hat\theta_n-\theta|\ge0.01)\le0.05$ according to Chebyshev's inequality and according to the CLT. (b) Find the almost-sure limit of the geometric mean $(U_1U_2\cdots U_n)^{1/n}$. (c) For iid $X_j$ with finite variance $\sigma^2$, show that the sample variance $S_n^2=\frac{1}{n-1}\sum_{j=1}^n(X_j-\bar X_n)^2$ converges to $\sigma^2$ almost surely.
(a) The $e^{U_j^2}$ are iid and bounded with mean $\theta\approx1.46265$, so the SLLN applies. Their variance is $v=\int_0^1e^{2x^2}dx-\theta^2\approx0.22510$. Chebyshev: $P(|\hat\theta_n-\theta|\ge0.01)\le\frac{v}{n(0.01)^2}\le0.05$ needs $n\ge\frac{v}{0.05\times10^{-4}}\approx45021$. The CLT (Theorem 2.36) gives $n\ge\left(\frac{1.96}{0.01}\right)^2v\approx8648$.
(b) $\log\left((U_1\cdots U_n)^{1/n}\right)=\frac1n\sum\log U_j\to E[\log U]=\int_0^1\log u\,du=-1$ a.s. (here $E|\log U|=1\lt\infty$). Since $\exp$ is continuous, the geometric mean tends to $e^{-1}\approx0.3679$ a.s., while the arithmetic mean tends to $\frac12$: AM–GM in the limit.
(c) $\frac1n\sum(X_j-\bar X_n)^2=\frac1n\sum X_j^2-\bar X_n^2$. By the SLLN, $\frac1n\sum X_j^2\to E[X_1^2]$ and $\bar X_n\to\mu$ a.s. (both need only $E[X_1^2]\lt\infty$). Hence $\frac1n\sum(X_j-\bar X_n)^2\to E[X_1^2]-\mu^2=\sigma^2$ a.s., and the factor $\frac{n}{n-1}\to1$.
Interactive 1 — Sample Paths: Almost Surely, In Probability, In Mean

Each of the $40$ rows or curves is one outcome $\omega$, followed for $n=1,\dots,512$ (a seeded generator). In the first two modes the curves are running means $\bar X_n(\omega)$ with an $\varepsilon$-band (amber) around the target; curves outside the band at the cursor are red. In the other modes each tick marks a time $n$ with $X_n(\omega)\ne0$. Convergence in probability asks only that few rows be "on" at the cursor; almost-sure convergence asks that each row eventually switch off for good. Compare the first two readouts as you move the cursor or press Play.

5120.15
P(|Xₙ − X| > ε) at the cursor: sample / exact—
rows with a deviation at some m ≥ n (up to 512)—
E|Xₙ − X|—
modes—

The Central Limit Theorem and Normal Approximations

The law of large numbers says $\bar X_n-\mu\to0$. The central limit theorem describes the size and shape of the error: it is of order $\sigma/\sqrt n$, and after dividing by that scale it is approximately standard normal, whatever the law of $X_1$, provided only that the variance is finite.

📐 Theorem 2.36 — Lindeberg–Lévy central limit theorem

Let $X_1,X_2,\dots$ be iid with mean $\mu$ and variance $\sigma^2$, where $0\lt\sigma^2\lt\infty$. Then $$Z_n=\frac{S_n-n\mu}{\sigma\sqrt n}=\frac{\sqrt n\,(\bar X_n-\mu)}{\sigma}\xrightarrow{d}N(0,1),$$ that is, $P(Z_n\le z)\to\Phi(z)$ for every real $z$. Because $\Phi$ is continuous, the convergence is even uniform in $z$ (Pólya's theorem).

Proof. Step 1 (reduce). Let $Y_j=(X_j-\mu)/\sigma$: iid with mean $0$, variance $1$ and cf $\psi$. Then $Z_n=\frac{1}{\sqrt n}\sum_{j\le n}Y_j$ and, by Theorems 2.17(4) and 2.18 of §2.2, $\varphi_{Z_n}(t)=\psi\left(t/\sqrt n\right)^n$.

Step 2 (expand). By Theorem 2.19(2) with two moments, $\psi(s)=1-\frac{s^2}{2}+o(s^2)$ as $s\to0$. So for fixed $t$, $$\psi\!\left(\frac{t}{\sqrt n}\right)=1-\frac{t^2}{2n}+r_n,\qquad n\,r_n\to0.$$

Step 3 (compare products). For complex numbers with $|a_j|,|b_j|\le1$, $$\Big|\prod_{j=1}^na_j-\prod_{j=1}^nb_j\Big|\le\sum_{j=1}^n|a_j-b_j|,$$ by telescoping: $\prod a-\prod b=\sum_ja_1\cdots a_{j-1}(a_j-b_j)b_{j+1}\cdots b_n$. Take $a_j=\psi(t/\sqrt n)$ and $b_j=1-\frac{t^2}{2n}$, which lies in $[-1,1]$ once $n\ge t^2/4$. Then $$\left|\psi\!\left(\tfrac{t}{\sqrt n}\right)^n-\left(1-\tfrac{t^2}{2n}\right)^n\right|\le n|r_n|\to0.$$

Step 4 (conclude). $\left(1-\frac{t^2}{2n}\right)^n\to e^{-t^2/2}$, so $\varphi_{Z_n}(t)\to e^{-t^2/2}$, the continuous cf of $N(0,1)$. Lévy's continuity theorem gives $Z_n\xrightarrow{d}N(0,1)$. For uniformity: given $\varepsilon\gt0$ choose $z_0\lt z_1\lt\dots\lt z_m$ with $\Phi(z_0)\lt\varepsilon$, $1-\Phi(z_m)\lt\varepsilon$ and $\Phi(z_i)-\Phi(z_{i-1})\lt\varepsilon$; convergence at these finitely many points and monotonicity of cdfs give $\sup_z|P(Z_n\le z)-\Phi(z)|\lt2\varepsilon$ for large $n$. ∎

The proof used exactly the second moment: the expansion $\psi(s)=1-\frac{s^2}2+o(s^2)$ needs $E[Y^2]\lt\infty$ but no third moment. If the variance is infinite the conclusion can fail completely: for Cauchy data $Z_n$ is not even defined, and $\bar X_n$ stays Cauchy.

📐 Theorem 2.37 — Normal approximations and companions of the CLT
  1. (De Moivre–Laplace) If $X\sim\operatorname{Bin}(n,p)$ with $0\lt p\lt1$ fixed, then $\frac{X-np}{\sqrt{np(1-p)}}\xrightarrow{d}N(0,1)$. For integers $a\le b$ the continuity-corrected approximation is $$P(a\le X\le b)\approx\Phi\left(\frac{b+\frac12-np}{\sqrt{npq}}\right)-\Phi\left(\frac{a-\frac12-np}{\sqrt{npq}}\right).$$
  2. (Poisson) If $X_\lambda\sim\operatorname{Poi}(\lambda)$, then $\frac{X_\lambda-\lambda}{\sqrt\lambda}\xrightarrow{d}N(0,1)$ as $\lambda\to\infty$, with the same continuity correction.
  3. (Lyapunov and Lindeberg, statement) Let $X_k$ be independent with means $\mu_k$, finite variances $\sigma_k^2$ and $s_n^2=\sum_{k\le n}\sigma_k^2$. If for some $\delta\gt0$ $$\frac{1}{s_n^{2+\delta}}\sum_{k=1}^nE|X_k-\mu_k|^{2+\delta}\to0\qquad\text{(Lyapunov's condition)},$$ or more generally if for every $\varepsilon\gt0$ $$\frac{1}{s_n^2}\sum_{k\le n}E\left[(X_k-\mu_k)^2\mathbf{1}\{|X_k-\mu_k|\gt\varepsilon s_n\}\right]\to0$$ (Lindeberg's condition), then $\frac{S_n-E[S_n]}{s_n}\xrightarrow{d}N(0,1)$.
  4. (Berry–Esseen, statement) In Theorem 2.36, if $\rho=E|X_1-\mu|^3\lt\infty$, then $$\sup_z\left|P(Z_n\le z)-\Phi(z)\right|\le\frac{C\rho}{\sigma^3\sqrt n},$$ with an absolute constant $C$; $C=0.4748$ is known to suffice.

Proof. (1) $X$ has the law of a sum of $n$ iid $\operatorname{Ber}(p)$ variables with mean $p$ and variance $pq\gt0$; apply Theorem 2.36. (2) was proved with cfs in Example 3(b) of §2.2; alternatively, for integer $\lambda=m$, $X_m$ has the law of a sum of $m$ iid $\operatorname{Poi}(1)$ variables. (3) and (4) are stated without proof. ∎

Why the continuity correction? An integer-valued $X$ puts mass $P(X=k)$ at the single point $k$, while the normal curve spreads it over an interval. Representing $P(X=k)$ by the normal area over $[k-\frac12,k+\frac12]$ matches the two, so $P(X\le b)$ is approximated by the area up to $b+\frac12$ and $P(X\ge a)$ by the area from $a-\frac12$. The correction matters most when $\sqrt{npq}$ is small, and for a single value $P(X=k)$ it is indispensable (without it the answer would be $0$). By Berry–Esseen, the error of the uncorrected approximation for the binomial is at most $0.4748\,\frac{p^2+q^2}{\sqrt{npq}}$; a common rule of thumb is to use the normal approximation when $np\ge5$ and $n(1-p)\ge5$.

✏️ Example 5 — Normal approximations with continuity correction
(a) A fair die is rolled $600$ times. Approximate the probability that the number of sixes is between $90$ and $110$ inclusive. (b) Calls reach a helpline as a Poisson stream with mean $50$ per hour. Approximate the probability of at least $60$ calls in an hour. (c) A shop rounds each of $300$ bills to the nearest rupee; the rounding errors are iid $U(-\tfrac12,\tfrac12)$ rupees. Approximate the probability that the total rounding error exceeds $10$ rupees in absolute value. (d) How many voters must be polled so that the approximate $95\%$ margin of error of a proportion is at most $0.03$ whatever $p$ is?
(a) $X\sim\operatorname{Bin}(600,\tfrac16)$: mean $100$, variance $\frac{500}{6}$, SD $9.1287$. With continuity correction, $$P(90\le X\le110)\approx\Phi\left(\tfrac{10.5}{9.1287}\right)-\Phi\left(-\tfrac{10.5}{9.1287}\right)=2\Phi(1.1502)-1\approx0.7499.$$ The exact binomial value is $0.7501$. Without the correction, $2\Phi(1.0954)-1\approx0.7267$, a much worse answer.
(b) $X\sim\operatorname{Poi}(50)\approx N(50,50)$: $P(X\ge60)\approx1-\Phi\left(\frac{59.5-50}{\sqrt{50}}\right)=1-\Phi(1.3435)\approx0.0896$. The exact value is $0.0923$; the remaining error comes from the right skewness of the Poisson law.
(c) Each error has mean $0$ and variance $\frac1{12}$, so the total $T$ has mean $0$ and variance $300/12=25$. By the CLT, $P(|T|\gt10)\approx P(|Z|\gt2)=2(1-\Phi(2))\approx0.0455$. No continuity correction: $T$ is continuous.
(d) The margin is $1.96\sqrt{p(1-p)/n}\le\frac{1.96}{2\sqrt n}$, which is at most $0.03$ when $n\ge\left(\frac{1.96}{0.06}\right)^2\approx1067.1$, i.e. $n\ge1068$. (Compare Chebyshev's guarantee in §2.2, which needs several times as many.)
Interactive 2 — The Central Limit Theorem in Action

For the chosen parent law, $4000$ samples of size $n$ are drawn (a seeded generator) and each is reduced to $Z_n=\sqrt n(\bar X_n-\mu)/\sigma$. The blue histogram (a density scale) is compared with the purple $N(0,1)$ density. Increase $n$ or press Play. Symmetric parents converge fast; skewed parents (exponential, Bernoulli with small $p$, lognormal) keep a visible tilt whose size is the skewness $\gamma_1/\sqrt n$, and the Berry–Esseen bound in the last readout guarantees the accuracy. For the Cauchy parent the histogram shows $\bar X_n$ itself against the Cauchy density: it never narrows.

2
sample mean, variance of Zₙ—
sup |F̂ − Φ| (Kolmogorov distance)—
skewness of Zₙ: γ₁/√n—
Berry–Esseen bound 0.4748ρ/(σ³√n)—

Slutsky's Theorem, Continuous Mapping and the Delta Method

In statistics the CLT is rarely used in its pure form. The standard deviation $\sigma$ is unknown and is replaced by an estimate, and one needs the limit law of a function of $\bar X_n$, such as $1/\bar X_n$ or $\log\bar X_n$, rather than of $\bar X_n$ itself. Three results handle this.

📐 Theorem 2.38 — The continuous mapping theorem
Let $g:\mathbb{R}\to\mathbb{R}$ be continuous at every point of a set $C$ with $P(X\in C)=1$. If $X_n\to X$ almost surely, in probability or in distribution, then $g(X_n)\to g(X)$ in the same mode. In particular, if $X_n\xrightarrow{P}c$ and $g$ is continuous at $c$, then $g(X_n)\xrightarrow{P}g(c)$.

Proof. Almost surely: if $X_n(\omega)\to X(\omega)\in C$, then $g(X_n(\omega))\to g(X(\omega))$ by continuity at $X(\omega)$; this holds for almost every $\omega$. In probability: by Theorem 2.31(6), every subsequence of $(X_n)$ has a further subsequence converging a.s. to $X$; along it $g(X_n)\to g(X)$ a.s.; by the same criterion $g(X_n)\xrightarrow{P}g(X)$. In distribution, for continuous $g$: if $h$ is bounded and continuous, so is $h\circ g$, and Theorem 2.32(1) gives $E[h(g(X_n))]\to E[h(g(X))]$, so $g(X_n)\xrightarrow{d}g(X)$. (With exceptional points outside $C$ the proof uses Skorokhod's representation and the a.s. case.) ∎

📘 Definition 2.12 — Bounded in probability; the symbols $O_p$ and $o_p$
A sequence $(X_n)$ is bounded in probability (or tight), written $X_n=O_p(1)$, if for every $\varepsilon\gt0$ there is $K$ with $P(|X_n|\gt K)\lt\varepsilon$ for all $n$. We write $X_n=o_p(1)$ if $X_n\xrightarrow{P}0$, and $X_n=O_p(a_n)$, $X_n=o_p(a_n)$ if $X_n/a_n$ is $O_p(1)$, $o_p(1)$. For example, the CLT says $\bar X_n-\mu=O_p(n^{-1/2})$.

Two facts are used repeatedly. (i) If $X_n\xrightarrow{d}X$, then $X_n=O_p(1)$: choose $K$ with $\pm K$ continuity points of $F_X$ and $P(|X|\gt K)\lt\varepsilon/2$; then $P(|X_n|\gt K)\lt\varepsilon$ for all large $n$, and enlarging $K$ takes care of the finitely many remaining $n$. (ii) $O_p(1)\cdot o_p(1)=o_p(1)$, because $$P(|X_nY_n|\gt\varepsilon)\le P(|X_n|\gt K)+P(|Y_n|\gt\varepsilon/K).$$

📐 Theorem 2.39 — Slutsky's theorem

If $X_n\xrightarrow{d}X$ and $Y_n\xrightarrow{P}c$, where $c$ is a constant, then

  1. $X_n+Y_n\xrightarrow{d}X+c$;
  2. $Y_nX_n\xrightarrow{d}cX$;
  3. $X_n/Y_n\xrightarrow{d}X/c$, provided $c\ne0$.

Proof. (1) Let $x$ be a continuity point of the cdf of $X+c$, i.e. $x-c$ is a continuity point of $F_X$, and choose $\varepsilon\gt0$ with $x-c\pm\varepsilon$ also continuity points (all but countably many $\varepsilon$ qualify). Then $$P(X_n+Y_n\le x)\le P(X_n\le x-c+\varepsilon)+P(|Y_n-c|\gt\varepsilon)\to F_X(x-c+\varepsilon),$$ $$P(X_n+Y_n\le x)\ge P(X_n\le x-c-\varepsilon)-P(|Y_n-c|\gt\varepsilon)\to F_X(x-c-\varepsilon).$$ Letting $\varepsilon\downarrow0$ along such values, $P(X_n+Y_n\le x)\to F_X(x-c)=P(X+c\le x)$.

(2) Write $Y_nX_n=cX_n+(Y_n-c)X_n$. By the continuous mapping theorem, $cX_n\xrightarrow{d}cX$. By fact (i), $X_n=O_p(1)$, and $Y_n-c=o_p(1)$, so $(Y_n-c)X_n=o_p(1)$ by fact (ii). Part (1) (with the constant $0$) gives the claim.

(3) $g(y)=1/y$ is continuous at $c\ne0$, so $1/Y_n\xrightarrow{P}1/c$ by Theorem 2.38; apply (2). ∎

Slutsky's theorem fails if $Y_n$ converges only in distribution to a non-degenerate limit: with $Z\sim N(0,1)$, $X_n=Z$ and $Y_n=-Z$, both converge in distribution to $N(0,1)$, but $X_n+Y_n=0$. Convergence in distribution of each coordinate says nothing about the joint law.

📐 Theorem 2.40 — The delta method

Suppose $\sqrt n\,(T_n-\theta)\xrightarrow{d}N(0,\tau^2)$, and let $g$ be differentiable at $\theta$.

  1. (First order) $\sqrt n\left(g(T_n)-g(\theta)\right)\xrightarrow{d}N\left(0,\,g'(\theta)^2\tau^2\right)$. (If $g'(\theta)=0$ the limit is the constant $0$.)
  2. (Second order) If $g'(\theta)=0$ and $g''(\theta)$ exists, then $n\left(g(T_n)-g(\theta)\right)\xrightarrow{d}\tfrac12g''(\theta)\,\tau^2\,\chi^2_1$.

(Multivariate form, statement: if $\sqrt n(\mathbf{T}_n-\mathbf{t})\xrightarrow{d}N_k(\mathbf 0,\Sigma)$ for a vector $\mathbf{t}\in\mathbb{R}^k$ and $g:\mathbb{R}^k\to\mathbb{R}$ is differentiable at $\mathbf{t}$, then $\sqrt n(g(\mathbf{T}_n)-g(\mathbf{t}))\xrightarrow{d}N(0,\nabla g^{\mathsf T}\Sigma\nabla g)$, the gradient taken at $\mathbf{t}$. It gives the asymptotic variances of ratio estimators in §8.2 and of the sample correlation coefficient.)

Proof. (1) First, $T_n-\theta=\frac{1}{\sqrt n}\cdot\sqrt n(T_n-\theta)=o(1)\cdot O_p(1)\xrightarrow{P}0$. Define $h(t)=\frac{g(t)-g(\theta)}{t-\theta}$ for $t\ne\theta$ and $h(\theta)=g'(\theta)$. Differentiability of $g$ at $\theta$ means exactly that $h$ is continuous at $\theta$, so $h(T_n)\xrightarrow{P}g'(\theta)$ by Theorem 2.38. Now $$\sqrt n\left(g(T_n)-g(\theta)\right)=h(T_n)\cdot\sqrt n(T_n-\theta),$$ and Slutsky's theorem (2) gives the limit $g'(\theta)\,N(0,\tau^2)=N(0,g'(\theta)^2\tau^2)$.

(2) By Taylor's formula with Peano remainder, $g(t)-g(\theta)=\left(\tfrac12g''(\theta)+k(t)\right)(t-\theta)^2$ with $k(t)\to0$ as $t\to\theta$ (and $k(\theta)=0$). Hence $n(g(T_n)-g(\theta))=\left(\tfrac12g''(\theta)+k(T_n)\right)\left(\sqrt n(T_n-\theta)\right)^2$. The first factor tends in probability to $\frac12g''(\theta)$, and by the continuous mapping theorem the second tends in distribution to $\tau^2Z^2$, $Z\sim N(0,1)$, i.e. to $\tau^2\chi^2_1$. Slutsky's theorem finishes the proof. ∎

✏️ Example 6 — Studentising: Slutsky in action
(a) Let $X_1,X_2,\dots$ be iid with mean $\mu$ and finite variance $\sigma^2\gt0$, and let $S_n$ be the sample standard deviation. Show that $T_n=\dfrac{\sqrt n(\bar X_n-\mu)}{S_n}\xrightarrow{d}N(0,1)$, whatever the law of $X_1$. (b) For Bernoulli trials, show that $\dfrac{\hat p-p}{\sqrt{\hat p(1-\hat p)/n}}\xrightarrow{d}N(0,1)$ for $0\lt p\lt1$, and use it for an approximate $95\%$ interval when $400$ patients are treated and $88$ have side effects. (c) Prove or disprove: if $X_n\xrightarrow{d}N(0,1)$ and $Y_n\xrightarrow{d}N(0,1)$, then $X_n+Y_n\xrightarrow{d}N(0,2)$.
(a) By Example 4(c), $S_n^2\to\sigma^2$ a.s., hence in probability, and $S_n\xrightarrow{P}\sigma$ by continuous mapping ($\sqrt{\cdot}$ is continuous). By the CLT, $\sqrt n(\bar X_n-\mu)\xrightarrow{d}N(0,\sigma^2)$. Slutsky (3) with $c=\sigma\ne0$ gives $T_n\xrightarrow{d}N(0,\sigma^2)/\sigma=N(0,1)$. For normal data $T_n$ has exactly Student's $t_{n-1}$ law (§4.1), so this also shows $t_{n-1}\xrightarrow{d}N(0,1)$.
(b) $\hat p\xrightarrow{P}p$ (WLLN), so $\sqrt{\hat p(1-\hat p)}\xrightarrow{P}\sqrt{p(1-p)}\gt0$, and $\frac{\sqrt n(\hat p-p)}{\sqrt{p(1-p)}}\xrightarrow{d}N(0,1)$ (De Moivre–Laplace). Divide and use Slutsky. With $\hat p=0.22$: the standard error is $\sqrt{0.22\times0.78/400}\approx0.02071$, and the approximate $95\%$ interval is $0.22\pm1.96\times0.02071=(0.1794,\,0.2606)$.
(c) False. With $Y_n=-X_n$ and $X_n\sim N(0,1)$, $X_n+Y_n=0$. The statement is true if $X_n$ and $Y_n$ are independent for each $n$: then $\varphi_{X_n+Y_n}=\varphi_{X_n}\varphi_{Y_n}\to e^{-t^2}$, the $N(0,2)$ cf. If instead $Y_n-X_n\xrightarrow{P}0$, then $X_n+Y_n=2X_n+o_p(1)\xrightarrow{d}N(0,4)$. The limit of a sum depends on the joint behaviour of $X_n$ and $Y_n$, not only on their marginal limits.
✏️ Example 7 — The delta method
(a) $X_j$ iid $\operatorname{Poi}(\lambda)$. Find the limit law of $\sqrt n\left(\sqrt{\bar X_n}-\sqrt\lambda\right)$. (b) $X_j$ iid $\operatorname{Exp}(\lambda)$ (rate $\lambda$). Find the limit law of $\sqrt n\left(1/\bar X_n-\lambda\right)$, and an approximate $95\%$ interval for $\lambda$ when $n=100$ and $\bar x=0.4$. (c) For Bernoulli trials, find the limit law of $\sqrt n\left(\hat p(1-\hat p)-p(1-p)\right)$ for $p\ne\frac12$, and of $n\left(\hat p(1-\hat p)-\frac14\right)$ for $p=\frac12$.
(a) $\sqrt n(\bar X_n-\lambda)\xrightarrow{d}N(0,\lambda)$, and $g(x)=\sqrt x$ has $g'(\lambda)=\frac{1}{2\sqrt\lambda}$. So the limit is $N\left(0,\frac{1}{4\lambda}\cdot\lambda\right)=N(0,\tfrac14)$, free of $\lambda$: the square root is a variance-stabilising transformation for Poisson counts.
(b) $\sqrt n(\bar X_n-\frac1\lambda)\xrightarrow{d}N(0,\frac1{\lambda^2})$ and $g(x)=\frac1x$ has $g'(\frac1\lambda)=-\lambda^2$. The limit is $N\left(0,\lambda^4\cdot\frac{1}{\lambda^2}\right)=N(0,\lambda^2)$. So $1/\bar X_n\approx N(\lambda,\lambda^2/n)$; estimating $\lambda$ by $\hat\lambda=1/\bar x=2.5$ in the standard error (Slutsky), the interval is $2.5\pm1.96\times\frac{2.5}{10}=(2.01,\,2.99)$.
(c) $\sqrt n(\hat p-p)\xrightarrow{d}N(0,p(1-p))$ and $g(x)=x(1-x)$ has $g'(p)=1-2p$. For $p\ne\frac12$ the limit is $N\left(0,(1-2p)^2p(1-p)\right)$. For $p=\frac12$, $g'(\frac12)=0$ and $g''=-2$; the second-order delta method with $\tau^2=\frac14$ gives $n\left(\hat p(1-\hat p)-\frac14\right)\xrightarrow{d}\frac12(-2)\cdot\frac14\chi^2_1=-\frac14\chi^2_1$. Directly: $\hat p(1-\hat p)-\frac14=-(\hat p-\frac12)^2$, so the limit is $-\left(\sqrt n(\hat p-\frac12)\right)^2\to-\frac14Z^2$.
⚠️ Counterexamples and pitfalls
  1. In probability does not imply almost surely. The typewriter sequence converges in probability and in every $L^r$ but at no point $\omega$; independent $\operatorname{Ber}(1/n)$ variables converge in probability but have infinitely many $1$s a.s. (Example 1). A subsequence always converges a.s.
  2. Almost surely does not imply convergence of means. $n\mathbf{1}\{U\lt1/n\}\to0$ everywhere, yet its mean is $1$. Convergence of $E[X_n]$ needs domination or uniform integrability (Theorem 2.31(7)).
  3. In distribution does not imply in probability ($X_n=-X$ with $X$ symmetric), except when the limit is a constant (Theorem 2.31(5)).
  4. Convergence in distribution is a property of the laws only. $X_n\xrightarrow{d}X$ and $Y_n\xrightarrow{d}Y$ do not give $X_n+Y_n\xrightarrow{d}X+Y$ or $X_nY_n\xrightarrow{d}XY$ (Example 6(c)); Slutsky needs one of the limits to be a constant.
  5. Cdfs need not converge at jumps of the limit. $X_n\sim N(0,\tfrac1n)$ has $F_n(0)=\tfrac12\not\to1$ (Example 2(c)). Nor do $P(X_n\in B)$ converge for all sets $B$: $X_n$ uniform on $\{\frac1n,\frac2n,\dots,1\}$ converges in law to $U(0,1)$, but $P(X_n\in\mathbb{Q})=1$ while $P(U\in\mathbb{Q})=0$.
  6. Convergence in distribution does not give convergence of moments. $X_n=n$ with probability $\frac1n$ and $0$ otherwise: $X_n\xrightarrow{d}0$, but $E[X_n]=1$ and $E[X_n^2]=n\to\infty$.
  7. No mean, no law of large numbers. For Cauchy data $\bar X_n\sim C(0,1)$ for every $n$; by Theorem 2.35(2) the SLLN holds for iid data iff $E|X_1|\lt\infty$. (The weak law is subtler: it can hold without a mean, e.g. for the symmetric law $P(X=\pm k)\propto\frac{1}{k^2\log k}$ mentioned in §2.2, while the strong law fails there.)
  8. Uncorrelated with growing variances can fail the weak law (Example 3(a) with $\alpha\ge\frac12$), and a common random component defeats it altogether (Example 3(b)).
  9. The CLT needs a finite variance, and it is a statement about the $\sqrt n$-scale. It does not say that $S_n$ is approximately normal far in the tails: for $\operatorname{Bin}(100,\frac12)$, $P(X\ge80)\approx5.6\times10^{-10}$, while the corrected normal value is $1.8\times10^{-9}$, more than three times too large.
  10. Identically distributed matters in Lindeberg–Lévy. For independent $X_k=\pm2^k$ (probability $\frac12$ each), the last term carries three quarters of the variance and $S_n/s_n$ is not asymptotically normal; Lindeberg's condition fails.
  11. Continuity correction only for integer-valued variables. Do not apply it to sums of continuous variables (Example 5(c)).
  12. The delta method with $g'(\theta)=0$ gives the useless limit $0$ at the $\sqrt n$ scale; use the second-order version at scale $n$ (Example 7(c)).

Exercises

I
Modes of convergence.
  • 1 $U\sim U(0,1)$ and $X_n=U^n$. Show that $X_n\to0$ almost surely and in $L^r$ for every $r\gt0$.
  • 2 The $X_n$ are independent with $P(X_n=n^a)=n^{-b}$ and $P(X_n=0)=1-n^{-b}$, where $a,b\gt0$. For which $(a,b)$ does $X_n\to0$ (i) in probability, (ii) almost surely, (iii) in $L^1$, (iv) in $L^2$?
  • 3 Prove that limits in probability are a.s. unique: if $X_n\xrightarrow{P}X$ and $X_n\xrightarrow{P}Y$, then $P(X=Y)=1$.
  • 4 Show that $X_n\xrightarrow{P}X$ and $Y_n\xrightarrow{P}Y$ imply $X_n+Y_n\xrightarrow{P}X+Y$. Give an example with $X_n\xrightarrow{d}X$ and $Y_n\xrightarrow{d}Y$ where $X_n+Y_n$ does not converge in distribution to $X+Y$.
  • 5 Show that if $X_n\xrightarrow{P}X$ and $Y_n\xrightarrow{P}Y$, then $X_nY_n\xrightarrow{P}XY$ (use Theorem 2.31(6)).
  • 6 $X_n\sim N(\mu_n,\sigma_n^2)$ with $\mu_n\to\mu$ and $\sigma_n\to\sigma\gt0$. Show that $X_n\xrightarrow{d}N(\mu,\sigma^2)$. What is the limit if $\sigma_n\to0$?
  • 7 $X_n$ is uniform on $\{\frac1n,\frac2n,\dots,\frac nn\}$. Show that $X_n\xrightarrow{d}U(0,1)$, and that $P(X_n\in B)\to P(U\in B)$ fails for $B=\mathbb{Q}$. Why does this not contradict Theorem 2.32(4)?
  • 8 Prove that $X_n\xrightarrow{P}X$ if and only if $E\left[\min(|X_n-X|,1)\right]\to0$.
1 For every $\omega$ with $U(\omega)\lt1$ (probability $1$), $U^n\to0$. And $E[U^{nr}]=\frac{1}{nr+1}\to0$.
2 (i) $P(|X_n|\gt\varepsilon)=n^{-b}\to0$ for every $b\gt0$: always. (ii) By Theorem 2.30(3), iff $\sum n^{-b}\lt\infty$, i.e. $b\gt1$. (iii) $E[X_n]=n^{a-b}\to0$ iff $a\lt b$. (iv) $E[X_n^2]=n^{2a-b}\to0$ iff $b\gt2a$. For instance $a=2$, $b=3$ gives a.s. and $L^1$ convergence but not $L^2$; $a=\frac14$, $b=\frac34$ gives $L^2$ but not a.s.
3 $P(|X-Y|\gt2\varepsilon)\le P(|X_n-X|\gt\varepsilon)+P(|X_n-Y|\gt\varepsilon)\to0$, so $P(|X-Y|\gt2\varepsilon)=0$ for every $\varepsilon\gt0$; let $\varepsilon\downarrow0$ through $1/k$.
4 $\{|X_n+Y_n-X-Y|\gt\varepsilon\}\subseteq\{|X_n-X|\gt\varepsilon/2\}\cup\{|Y_n-Y|\gt\varepsilon/2\}$. For the example, let $X\sim N(0,1)$, $X_n=X$, $Y_n=-X$ and $Y=X$: then $X_n\xrightarrow{d}X$, $Y_n\xrightarrow{d}Y$ (same laws), but $X_n+Y_n=0$ while $X+Y=2X\sim N(0,4)$.
5 Take any subsequence. By Theorem 2.31(6) it has a further subsequence along which $X_n\to X$ a.s., and a further one along which also $Y_n\to Y$ a.s.; along it $X_nY_n\to XY$ a.s. So every subsequence of $(X_nY_n)$ has an a.s. convergent sub-subsequence with limit $XY$, and Theorem 2.31(6) gives $X_nY_n\xrightarrow{P}XY$.
6 $\varphi_{X_n}(t)=e^{i\mu_nt-\sigma_n^2t^2/2}\to e^{i\mu t-\sigma^2t^2/2}$, continuous, so Lévy's theorem applies. (Or: $F_n(x)=\Phi\left(\frac{x-\mu_n}{\sigma_n}\right)\to\Phi\left(\frac{x-\mu}{\sigma}\right)$ by continuity of $\Phi$.) If $\sigma_n\to0$ the limit is the constant $\mu$, and the convergence is even in probability.
7 For $0\le x\le1$, $P(X_n\le x)=\lfloor nx\rfloor/n\to x$. $P(X_n\in\mathbb{Q})=1$ for every $n$, but $P(U\in\mathbb{Q})=0$. Theorem 2.32(4) requires densities, and $X_n$ has none: convergence in distribution only controls sets whose boundary has limit probability $0$, and the boundary of $\mathbb{Q}$ is all of $\mathbb{R}$.
8 Let $D_n=\min(|X_n-X|,1)$. For $0\lt\varepsilon\lt1$, Markov gives $P(|X_n-X|\gt\varepsilon)=P(D_n\gt\varepsilon)\le E[D_n]/\varepsilon$, proving "if". Conversely $E[D_n]\le\varepsilon+P(|X_n-X|\gt\varepsilon)$ because $D_n\le1$, so $\limsup E[D_n]\le\varepsilon$ for every $\varepsilon$. (So $d(X,Y)=E[\min(|X-Y|,1)]$ is a metric for convergence in probability; there is no such metric for a.s. convergence.)
II
Laws of large numbers.
  • 1 $U_1,U_2,\dots$ are iid $U(0,1)$. Find the almost-sure limits of $\dfrac{U_1^2+\dots+U_n^2}{U_1+\dots+U_n}$ and of $\dfrac{1}{n}\sum_{j=1}^n\mathbf{1}\{U_j\le U_j^{\,2}+\tfrac14\}$.
  • 2 Show that $\frac1n\sum_{j=1}^n(X_j-\bar X_n)^2\xrightarrow{P}\sigma^2$ for iid $X_j$ with finite variance, using only Khintchine's weak law and the continuous mapping theorem.
  • 3 (Monte Carlo) $\frac\pi4=\int_0^1\sqrt{1-x^2}\,dx$ is estimated by $\frac1n\sum\sqrt{1-U_j^2}$. Find the variance of one term, and the $n$ needed for an error of at most $0.01$ with probability at least $0.95$ according to Chebyshev's inequality and according to the CLT.
  • 4 The $X_k$ are independent with $X_k\sim U(-k^\beta,k^\beta)$. Does $\bar X_n\xrightarrow{P}0$ when (a) $\beta=\frac14$, (b) $\beta=\frac12$?
  • 5 Let $X_j$ be iid with $E|X_1|\lt\infty$. Show that $X_n/n\to0$ a.s. and deduce that $\frac1n\max_{j\le n}|X_j|\to0$ a.s.
  • 6 The $X_j$ are iid with density $x^{-2}$ on $(1,\infty)$. Show that $\bar X_n\to\infty$ almost surely. (Hint: truncate at a level $M$.)
  • 7 (Borel's normal numbers) Let $U\sim U(0,1)$ have decimal expansion $0.D_1D_2D_3\dots$. Show that the $D_j$ are iid uniform on $\{0,\dots,9\}$ and that, almost surely, each digit appears with limiting frequency $\frac1{10}$.
  • 8 (Weierstrass via the weak law) Let $f$ be continuous on $[0,1]$ and $B_n(x)=\sum_{k=0}^nf\left(\frac kn\right)\binom nkx^k(1-x)^{n-k}$, the Bernstein polynomial. Prove that $\sup_{0\le x\le1}|B_n(x)-f(x)|\to0$.
1 Divide numerator and denominator by $n$: by the SLLN they tend a.s. to $E[U^2]=\frac13$ and $E[U]=\frac12$, so the ratio tends to $\frac23$ (continuous mapping, $(a,b)\mapsto a/b$ at $b=\frac12$). Since $u^2-u+\frac14=(u-\frac12)^2\ge0$, the indicator is $1$ always, and the average is $1$. (Check your inequality before computing!)
2 $\frac1n\sum(X_j-\bar X_n)^2=\frac1n\sum X_j^2-\bar X_n^2$. Khintchine applies to the iid $X_j^2$ (finite mean $E[X_1^2]$) and to $X_j$, and $(a,b)\mapsto a-b^2$ is continuous; convergence in probability is preserved (Theorem 2.38, applied coordinatewise via Exercise I.4–5).
3 One term has mean $\frac\pi4$ and second moment $E[1-U^2]=\frac23$, so variance $v=\frac23-\frac{\pi^2}{16}\approx0.04982$. Chebyshev: $n\ge\frac{v}{0.05\times10^{-4}}\approx9964$. CLT: $n\ge\left(\frac{1.96}{0.01}\right)^2v\approx1914$.
4 $\operatorname{Var}(X_k)=\frac{k^{2\beta}}{3}$. (a) $\frac{1}{n^2}\sum\frac{k^{1/2}}{3}\sim\frac{2n^{3/2}}{9n^2}\to0$: the weak law holds by Theorem 2.33. (b) $\frac{1}{n^2}\sum\frac k3\to\frac16$. The $X_k$ are bounded by $\sqrt n$ while $s_n\sim n/\sqrt6$, so Lindeberg's condition holds and $S_n/s_n\xrightarrow{d}N(0,1)$; hence $\bar X_n\xrightarrow{d}N(0,\frac16)$ and the weak law fails.
5 $\sum_nP(|X_n|\gt\varepsilon n)=\sum_nP(|X_1|/\varepsilon\gt n)\le E|X_1|/\varepsilon\lt\infty$ (tail formula), so $|X_n|\le\varepsilon n$ eventually, a.s., for every rational $\varepsilon$: $X_n/n\to0$ a.s. Given $\varepsilon$, fix such an $\omega$ and $N$ with $|X_j|\le\varepsilon j$ for $j\gt N$; then $\frac1n\max_{j\le n}|X_j|\le\frac1n\max_{j\le N}|X_j|+\varepsilon\to\varepsilon$.
6 $E[X_1]=\int_1^\infty x\cdot x^{-2}dx=\infty$. For fixed $M\gt1$, $\bar X_n\ge\frac1n\sum\min(X_j,M)\to E[\min(X_1,M)]=\int_1^Mx^{-1}dx+M\cdot\frac1M=1+\log M$ a.s. by the SLLN for the bounded variables $\min(X_j,M)$. So $\liminf\bar X_n\ge1+\log M$ a.s. for every $M\in\mathbb{N}$, and $\bar X_n\to\infty$ a.s.
7 For any digits $d_1,\dots,d_k$, $P(D_1=d_1,\dots,D_k=d_k)$ is the length of an interval of length $10^{-k}$, which equals $\prod P(D_j=d_j)$ with $P(D_j=d)=\frac1{10}$: iid uniform digits. The SLLN applied to $\mathbf{1}\{D_j=d\}$ gives frequency $\frac1{10}$ for each $d$ a.s. Intersecting the ten events of probability $1$ (and doing the same for all bases and blocks of digits) shows that almost every number is normal, although no explicit normal number in base $10$ is easy to verify.
8 $B_n(x)=E[f(S_n/n)]$ with $S_n\sim\operatorname{Bin}(n,x)$. $f$ is bounded by some $K$ and uniformly continuous: given $\varepsilon$, choose $\delta$ with $|f(u)-f(x)|\lt\varepsilon$ when $|u-x|\lt\delta$. Then $$|B_n(x)-f(x)|\le E|f(S_n/n)-f(x)|\le\varepsilon+2K\,P(|S_n/n-x|\ge\delta)\le\varepsilon+\frac{2Kx(1-x)}{n\delta^2}\le\varepsilon+\frac{K}{2n\delta^2},$$ uniformly in $x$. Let $n\to\infty$, then $\varepsilon\to0$.
III
The central limit theorem and normal approximations.
  • 1 A fair coin is tossed $400$ times. Approximate the probability that the number of heads is between $190$ and $215$ inclusive.
  • 2 $X\sim\operatorname{Poi}(100)$. Approximate $P(X\le85)$ with continuity correction, and compare with the exact value $0.0708$.
  • 3 (Airline overbooking) A plane has $200$ seats and each ticket holder turns up independently with probability $0.9$. Using the normal approximation with continuity correction, find the largest number of tickets that can be sold with $P(\text{more than }200\text{ turn up})\le0.05$. The exact binomial answer is $215$; explain the discrepancy.
  • 4 Let $Z=U_1+\dots+U_{12}-6$ with $U_j$ iid $U(0,1)$ (an old method for simulating normals). Find $E[Z]$, $\operatorname{Var}(Z)$ and the normal approximation to $P(|Z|\gt2)$; the exact value is $0.04455$. Which feature of the normal law can $Z$ never reproduce?
  • 5 Let $Z_n=\sqrt n(\bar X_n-\mu)/\sigma$ for iid data with finite variance. Show that $Z_n$ does not converge in probability, by finding the limit law of $Z_{2n}-Z_n$.
  • 6 (Lift safety) Passenger weights are iid with mean $70$ kg and standard deviation $15$ kg. A lift's load limit is $1000$ kg. What is the largest number of passengers for which the approximate probability of overload is at most $0.05$?
  • 7 (Records) Let $I_k$ be independent $\operatorname{Ber}(p_k)$ with $\sum p_k(1-p_k)=\infty$. Verify Lyapunov's condition with $\delta=1$, and deduce that the number $R_n$ of records among $n$ iid continuous observations (where $p_k=1/k$) satisfies $\frac{R_n-\log n}{\sqrt{\log n}}\xrightarrow{d}N(0,1)$. (You may use $\sum_{k\le n}\frac1k=\log n+O(1)$.)
  • 8 Use the CLT for Poisson variables to prove that $e^{-n}\sum_{k=0}^n\frac{n^k}{k!}\to\frac12$.
Check lab: the continuity correction (Theorem 2.37(1))

Choose $n$, $p$ and $k$. The lab computes $P(X\le k)$ for $X\sim\operatorname{Bin}(n,p)$ exactly and by the normal approximation without and with the continuity correction $k+\frac12$. Try small $n$ or $p$ near $0$ to see when the correction matters and when skewness spoils both.

400.3010
exact P(X ≤ k)—
normal, no correction—
normal, with correction—
1 Mean $200$, SD $10$: $\Phi\left(\frac{215.5-200}{10}\right)-\Phi\left(\frac{189.5-200}{10}\right)=\Phi(1.55)-\Phi(-1.05)\approx0.9394-0.1469=0.7926$. The exact value is $0.7926$.
2 $\Phi\left(\frac{85.5-100}{10}\right)=\Phi(-1.45)\approx0.0735$, against the exact $0.0708$. The approximation overstates the left tail because the Poisson law is skewed to the right (skewness $0.1$).
3 With $n$ tickets, $X\sim\operatorname{Bin}(n,0.9)$ and we need $1-\Phi\left(\frac{200.5-0.9n}{\sqrt{0.09n}}\right)\le0.05$, i.e. $\frac{200.5-0.9n}{0.3\sqrt n}\ge1.645$. For $n=214$ the left side is $1.80$ (probability $0.036$), for $n=215$ it is $1.59$ (probability $0.056$); so the approximation allows $214$ tickets. The exact binomial probabilities are $0.030$ and $0.0496$, so $215$ is in fact acceptable. Here $n(1-p)\approx21$ no-shows is small and the law of $X$ is skewed to the left, so the normal approximation overstates the upper tail; near a cut-off like $0.05$ this decides the answer.
4 $E[Z]=0$ and $\operatorname{Var}(Z)=12\cdot\frac1{12}=1$, so $P(|Z|\gt2)\approx2(1-\Phi(2))\approx0.0455$, close to the exact $0.04455$. But $|Z|\le6$ always, so $Z$ has no tail beyond $6$ (and too light a tail near it), and its kurtosis is $3-\frac{6}{5\cdot12}=2.9$ instead of $3$.
5 Write $S_{2n}=S_n+S_n'$ with $S_n'$ the sum of $X_{n+1},\dots,X_{2n}$, independent of $S_n$. Centring at $\mu$, $Z_{2n}-Z_n=\left(\frac{1}{\sqrt2}-1\right)Z_n+\frac{1}{\sqrt2}Z_n'$ with $Z_n,Z_n'$ independent and each $\xrightarrow{d}N(0,1)$. By independence and cfs, $Z_{2n}-Z_n\xrightarrow{d}N\left(0,\left(1-\frac1{\sqrt2}\right)^2+\frac12\right)=N(0,2-\sqrt2)$, a non-degenerate law. If $Z_n\xrightarrow{P}Z$, then $Z_{2n}-Z_n\xrightarrow{P}0$, a contradiction. The CLT is genuinely a statement about laws.
6 The total weight of $n$ passengers is approximately $N(70n,225n)$. We need $\frac{1000-70n}{15\sqrt n}\ge1.645$. For $n=13$: $\frac{90}{54.08}=1.664$, overload probability $\approx0.048$; for $n=14$: $\frac{20}{56.12}=0.356$, probability $\approx0.36$. So at most $13$ passengers, although $1000/70\approx14.3$.
7 $E|I_k-p_k|^3=p_k(1-p_k)\left[(1-p_k)^2+p_k^2\right]\le p_k(1-p_k)$, so the Lyapunov ratio is at most $\frac{s_n^2}{s_n^3}=\frac1{s_n}\to0$. For records, $I_k=\mathbf{1}\{X_k\gt\max_{j\lt k}X_j\}$ are independent with $p_k=\frac1k$ (a classical fact: the rank of $X_k$ among the first $k$ observations is uniform and independent of the earlier ranks). Then $E[R_n]=\sum\frac1k=\log n+O(1)$ and $s_n^2=\sum\frac1k-\sum\frac1{k^2}=\log n+O(1)$; Slutsky allows replacing the exact centring and scaling by $\log n$ and $\sqrt{\log n}$.
8 $e^{-n}\sum_{k\le n}\frac{n^k}{k!}=P(Y_n\le n)$ for $Y_n\sim\operatorname{Poi}(n)$, which is the law of a sum of $n$ iid $\operatorname{Poi}(1)$ variables. By the CLT, $P(Y_n\le n)=P\left(\frac{Y_n-n}{\sqrt n}\le0\right)\to\Phi(0)=\frac12$. (Numerically: $0.583$ for $n=10$, $0.527$ for $n=100$, $0.508$ for $n=1000$; the slow approach reflects the atom at $n$ of size about $1/\sqrt{2\pi n}$.)
IV
Slutsky, continuous mapping and the delta method.
  • 1 $X_j$ iid $U(0,\theta)$. Find the limit laws of $\sqrt n(2\bar X_n-\theta)$ and of $n(\theta-M_n)$ with $M_n=\max_{j\le n}X_j$. Which estimator of $\theta$ converges faster?
  • 2 For Bernoulli trials, find the limit law of the estimated log-odds $\sqrt n\left(\log\frac{\hat p}{1-\hat p}-\log\frac{p}{1-p}\right)$.
  • 3 $X_j$ iid with mean $\mu$ and variance $\sigma^2$. Find the limit law of $\sqrt n(\bar X_n^2-\mu^2)$ when $\mu\ne0$, and of $n\bar X_n^2$ when $\mu=0$.
  • 4 Show that if $E[X_1^4]\lt\infty$, then $\sqrt n(S_n^2-\sigma^2)\xrightarrow{d}N(0,\mu_4-\sigma^4)$, where $\mu_4=E[(X_1-\mu)^4]$. Evaluate the limit variance for normal data.
  • 5 $X_j$ iid exponential with mean $\theta$. Find a function $g$ for which the limit variance of $\sqrt n(g(\bar X_n)-g(\theta))$ does not depend on $\theta$, and use it to write an approximate $95\%$ interval for $\theta$.
  • 6 Prove or disprove: (a) if $X_n\xrightarrow{d}X$ and $X_n-Y_n\xrightarrow{P}0$, then $Y_n\xrightarrow{d}X$; (b) if $X_n\xrightarrow{d}X$, then $X_n-X\xrightarrow{P}0$; (c) if $X_n\xrightarrow{d}X$ and $Y_n\xrightarrow{P}c$, then $(X_n+Y_n)^2\xrightarrow{d}(X+c)^2$.
  • 7 (Clinical trial) In $n=400$ patients, $x=88$ have a side effect. Give the Wald $95\%$ interval for $p$ and the delta-method $95\%$ interval for the log-odds, transformed back to $p$.
  • 8 $X_j$ iid with mean $0$ and variance $\sigma^2$. Find the limit law of $n\left(1-\cos\bar X_n\right)$.
1 $E[X]=\frac\theta2$, $\operatorname{Var}(X)=\frac{\theta^2}{12}$, so $\sqrt n(2\bar X_n-\theta)\xrightarrow{d}N\left(0,\frac{4\theta^2}{12}\right)=N(0,\frac{\theta^2}{3})$. As in Example 2(a), $P(n(\theta-M_n)\gt x)=(1-\frac{x}{n\theta})^n\to e^{-x/\theta}$: the limit is exponential with mean $\theta$. $M_n$ has error of order $\frac1n$, $2\bar X_n$ of order $\frac1{\sqrt n}$: the maximum is far better (it is the basis of the MLE in §4.2).
2 $g(p)=\log\frac{p}{1-p}$ has $g'(p)=\frac{1}{p(1-p)}$, so the limit is $N\left(0,\frac{p(1-p)}{p^2(1-p)^2}\right)=N\left(0,\frac{1}{p(1-p)}\right)$.
3 $g(x)=x^2$, $g'(\mu)=2\mu$: $\sqrt n(\bar X_n^2-\mu^2)\xrightarrow{d}N(0,4\mu^2\sigma^2)$. If $\mu=0$: $n\bar X_n^2=(\sqrt n\bar X_n)^2\xrightarrow{d}\sigma^2\chi^2_1$ by continuous mapping (the second-order delta method with $g''=2$).
4 With $Y_j=(X_j-\mu)^2$ (mean $\sigma^2$, variance $\mu_4-\sigma^4$), $\frac{n-1}{n}S_n^2=\bar Y_n-(\bar X_n-\mu)^2$. The CLT gives $\sqrt n(\bar Y_n-\sigma^2)\xrightarrow{d}N(0,\mu_4-\sigma^4)$, and $\sqrt n(\bar X_n-\mu)^2=\frac{1}{\sqrt n}\left(\sqrt n(\bar X_n-\mu)\right)^2=o(1)O_p(1)\xrightarrow{P}0$. Also $\sqrt n\left(S_n^2-\frac{n-1}nS_n^2\right)=\frac{S_n^2}{\sqrt n}\xrightarrow{P}0$. Slutsky gives the claim. For normal data $\mu_4=3\sigma^4$, so the limit variance is $2\sigma^4$.
5 $\sqrt n(\bar X_n-\theta)\xrightarrow{d}N(0,\theta^2)$. We need $g'(\theta)^2\theta^2$ constant: $g'(\theta)=\frac1\theta$, $g=\log$. Then $\sqrt n(\log\bar X_n-\log\theta)\xrightarrow{d}N(0,1)$, so $\log\theta\in\log\bar x\pm\frac{1.96}{\sqrt n}$ and $\theta\in\left(\bar xe^{-1.96/\sqrt n},\ \bar xe^{1.96/\sqrt n}\right)$ approximately.
6 (a) True: $Y_n=X_n+(Y_n-X_n)$ and Slutsky (1) with $c=0$. (b) False: Example 1(d), or $X_n=X'$, an independent copy of $X$. (c) True: Slutsky gives $X_n+Y_n\xrightarrow{d}X+c$, and $x\mapsto x^2$ is continuous.
7 Wald: $0.22\pm1.96\times0.02071=(0.1794,0.2606)$ (Example 6(b)). Log-odds: $\hat\ell=\log\frac{0.22}{0.78}\approx-1.2657$ with standard error $\frac{1}{\sqrt{400\times0.22\times0.78}}\approx0.1207$, so $\ell\in(-1.5022,-1.0292)$. Back-transforming with $p=\frac{e^\ell}{1+e^\ell}$ gives $(0.1821,\,0.2633)$, slightly asymmetric about $0.22$ and always inside $(0,1)$.
8 $g(x)=1-\cos x$ has $g(0)=0$, $g'(0)=0$ and $g''(0)=1$. By the second-order delta method with $\tau^2=\sigma^2$, $n(1-\cos\bar X_n)\xrightarrow{d}\frac12\sigma^2\chi^2_1$.
🎯 Exam corner CSIR Part BCSIR Part CGATE ST

E1. CSIR Part B GATE ST $X_1,X_2,\dots$ are iid exponential with mean $2$. With probability one, $\dfrac{X_1^2+\dots+X_n^2}{X_1+\dots+X_n}$ converges to (A) $2$; (B) $4$; (C) $8$; (D) it does not converge.

Idea Divide top and bottom by $n$ and apply the SLLN to each, then continuous mapping for the ratio.
Compute $E[X]=2$ and $E[X^2]=\operatorname{Var}(X)+4=4+4=8$. The ratio tends to $\frac82=4$ a.s. Option (C) is $E[X^2]$ alone, and (A) is the mean.
Answer (B).

E2. CSIR Part B $X\sim\operatorname{Bin}(100,\tfrac12)$. The normal approximation with continuity correction to $P(X=50)$ is closest to (A) $0$; (B) $0.040$; (C) $0.080$; (D) $0.159$.

Idea Mean $50$, SD $5$; the single value $50$ corresponds to the interval $[49.5,50.5]$.
Compute $\Phi(0.1)-\Phi(-0.1)=2\Phi(0.1)-1\approx0.0797$; the exact value $\binom{100}{50}2^{-100}\approx0.0796$. Option (A) is what one gets without the correction; (D) uses $[49,51]$, i.e. the width $2$ instead of $1$ (and is roughly double).
Answer (C).

E3. CSIR Part B GATE ST $X_1,X_2,\dots$ are iid exponential with rate $\lambda$. Then $\sqrt n\left(\log\bar X_n+\log\lambda\right)$ converges in distribution to $N(0,v)$ with $v$ equal to (A) $1$; (B) $\lambda^2$; (C) $1/\lambda^2$; (D) $\lambda$.

Idea Delta method with $g=\log$ at $\theta=1/\lambda$; note $\log\bar X_n+\log\lambda=g(\bar X_n)-g(1/\lambda)$.
Compute $\tau^2=\operatorname{Var}(X)=\frac{1}{\lambda^2}$ and $g'(1/\lambda)=\lambda$, so $v=\lambda^2\cdot\frac{1}{\lambda^2}=1$. The logarithm stabilises the variance of exponential means.
Answer (A).

E4. CSIR Part C Let $X_1,X_2,\dots$ be independent with $P(X_n=1)=\frac1n$ and $P(X_n=0)=1-\frac1n$. Which of the following are true? (One or more options may be correct.) (A) $X_n\to0$ in probability. (B) $X_n\to0$ almost surely. (C) $X_n\to0$ in mean square. (D) $\sum_{n\ge1}X_n\lt\infty$ almost surely.

(A) True $P(X_n\ne0)=\frac1n\to0$.
(B) False The events $\{X_n=1\}$ are independent with $\sum\frac1n=\infty$; by the second Borel–Cantelli lemma $X_n=1$ infinitely often a.s.
(C) True $E[X_n^2]=\frac1n\to0$.
(D) False Infinitely many terms equal $1$ a.s. (see (B)), so the sum is $+\infty$ a.s.
Answer (A) and (C).

E5. CSIR Part C Let $X_1,X_2,\dots$ be iid and $\bar X_n=\frac1n\sum_{j\le n}X_j$. Which of the following are true? (A) If $E[X_1^2]\lt\infty$, then $\bar X_n\to E[X_1]$ almost surely. (B) If $X_1$ is standard Cauchy, then $\bar X_n\to0$ in probability. (C) If $X_1\sim U(0,1)$, then $\frac1n\sum_{j\le n}\log X_j\to-1$ almost surely. (D) If $E|X_1|=\infty$, then $\bar X_n$ cannot converge almost surely to a finite constant.

(A) True A finite second moment implies $E|X_1|\lt\infty$; Kolmogorov's SLLN applies.
(B) False $\bar X_n\sim C(0,1)$ for every $n$, so $P(|\bar X_n|\gt1)=\frac12$ for all $n$.
(C) True $E|\log U|=\int_0^1(-\log u)\,du=1\lt\infty$ and $E[\log U]=-1$; apply the SLLN.
(D) True Theorem 2.35(2): then $\limsup|\bar X_n|=\infty$ a.s.
Answer (A), (C) and (D).

E6. CSIR Part C GATE ST Let $X_1,X_2,\dots$ be iid with mean $0$ and variance $1$, and $S_n=X_1+\dots+X_n$. Which of the following are true? (A) $S_n/\sqrt n\xrightarrow{d}N(0,1)$. (B) $S_n/n\to0$ almost surely. (C) $S_n^2/n\xrightarrow{d}\chi^2_1$. (D) $S_n/\sqrt n$ converges in probability.

(A) True The Lindeberg–Lévy CLT.
(B) True The SLLN ($E|X_1|\le\sqrt{E[X_1^2]}=1$).
(C) True $S_n^2/n=(S_n/\sqrt n)^2$; continuous mapping with $x\mapsto x^2$ turns the $N(0,1)$ limit into $\chi^2_1$.
(D) False Exercise III.5: $Z_{2n}-Z_n\xrightarrow{d}N(0,2-\sqrt2)$, which is impossible if $Z_n$ converges in probability.
Answer (A), (B) and (C).
🧑‍🏫 For the teacher

Suggested plan: 3 lectures × 60 min.

  1. Modes of convergence. Definition 2.11 with pictures of sample paths. Theorem 2.30 (prove (1) and (2) via Borel–Cantelli). Theorem 2.31 (1), (2), (4), (5), (6) on the board; (3) and (7) briefly. The implication table, then Example 1, ideally with Interactive 1 projected: typewriter versus $\operatorname{Ber}(1/n)$ versus $\operatorname{Ber}(1/n^2)$ versus $n\mathbf{1}\{U\lt1/n\}$. Theorem 2.32 (prove (1) "if" and (3)), Example 2.
  2. Laws of large numbers and the CLT. Theorem 2.33 (one line), Theorem 2.34 via cfs, Theorem 2.35 with the fourth-moment proof and the converse. Example 3 as a class discussion, Example 4(b). Then Theorem 2.36: do the four-step proof in full, emphasising where the second moment enters. Interactive 2 with the exponential and Bernoulli(0.1) parents, then the Cauchy parent.
  3. Using the limit theorems. Theorem 2.37 and the continuity correction with the check lab; Example 5. Theorem 2.38, Definition 2.12, Slutsky (prove (1) and (2)), Example 6(a) (why the $t$-statistic is asymptotically normal), the delta method and Example 7. Finish with E4–E6.

Discussion questions. (1) An insurer with a million independent policies relies on the law of large numbers. Which assumption fails for flood insurance in one river valley, and what does Example 3(b) say about it? (2) Why can convergence in probability be described by a metric (Exercise I.8) but almost-sure convergence cannot? (3) The CLT is often quoted as "$n\ge30$ is enough". Use Interactive 2 and the Berry–Esseen bound to argue for or against this rule for lognormal data.

Misconceptions to watch for. Treating convergence in distribution as a statement about the random variables rather than their laws; expecting $E[X_n]\to E[X]$ from a.s. convergence; forgetting that Slutsky needs a constant limit; applying the CLT to Cauchy or other infinite-variance data; continuity-correcting sums of continuous variables; reading "$\bar X_n\approx N(\mu,\sigma^2/n)$" as convergence of $\bar X_n$ to a normal law (it converges to the constant $\mu$); using the delta method blindly when $g'(\theta)=0$.

Suggested homework. I(2, 4, 7, 8), II(1, 3, 6, 8), III(1, 3, 5, 8), IV(1, 3, 4, 7); Exam corner E2, E4, E6.

🔬 Research–Teaching NexusResearch-tutored · optional, about 60 minutes

Research question. Is "$n\ge30$" enough for the central limit theorem? For realistic skewed data, such as hospital lengths of stay, household incomes or insurance claim sizes, how large must $n$ be before the nominal $95\%$ interval $\bar x\pm1.96\,s/\sqrt n$ really covers the mean $95\%$ of the time?

Why it matters. Health-economics studies report mean costs per patient, and regulators approve budgets based on them; insurers price policies from mean claim sizes. Cost and claim data are strongly right-skewed (close to lognormal or gamma), and for such data the CLT can be slow. An interval that covers only $88\%$ of the time while claiming $95\%$ misleads decision makers.

What to do.

  1. Choose parent laws: exponential, gamma with shape $0.5$, lognormal with $\sigma=1$ and with $\sigma=1.5$. For each, record the skewness $\gamma_1$.
  2. For $n\in\{10,30,100,300,1000\}$, simulate $10\,000$ samples with a seeded generator, compute the $t$-interval for each and record its coverage of the true mean, and separately the proportion of misses on each side.
  3. Plot coverage against $n$, and compare the miss rates with the Berry–Esseen bound $0.4748\rho/(\sigma^3\sqrt n)$ and with the heuristic error $\gamma_1/(6\sqrt n)$ suggested by an Edgeworth expansion.
  4. Propose and test one remedy: a log transformation followed by the delta method, or a bootstrap percentile interval.

Tools: Python (numpy, scipy.stats) or R; Interactive 2 for intuition about skewed parents.

Hand in: A coverage table and plot for each parent law, a paragraph replacing the "$n\ge30$" rule by one that depends on skewness, and the results of your remedy.

Going further. Read about Edgeworth expansions, which correct the normal approximation by a term proportional to $\gamma_1/\sqrt n$, and about the bootstrap, which estimates the sampling distribution of a statistic by resampling the data. Both are standard tools in modern applied statistics.

Instructor note: in our seeded simulations ($20\,000$ samples each) the nominal $95\%$ $t$-interval for exponential data covered the mean about $90\%$ of the time at $n=10$, $93\%$ at $n=30$ and $94\%$ at $n=100$, and almost all misses had the whole interval below the mean; for lognormal data with $\sigma=1.5$ the coverage was only about $78\%$ at $n=30$, $85\%$ at $n=100$ and $89\%$ at $n=300$. Students should notice that the two-sided coverage hides a much larger one-sided error.

🧪 Self-check quiz

0/5 correct

Q1. Which implication holds for every sequence of random variables?

Q2. The typewriter sequence of indicators of dyadic intervals sweeping across $(0,1)$ converges to $0$

Q3. A fair coin is tossed $400$ times. The normal approximation with continuity correction to $P(\text{at most }210\text{ heads})$ is closest to

Q4. $X_n\xrightarrow{d}N(0,1)$ and $Y_n\to2$ in probability. Then $Y_nX_n+1$ converges in distribution to

Q5. $X_j$ iid with mean $\mu\ne0$ and variance $\sigma^2$. The limit law of $\sqrt n(\bar X_n^2-\mu^2)$ is