Reference

Formula Sheet

Every key formula of the course on one page, grouped by chapter, with a link back to the section that develops it.

Chapter 1 · Probability

Extremal properties of mean and median§ 1.1 →
$\sum(x_i-c)^2=\sum(x_i-\bar x)^2+n(\bar x-c)^2$ (minimised at $c=\bar x$); $\ \sum|x_i-c|$ is minimised at any median.
Grouped data§ 1.1 →
$\mathrm{Me}=L+\frac{N/2-F}{f}h$, $\ \mathrm{Mo}=L+\frac{f_1-f_0}{2f_1-f_0-f_2}h$; Sheppard: $\mu_2\approx\bar\mu_2-\frac{h^2}{12}$, $\ \mu_4\approx\bar\mu_4-\frac{h^2}{2}\bar\mu_2+\frac{7h^4}{240}$ (continuous, high-contact data).
Variance identities and bounds§ 1.1 →
$s^2=\overline{x^2}-\bar x^2=\frac{1}{2n^2}\sum_{i,j}(x_i-x_j)^2$; pooled $s^2=\frac1N\sum n_js_j^2+\frac1N\sum n_j(\bar x_j-\bar x)^2$; $\ \mathrm{MD}(\mathrm{Me})\le\mathrm{MD}(\bar x)\le s\le R/2$; $\ |x_k-\bar x|\le s\sqrt{n-1}$.
Moments, skewness, kurtosis§ 1.1 →
$m_3=m_3'-3m_2'd+2d^3$, $\ m_4=m_4'-4m_3'd+6m_2'd^2-3d^4$ ($d=\bar x-A$); $\ \gamma_1=\frac{m_3}{m_2^{3/2}}$, $\beta_2=\frac{m_4}{m_2^2}\ge1+\beta_1$.
Correlation and regression lines§ 1.1 →
$r=\frac{s_{xy}}{s_xs_y}\in[-1,1]$; $\ y-\bar y=b_{yx}(x-\bar x)$, $b_{yx}=r\frac{s_y}{s_x}$; $\ b_{yx}b_{xy}=r^2$; residual MS $=s_y^2(1-r^2)$; $\ \tan\theta=\frac{1-r^2}{|r|}\frac{s_xs_y}{s_x^2+s_y^2}$.
Consequences of the axioms§ 1.2 →
$P(A^c)=1-P(A)$; $\ P(A\cup B)=P(A)+P(B)-P(AB)$; $\ P(\bigcup A_n)\le\sum P(A_n)$; $\ A_n\uparrow A$ or $A_n\downarrow A\Rightarrow P(A_n)\to P(A)$; $\ \max(0,P(A)+P(B)-1)\le P(AB)\le\min(P(A),P(B))$.
Inclusion–exclusion and matching§ 1.2 →
$P(\bigcup_1^nA_i)=\sum_k(-1)^{k+1}S_k$; $\ P(\text{no match})=\sum_{k=0}^n\frac{(-1)^k}{k!}\to\frac1e$; $\ P(\text{exactly }k)\to\frac{e^{-1}}{k!}$.
Total probability and Bayes§ 1.2 →
$P(A)=\sum_iP(A\mid B_i)P(B_i)$; $\ P(B_j\mid A)=\frac{P(A\mid B_j)P(B_j)}{\sum_iP(A\mid B_i)P(B_i)}$; posterior odds $=$ likelihood ratio $\times$ prior odds.
Classic problems§ 1.2 →
Birthday: $1-\prod_{k=0}^{n-1}(1-\frac{k}{365})$ ($n=23$: $0.507$); Pólya: $P(R_n)=\frac{r}{r+b}$; ruin: $P_i(\text{reach }N)=\frac{1-\rho^i}{1-\rho^N}$, $\rho=\frac qp$ ($\frac iN$ if $p=\frac12$).
Independence and Borel–Cantelli§ 1.2 →
Mutual independence: product rule for every finite subfamily ($2^n-n-1$ conditions). $\sum P(A_n)\lt\infty\Rightarrow P(A_n\text{ i.o.})=0$; independent and $\sum P(A_n)=\infty\Rightarrow P(A_n\text{ i.o.})=1$.
Probabilities from the cdf§ 1.3 →
$P(X=x)=F(x)-F(x-)$; $\ P(a\lt X\le b)=F(b)-F(a)$; $\ P(a\le X\le b)=F(b)-F(a-)$; $\ P(a\lt X\lt b)=F(b-)-F(a)$.
Quantile transform§ 1.3 →
$Q(u)=\inf\{x:F(x)\ge u\}$; $\ Q(u)\le x\iff u\le F(x)$; $\ U\sim U(0,1)\Rightarrow Q(U)\sim F$; $\ F$ continuous $\Rightarrow F(X)\sim U(0,1)$.
Density of g(X)§ 1.3 →
$f_Y(y)=\sum_jf_X(h_j(y))\,|h_j'(y)|$, one term per preimage $x=h_j(y)$ ($g$ strictly monotone and $C^1$ on each piece).
Tail formula§ 1.3 →
$X\ge0$: $E[X]=\int_0^\infty P(X\gt x)\,dx$, $\ E[X^r]=\int_0^\infty rx^{r-1}P(X\gt x)\,dx$; integer $X\ge0$: $E[X]=\sum_{n\ge1}P(X\ge n)$; general: $E[X]=\int_0^\infty(1-F)-\int_{-\infty}^0F$.
Variance and optimal centres§ 1.3 →
$\operatorname{Var}X=E[X^2]-(EX)^2$; $\ \operatorname{Var}(aX+b)=a^2\operatorname{Var}X$; $\ E(X-c)^2=\operatorname{Var}X+(EX-c)^2$; $\ E|X-c|$ is minimised at a median.
Generating functions§ 1.3 →
$G(s)=E[s^X]$: $p_k=\frac{G^{(k)}(0)}{k!}$, $EX=G'(1)$, $\operatorname{Var}X=G''(1)+G'(1)-G'(1)^2$. $\ M(t)=E[e^{tX}]$: $E[X^r]=M^{(r)}(0)$, $K=\log M$: $K'(0)=EX$, $K''(0)=\operatorname{Var}X$.
Joint laws and independence§ 1.4 →
$P(a\lt X\le b,c\lt Y\le d)=F(b,d)-F(a,d)-F(b,c)+F(a,c)$; $\ f_X=\int f\,dy$; independent $\iff f=f_Xf_Y\iff f=g(x)h(y)$ on all of $\mathbb{R}^2$.
Covariance and correlation§ 1.4 →
$\operatorname{Cov}=E[XY]-EXEY$; $\ \operatorname{Var}(aX+bY)=a^2\operatorname{Var}X+2ab\operatorname{Cov}+b^2\operatorname{Var}Y$; $\ |\rho|\le1$; best linear predictor slope $\operatorname{Cov}/\operatorname{Var}X$, error $\sigma_Y^2(1-\rho^2)$.
Conditioning§ 1.4 →
$E[E[X\mid Y]]=EX$; $\ \operatorname{Var}X=E[\operatorname{Var}(X\mid Y)]+\operatorname{Var}(E[X\mid Y])$; random sum: $E[S_N]=E[N]\mu$, $\ \operatorname{Var}(S_N)=E[N]\sigma^2+\mu^2\operatorname{Var}N$.
Jacobian and convolution§ 1.4 →
$f_{U,V}(u,v)=f(x(u,v),y(u,v))\,|J|$, $J=\frac{\partial(x,y)}{\partial(u,v)}$; independent: $f_{X+Y}(z)=\int f_X(x)f_Y(z-x)\,dx$, $\ M_{X+Y}=M_XM_Y$.
Maxima and minima§ 1.4 →
Independent: $F_{\max}=\prod F_i$, $\ P(\min\gt x)=\prod(1-F_i)$; iid: $f_{\max}=nF^{n-1}f$, $\ f_{\min}=n(1-F)^{n-1}f$; exponentials: $\min\sim\mathrm{Exp}(\sum\lambda_i)$, $P(X_j\text{ first})=\lambda_j/\sum\lambda_i$.
Bivariate normal§ 1.4 →
$Y\mid X=x\sim N\left(\mu_2+\rho\frac{\sigma_2}{\sigma_1}(x-\mu_1),\ \sigma_2^2(1-\rho^2)\right)$; $\ aX+bY$ normal; $\ \rho=0\iff$ independent (jointly normal only).

Chapter 2 · Distributions and Limit Theorems

Discrete laws: mean, variance, pgf§ 2.1 →
$\operatorname{Bin}(n,p)$: $np$, $npq$, $(q+ps)^n$. $\ \operatorname{Poi}(\lambda)$: $\lambda$, $\lambda$, $e^{\lambda(s-1)}$. $\ \operatorname{Geo}(p)$ on $\{1,2,\dots\}$: $\tfrac1p$, $\tfrac q{p^2}$, $\tfrac{ps}{1-qs}$. $\ \operatorname{NB}(r,p)$ (failures): $\tfrac{rq}p$, $\tfrac{rq}{p^2}$, $\left(\tfrac{p}{1-qs}\right)^r$.
Hypergeometric law§ 2.1 →
$P(X=k)=\binom Kk\binom{N-K}{n-k}\big/\binom Nn$; with $p=\frac KN$: $E[X]=np$, $\operatorname{Var}(X)=np(1-p)\frac{N-n}{N-1}$.
Poisson limit, thinning, conditioning§ 2.1 →
$np_n\to\lambda$: $\operatorname{Bin}(n,p_n)\to\operatorname{Poi}(\lambda)$, error $\le\sum p_i^2$ (Le Cam). Thinning $\operatorname{Poi}(\lambda)$ with prob. $p$: independent $\operatorname{Poi}(\lambda p)$, $\operatorname{Poi}(\lambda q)$. $X\mid X+Y=n\sim\operatorname{Bin}\left(n,\tfrac{\lambda}{\lambda+\mu}\right)$.
Gamma and beta laws§ 2.1 →
$\Gamma(\alpha+1)=\alpha\Gamma(\alpha)$, $\Gamma(\tfrac12)=\sqrt\pi$, $B(a,b)=\frac{\Gamma(a)\Gamma(b)}{\Gamma(a+b)}$. $\operatorname{Gamma}(\alpha,\lambda)$: $E[X^r]=\frac{\Gamma(\alpha+r)}{\Gamma(\alpha)\lambda^r}$, $M(t)=\left(\frac{\lambda}{\lambda-t}\right)^\alpha$. $\operatorname{Beta}(a,b)$: mean $\frac a{a+b}$, variance $\frac{ab}{(a+b)^2(a+b+1)}$.
Gamma–Poisson and beta–binomial identities§ 2.1 →
$P(\operatorname{Gamma}(n,\lambda)\le t)=P(\operatorname{Poi}(\lambda t)\ge n)$. $\ P(\operatorname{Bin}(n,p)\ge k)=\frac{n!}{(k-1)!(n-k)!}\int_0^pt^{k-1}(1-t)^{n-k}dt$. $\ X,Y$ independent gamma, common rate: $\frac{X}{X+Y}\sim\operatorname{Beta}(a,b)$.
Normal family and relatives§ 2.1 →
$N(\mu,\sigma^2)$: $M(t)=e^{\mu t+\sigma^2t^2/2}$, $E[Z^{2k}]=1\cdot3\cdots(2k-1)$. $\operatorname{LN}$: $E[X^r]=e^{r\mu+r^2\sigma^2/2}$. $Z_1/Z_2\sim C(0,1)$. $\operatorname{Laplace}(\theta,b)$: variance $2b^2$. $\operatorname{Weib}(k,\lambda)$: $E[X^r]=\lambda^r\Gamma(1+\tfrac rk)$, hazard $\tfrac k\lambda(\tfrac x\lambda)^{k-1}$.
Characteristic functions: key properties§ 2.2 →
$\varphi_X(t)=E[e^{itX}]$, $|\varphi|\le1$, $\varphi(-t)=\overline{\varphi(t)}$, $\varphi_{aX+b}(t)=e^{ibt}\varphi_X(at)$; $X\perp Y\Rightarrow\varphi_{X+Y}=\varphi_X\varphi_Y$; if $E|X|^n\lt\infty$, $\varphi^{(k)}(0)=i^kE[X^k]$ ($k\le n$); $\varphi$ real $\iff X\overset{d}{=}-X$.
Characteristic functions of standard laws§ 2.2 →
$\operatorname{Bin}$: $(q+pe^{it})^n$; $\operatorname{Poi}$: $e^{\lambda(e^{it}-1)}$; $U(-a,a)$: $\frac{\sin at}{at}$; $\operatorname{Gamma}(\alpha,\lambda)$: $(1-it/\lambda)^{-\alpha}$; $N(\mu,\sigma^2)$: $e^{i\mu t-\sigma^2t^2/2}$; $\operatorname{Laplace}(\theta,b)$: $\frac{e^{i\theta t}}{1+b^2t^2}$; $C(\theta,\gamma)$: $e^{i\theta t-\gamma|t|}$.
Inversion and Lévy's continuity theorem§ 2.2 →
If $\int|\varphi|\lt\infty$: $f(x)=\frac1{2\pi}\int e^{-itx}\varphi(t)\,dt$. $P(X=a)=\lim\frac1{2T}\int_{-T}^Te^{-ita}\varphi(t)dt$. $X_n\xrightarrow{d}X\iff\varphi_n\to\varphi_X$ pointwise (limit continuous at $0$).
Tail inequalities§ 2.2 →
Markov ($X\ge0$): $P(X\ge a)\le\frac{E X}{a}$. Chebyshev: $P(|X-\mu|\ge k\sigma)\le\frac1{k^2}$. Cantelli: $P(X-\mu\ge\lambda)\le\frac{\sigma^2}{\sigma^2+\lambda^2}$. Chernoff: $P(X\ge a)\le\inf_{t\ge0}e^{-ta}M_X(t)$.
Moment inequalities§ 2.2 →
Jensen ($g$ convex): $g(EX)\le E g(X)$. Lyapunov ($0\lt r\lt s$): $(E|X|^r)^{1/r}\le(E|X|^s)^{1/s}$. Cauchy–Schwarz: $(E XY)^2\le EX^2\,EY^2$. Hölder ($\frac1p+\frac1q=1$): $E|XY|\le\|X\|_p\|Y\|_q$.
Modes of convergence: implications§ 2.3 →
a.s. $\Rightarrow$ P $\Rightarrow$ d; $L^s\Rightarrow L^r$ ($s\gt r$) $\Rightarrow$ P; d to a constant $\Rightarrow$ P; P $\Rightarrow$ a.s. along a subsequence; $\sum_nP(|X_n-X|\gt\varepsilon)\lt\infty\ \forall\varepsilon\Rightarrow$ a.s.
Laws of large numbers§ 2.3 →
Chebyshev: $\operatorname{Var}(S_n)/n^2\to0\Rightarrow(S_n-ES_n)/n\xrightarrow{P}0$. Khintchine (iid, $E|X|\lt\infty$): $\bar X_n\xrightarrow{P}\mu$. Kolmogorov (iid): $\bar X_n\xrightarrow{a.s.}\mu\iff E|X_1|\lt\infty$.
Central limit theorem and normal approximation§ 2.3 →
iid, $0\lt\sigma^2\lt\infty$: $\frac{\sqrt n(\bar X_n-\mu)}{\sigma}\xrightarrow{d}N(0,1)$; error $\le\frac{0.4748\,E|X-\mu|^3}{\sigma^3\sqrt n}$. Binomial: $P(X\le b)\approx\Phi\left(\frac{b+\frac12-np}{\sqrt{npq}}\right)$.
Slutsky and the delta method§ 2.3 →
$X_n\xrightarrow{d}X$, $Y_n\xrightarrow{P}c$: $X_n+Y_n\xrightarrow{d}X+c$, $Y_nX_n\xrightarrow{d}cX$. $\sqrt n(T_n-\theta)\xrightarrow{d}N(0,\tau^2)\Rightarrow\sqrt n(g(T_n)-g(\theta))\xrightarrow{d}N(0,g'(\theta)^2\tau^2)$; if $g'(\theta)=0$: $n(g(T_n)-g(\theta))\xrightarrow{d}\tfrac12g''(\theta)\tau^2\chi^2_1$.

Chapter 3 · Markov Chains and Stochastic Processes

Path probabilities and Chapman–Kolmogorov§ 3.1 →
$P(X_0=i_0,\dots,X_n=i_n)=\lambda_{i_0}p_{i_0i_1}\cdots p_{i_{n-1}i_n}$; $\ p_{ij}^{(m+n)}=\sum_kp_{ik}^{(m)}p_{kj}^{(n)}$, so $P^{(n)}=P^n$ and $X_n\sim\lambda P^n$.
Two-state chain§ 3.1 →
$P=\begin{pmatrix}1-a&a\\b&1-b\end{pmatrix}$, $a+b\gt0$: $\ p_{11}^{(n)}=\frac{b}{a+b}+\frac{a}{a+b}(1-a-b)^n$; rows of $P^n\to\left(\frac b{a+b},\frac a{a+b}\right)$ if $|1-a-b|\lt1$.
Recurrence criterion§ 3.1 →
$P_i(N_i\ge k)=f_{ii}^k$; $\ \sum_{n\ge1}p_{ii}^{(n)}=\frac{f_{ii}}{1-f_{ii}}$; recurrent $\iff\sum_np_{ii}^{(n)}=\infty$. $\ P_{ii}(s)=\frac1{1-F_{ii}(s)}$, $\ p_{ij}^{(n)}=\sum_{k=1}^nf_{ij}^{(k)}p_{jj}^{(n-k)}$.
Simple random walk on ℤ§ 3.1 →
$p_{00}^{(2n)}=\binom{2n}{n}(pq)^n$, $\ F_{00}(s)=1-\sqrt{1-4pqs^2}$, $\ f_{00}=1-|p-q|$; recurrent iff $p=\frac12$. Symmetric walk on $\mathbb{Z}^d$: recurrent iff $d\le2$.
Absorption: first-step analysis§ 3.1 →
$h_i=\sum_jp_{ij}h_j$ ($i\notin A$), $h=1$ on $A$ (minimal solution); $k_i=1+\sum_{j\notin A}p_{ij}k_j$. Finite: $N=(I-Q)^{-1}$, absorption probabilities $NR$, times $N\mathbf 1$.
Gambler's ruin: duration§ 3.1 →
$D_i=i(N-i)$ ($p=\frac12$); $\ D_i=\frac{i}{q-p}-\frac{N}{q-p}\cdot\frac{1-\rho^i}{1-\rho^N}$, $\rho=\frac qp$. Infinite opponent: $P(\text{ruin})=1$ if $p\le q$, $\rho^i$ if $p\gt q$.
Stationary distribution and return times§ 3.2 →
$\pi P=\pi$, $\pi_i\ge0$, $\sum\pi_i=1$. Irreducible: exists $\iff$ positive recurrent; then unique and $\pi_i=\dfrac1{\mu_i}$, $\ \mu_i=E_i(T_i)$. Finite irreducible chains are always positive recurrent.
Limit theorems for Markov chains§ 3.2 →
Irreducible, aperiodic, positive recurrent: $p_{ij}^{(n)}\to\pi_j$. Period $d$: $p_{jj}^{(nd)}\to d\pi_j$. Always: $\frac1n\sum_{k\lt n}\mathbf 1\{X_k=j\}\to\frac1{\mu_j}$ a.s. Transient or null: $p_{ij}^{(n)}\to0$.
Shortcuts for finding π§ 3.2 →
Doubly stochastic (finite, $s$ states) $\iff\pi=\left(\frac1s,\dots,\frac1s\right)$. Detailed balance $\pi_ip_{ij}=\pi_jp_{ji}\Rightarrow\pi P=\pi$. Reversed chain $\hat p_{ij}=\pi_jp_{ji}/\pi_i$.
Random walk on a graph§ 3.2 →
$\pi_i=\dfrac{\deg i}{2|E|}$, $\ \mu_i=\dfrac{2|E|}{\deg i}$; weighted: $\pi_i=\dfrac{w_i}{\sum_kw_k}$.
Birth–death chains on {0, 1, 2, …}§ 3.2 →
$\theta_n=\frac{p_0\cdots p_{n-1}}{q_1\cdots q_n}$, $\rho_n=\frac{q_1\cdots q_n}{p_1\cdots p_n}$. Positive recurrent iff $\sum\theta_n\lt\infty$, $\pi_n=\theta_n/\sum\theta_k$; recurrent iff $\sum\rho_n=\infty$. Reflecting walk: $\pi_n=(1-\rho)\rho^n$, $\rho=p/q\lt1$.
Poisson process of rate λ§ 3.3 →
$N(t+s)-N(s)\sim\operatorname{Poi}(\lambda t)$, independent over disjoint intervals; gaps iid $\operatorname{Exp}(\lambda)$; $S_n\sim\operatorname{Gamma}(n,\lambda)$, $P(S_n\le t)=P(N(t)\ge n)$; $\operatorname{Cov}(N(s),N(t))=\lambda\min(s,t)$.
Arrivals given N(t) = n§ 3.3 →
$(S_1,\dots,S_n)\mid N(t)=n\ \sim$ order statistics of $n$ iid $U(0,t)$; $\ N(s)\mid N(t)=n\sim\operatorname{Bin}(n,s/t)$ for $s\lt t$.
Thinning, superposition, compound Poisson§ 3.3 →
Independent $\mathrm{PP}(\lambda_i)$ add to $\mathrm{PP}(\sum\lambda_i)$; independent thinning with probability $p$ gives independent $\mathrm{PP}(\lambda p)$, $\mathrm{PP}(\lambda(1-p))$. $X(t)=\sum_{i\le N(t)}Y_i$: $E=\lambda tE[Y]$, $\operatorname{Var}=\lambda tE[Y^2]$, $M_{X(t)}(u)=e^{\lambda t(M_Y(u)-1)}$.
Continuous-time Markov chains§ 3.3 →
$P'(t)=QP(t)=P(t)Q$, $P(0)=I$, $P(t)=e^{tQ}$ (finite $S$); holding time in $i$ is $\operatorname{Exp}(q_i)$, jump $i\to j$ with probability $q_{ij}/q_i$. Steady state: $\pi Q=0$; jump chain $\nu_j\propto\pi_jq_j$; $\pi_i=1/(q_im_i)$.
Birth–death and Yule processes§ 3.3 →
$\lambda_n\pi_n=\mu_{n+1}\pi_{n+1}$, $\ \pi_n\propto\theta_n=\frac{\lambda_0\cdots\lambda_{n-1}}{\mu_1\cdots\mu_n}$ (need $\sum\theta_n\lt\infty$ and $\sum\frac1{\lambda_n\theta_n}=\infty$). Yule from $1$: $P(X(t)=n)=e^{-\lambda t}(1-e^{-\lambda t})^{n-1}$, $E X(t)=e^{\lambda t}$. Pure birth explodes iff $\sum1/\lambda_n\lt\infty$.

Chapter 4 · Sampling Distributions and Estimation

Moments of $\bar X$ and $S^2$ (any population)§ 4.1 →
$E\bar X=\mu$, $\operatorname{Var}\bar X=\frac{\sigma^2}{n}$; $ES^2=\sigma^2$; $\operatorname{Cov}(\bar X,S^2)=\frac{\mu_3}{n}$; $\operatorname{Var}(S^2)=\frac{\mu_4}{n}-\frac{(n-3)\sigma^4}{n(n-1)}$ ($=\frac{2\sigma^4}{n-1}$ if normal).
Normal samples§ 4.1 →
$\bar X\sim N(\mu,\frac{\sigma^2}{n})$, $\frac{(n-1)S^2}{\sigma^2}\sim\chi^2_{n-1}$, $\bar X\perp S^2$; $\frac{\sqrt n(\bar X-\mu)}{S}\sim t_{n-1}$; $\frac{S_1^2/\sigma_1^2}{S_2^2/\sigma_2^2}\sim F_{n_1-1,n_2-1}$.
$t$ and $F$ laws§ 4.1 →
$t_k=\frac{Z}{\sqrt{V/k}}$: $\operatorname{Var}=\frac{k}{k-2}$ ($k\gt2$), $t_1$ = Cauchy, $t_k^2=F_{1,k}$. $F_{m,n}=\frac{U/m}{V/n}$: $EF=\frac{n}{n-2}$, $\frac1F\sim F_{n,m}$, $F_{m,n;1-\alpha}=\frac{1}{F_{n,m;\alpha}}$, $\frac{mF}{n+mF}\sim\operatorname{Beta}(\frac m2,\frac n2)$.
Order statistics§ 4.1 →
$f_{(r)}=\frac{n!}{(r-1)!(n-r)!}F^{r-1}(1-F)^{n-r}f$; $f_{(1),(n)}(x,y)=n(n-1)[F(y)-F(x)]^{n-2}f(x)f(y)$; range: $f_R(w)=n(n-1)\int f(x)f(x+w)[F(x+w)-F(x)]^{n-2}dx$.
Uniform and exponential order statistics§ 4.1 →
$U_{(r)}\sim\operatorname{Beta}(r,n-r+1)$, $EU_{(r)}=\frac{r}{n+1}$; $R\sim\operatorname{Beta}(n-1,2)$. $\operatorname{Exp}(\lambda)$: $(n-i+1)(X_{(i)}-X_{(i-1)})$ iid $\operatorname{Exp}(\lambda)$, $EX_{(r)}=\frac1\lambda\sum_{i=1}^r\frac{1}{n-i+1}$.
Asymptotic laws§ 4.1 →
$\sqrt n(S^2-\sigma^2)\xrightarrow{d}N(0,\mu_4-\sigma^4)$; $\sqrt n(X_{(\lceil np\rceil)}-\xi_p)\xrightarrow{d}N\left(0,\frac{p(1-p)}{f(\xi_p)^2}\right)$ if $f(\xi_p)\gt0$; median: variance $\frac{1}{4f(m)^2n}$.
MSE and consistency§ 4.2 →
$\operatorname{MSE}(T)=\operatorname{Var}(T)+b(\theta)^2$; bias $\to0$ and variance $\to0$ $\Rightarrow$ consistent.
Sufficiency and completeness§ 4.2 →
$T$ sufficient $\iff f(x;\theta)=g(T(x);\theta)h(x)$. Minimal: $\frac{f(x;\theta)}{f(y;\theta)}$ free of $\theta\iff T(x)=T(y)$. Full-rank exponential family $h(x)e^{\sum\eta_j(\theta)T_j(x)-A(\theta)}$: $(\sum_iT_j(X_i))_j$ complete sufficient.
Rao–Blackwell, Lehmann–Scheffé, Basu§ 4.2 →
$W$ unbiased, $T$ sufficient: $E[W\mid T]$ unbiased with $\operatorname{Var}\le\operatorname{Var}W$. $T$ complete sufficient, $g(T)$ unbiased $\Rightarrow$ $g(T)$ is the unique UMVUE. Complete sufficient $\perp$ ancillary.
Fisher information and the Cramér–Rao bound§ 4.2 →
$I(\theta)=E[(\partial_\theta\log f)^2]=-E[\partial^2_\theta\log f]$; regular model, $ET=\psi(\theta)$: $\operatorname{Var}(T)\ge\frac{\psi'(\theta)^2}{nI(\theta)}$ (not valid if the support depends on $\theta$).
The MLE§ 4.2 →
$\widehat{\tau(\theta)}=\tau(\hat\theta)$; regular: $\sqrt n(\hat\theta-\theta)\xrightarrow{d}N\left(0,\frac{1}{I(\theta)}\right)$.
UMVUEs to remember§ 4.2 →
Poisson: $e^{-\lambda}\to(1-\frac1n)^T$, $\lambda^2\to\frac{T(T-1)}{n^2}$. Bernoulli: $p^2\to\frac{T(T-1)}{n(n-1)}$. $U(0,\theta)$: $\theta\to\frac{n+1}{n}X_{(n)}$. $\operatorname{Exp}(\lambda)$: $\lambda\to\frac{n-1}{T}$ ($T=\sum X_i$).
Normal sample: intervals for μ and σ²§ 4.3 →
iid $N(\mu,\sigma^2)$: $\bar x\pm z_{\alpha/2}\frac{\sigma}{\sqrt n}$ ($\sigma$ known); $\bar x\pm t_{n-1,\alpha/2}\frac{s}{\sqrt n}$ ($\sigma$ unknown); $\sigma^2\in\left[\frac{(n-1)s^2}{\chi^2_{n-1,\alpha/2}},\frac{(n-1)s^2}{\chi^2_{n-1,1-\alpha/2}}\right]$ (upper $\alpha$ points).
Two normal samples§ 4.3 →
Equal variances: $\bar x-\bar y\pm t_{m+n-2,\alpha/2}\,s_p\sqrt{\frac1m+\frac1n}$, $s_p^2=\frac{(m-1)s_1^2+(n-1)s_2^2}{m+n-2}$. $\frac{\sigma_1^2}{\sigma_2^2}\in\left[\frac{s_1^2/s_2^2}{F_{m-1,n-1,\alpha/2}},\frac{s_1^2}{s_2^2}F_{n-1,m-1,\alpha/2}\right]$. Paired: one-sample $t$ on the differences.
Exponential and uniform pivots§ 4.3 →
iid Exp(mean $\theta$): $2\sum X_i/\theta\sim\chi^2_{2n}$. iid $U(0,\theta)$: $X_{(n)}/\theta$ has cdf $t^n$; shortest $1-\alpha$ interval $\left[X_{(n)},\ X_{(n)}\alpha^{-1/n}\right]$. Continuous $F$: $-2\sum\log F(X_i;\theta)\sim\chi^2_{2n}$.
Intervals for a proportion§ 4.3 →
Wald $\hat p\pm z\sqrt{\frac{\hat p\hat q}{n}}$; Wilson $\frac{\hat p+\frac{z^2}{2n}\pm z\sqrt{\frac{\hat p\hat q}{n}+\frac{z^2}{4n^2}}}{1+z^2/n}$; Clopper–Pearson $\left[B_{\alpha/2}(x,n-x+1),\ B_{1-\alpha/2}(x+1,n-x)\right]$ (beta quantiles).
Conjugate updating§ 4.3 →
Beta$(a,b)$ + $x$ of $n$ → Beta$(a+x,b+n-x)$, mean $\frac{a+x}{a+b+n}$. Gamma$(\alpha,\beta)$ + Poisson data → Gamma$(\alpha+\sum x_i,\beta+n)$. $N(\mu_0,\tau_0^2)$ + $N(\theta,\sigma^2)$ data → $N(\mu_n,\tau_n^2)$, $\frac1{\tau_n^2}=\frac1{\tau_0^2}+\frac n{\sigma^2}$, $\mu_n=\tau_n^2\left(\frac{\mu_0}{\tau_0^2}+\frac{n\bar x}{\sigma^2}\right)$.
Bayes estimators and credible sets§ 4.3 →
Squared loss → posterior mean; absolute loss → posterior median. HPD $\{\theta:\pi(\theta\mid x)\ge k\}$ is the shortest $1-\alpha$ credible set.

Chapter 5 · Testing of Hypotheses

Size, power and p-value§ 5.1 →
$\beta_\phi(\theta)=E_\theta\phi(X)$; size $=\sup_{\theta\in\Theta_0}\beta_\phi(\theta)$; power at $\theta\in\Theta_1$ is $\beta_\phi(\theta)$. $p(x)=\sup_{\Theta_0}P_\theta(T\ge T(x))$; reject iff $p\le\alpha$; $P_{H_0}(p\le u)\le u$.
Neyman–Pearson lemma§ 5.1 →
$\phi=1$ if $f_1\gt kf_0$, $\gamma$ if $f_1=kf_0$, $0$ if $f_1\lt kf_0$, with $E_0\phi=\alpha$, is most powerful at level $\alpha$; $\gamma=\frac{\alpha-P_0(R\gt k)}{P_0(R=k)}$, $R=f_1/f_0$.
Karlin–Rubin (UMP one-sided)§ 5.1 →
MLR in $T$ (e.g. $h(x)e^{c(\theta)T(x)-A(\theta)}$, $c\uparrow$): for $H_0:\theta\le\theta_0$ vs $\theta\gt\theta_0$, reject when $T\gt c$ (randomise at $T=c$), $E_{\theta_0}\phi=\alpha$. Power is non-decreasing in $\theta$.
One-sided z-test: power and n§ 5.1 →
$\beta(\mu)=\Phi\left(\frac{\sqrt n(\mu-\mu_0)}{\sigma}-z_\alpha\right)$; $n=\left(\frac{(z_\alpha+z_\beta)\sigma}{\mu_1-\mu_0}\right)^2$.
Likelihood ratio test and Wilks§ 5.1 →
$\Lambda=\frac{\sup_{\Theta_0}L}{\sup_\Theta L}$; regular models: $-2\log\Lambda\xrightarrow d\chi^2_{r}$, $r=\dim\Theta-\dim\Theta_0$. Normal mean, $\sigma$ unknown: $\Lambda=\left(1+\frac{t^2}{n-1}\right)^{-n/2}$ (the $t$-test).
Large-sample z-tests§ 5.2 →
$\frac{\bar X-\mu_0}{S/\sqrt n}$; $\frac{\hat p-p_0}{\sqrt{p_0q_0/n}}$; $\frac{\hat p_1-\hat p_2}{\sqrt{\hat p\hat q(1/m+1/n)}}$ with pooled $\hat p$; $\frac{\bar X-\lambda_0}{\sqrt{\lambda_0/n}}$; $\sqrt{n-3}\,(\operatorname{artanh}r-\operatorname{artanh}\rho_0)$. All $\approx N(0,1)$ under $H_0$.
Chi-square goodness of fit§ 5.2 →
$X^2=\sum_{j=1}^k\frac{(N_j-E_j)^2}{E_j}\approx\chi^2_{k-1-s}$ ($s$ parameters estimated from the grouped counts; $k$ after pooling so that $E_j\ge5$); $E[X^2]=k-1$ under a simple $H_0$; $G^2=2\sum N_j\log\frac{N_j}{E_j}$.
Contingency tables§ 5.2 →
$\hat E_{ij}=\frac{R_iC_j}{n}$, $X^2\approx\chi^2_{(r-1)(c-1)}$ (independence or homogeneity).
The 2×2 table§ 5.2 →
$X^2=\frac{n(ad-bc)^2}{R_1R_2C_1C_2}=Z^2_{\text{pooled}}$; Yates: $|ad-bc|\to|ad-bc|-\frac n2$. Fisher: $P(A=a)=\binom{R_1}{a}\binom{R_2}{C_1-a}\big/\binom n{C_1}$. Odds ratio $\frac{ad}{bc}$, $\operatorname{SE}(\log)=\sqrt{\frac1a+\frac1b+\frac1c+\frac1d}$.
McNemar and dispersion§ 5.2 →
Paired: $\frac{(b-c)^2}{b+c}\approx\chi^2_1$. Poisson: $D=\frac{\sum(X_i-\bar X)^2}{\bar X}\approx\chi^2_{n-1}$.
Sign test and median interval§ 5.3 →
$S=\#\{X_i\gt\theta_0\}\sim\operatorname{Bin}(n,\frac12)$ under $H_0$ (any continuous $F$, zeros dropped). $P\left(X_{(k)}\le m\le X_{(n-k+1)}\right)=1-2\sum_{j\lt k}\binom nj2^{-n}$.
Wilcoxon signed-rank statistic§ 5.3 →
Symmetric continuous null: $T^+\overset{d}{=}\sum_{k=1}^nkB_k$, $B_k$ iid Bernoulli$(\frac12)$; $E[T^+]=\frac{n(n+1)}4$, $\operatorname{Var}(T^+)=\frac{n(n+1)(2n+1)}{24}$; $T^+=\#\{i\le j:Z_i+Z_j\gt0\}$.
Mann–Whitney–Wilcoxon§ 5.3 →
$U=\#\{X_i\lt Y_j\}$, $W=U+\frac{n(n+1)}2$. Under $F=G$: $E[U]=\frac{mn}2$, $E[W]=\frac{n(m+n+1)}2$, $\operatorname{Var}(U)=\operatorname{Var}(W)=\frac{mn(m+n+1)}{12}$. In general $E[U]=mn\,P(X\lt Y)$.
Number of runs§ 5.3 →
$m$ and $n$ symbols, $N=m+n$: $E[R]=1+\frac{2mn}N$, $\operatorname{Var}(R)=\frac{2mn(2mn-N)}{N^2(N-1)}$.
Kolmogorov–Smirnov§ 5.3 →
$D_n=\max_i\max\left(\frac in-U_{(i)},\,U_{(i)}-\frac{i-1}n\right)$, $U_{(i)}=F_0(X_{(i)})$, $F_0$ continuous and fully specified. $\sqrt n\,D_n\to K$; $5\%$ point $\approx1.3581/\sqrt n$. Two samples: $\sqrt{mn/N}\,D_{m,n}\to K$.
Rank correlations§ 5.3 →
$r_s=1-\frac{6\sum d_i^2}{n(n^2-1)}$, $\tau=\frac{C-D}{\binom n2}$. Under independence: $\operatorname{Var}(r_s)=\frac1{n-1}$, $\operatorname{Var}(\tau)=\frac{2(2n+5)}{9n(n-1)}$. Bivariate normal: $\tau=\frac2\pi\arcsin\rho$, $\rho_S=\frac6\pi\arcsin\frac\rho2$.

Chapter 6 · Linear Models and Regression

Least squares and the hat matrix§ 6.1 →
$X^{\mathsf T}X\hat\beta=X^{\mathsf T}y$; full rank: $\hat\beta=(X^{\mathsf T}X)^{-1}X^{\mathsf T}y$. $H=X(X^{\mathsf T}X)^-X^{\mathsf T}$ is symmetric, idempotent, $\operatorname{tr}H=\operatorname{rank}X=r$; $\hat y=Hy$, $e=(I-H)y$, $X^{\mathsf T}e=0$.
Estimability and Gauss–Markov§ 6.1 →
$\lambda^{\mathsf T}\beta$ estimable $\iff\lambda\in\mathcal C(X^{\mathsf T})\iff\lambda^{\mathsf T}(X^{\mathsf T}X)^-X^{\mathsf T}X=\lambda^{\mathsf T}$. If $E\varepsilon=0$, $\operatorname{Cov}\varepsilon=\sigma^2I$: BLUE $=\lambda^{\mathsf T}\hat\beta$, $\operatorname{Var}=\sigma^2\lambda^{\mathsf T}(X^{\mathsf T}X)^-\lambda$.
Aitken (GLS)§ 6.1 →
$\operatorname{Cov}\varepsilon=\sigma^2V$ ($V\succ0$ known, full rank $X$): $\tilde\beta=(X^{\mathsf T}V^{-1}X)^{-1}X^{\mathsf T}V^{-1}y$, $\operatorname{Cov}\tilde\beta=\sigma^2(X^{\mathsf T}V^{-1}X)^{-1}$.
σ² and normal theory§ 6.1 →
$\hat\sigma^2=\frac{\operatorname{RSS}}{n-r}$ (unbiased). Normal errors: $\frac{\operatorname{RSS}}{\sigma^2}\sim\chi^2_{n-r}$, independent of $\hat\beta$; $\lambda^{\mathsf T}\hat\beta\pm t_{n-r,\alpha/2}\,\hat\sigma\sqrt{\lambda^{\mathsf T}(X^{\mathsf T}X)^-\lambda}$.
F-test of Λᵀβ = d§ 6.1 →
$F=\frac{(\operatorname{RSS}_0-\operatorname{RSS})/q}{\operatorname{RSS}/(n-r)}\sim F_{q,n-r}$ under $H_0$ (testable, rank $q$); $\operatorname{RSS}_0-\operatorname{RSS}=(\Lambda^{\mathsf T}\hat\beta-d)^{\mathsf T}(\Lambda^{\mathsf T}G\Lambda)^{-1}(\Lambda^{\mathsf T}\hat\beta-d)$, $G=(X^{\mathsf T}X)^-$.
Straight line§ 6.1 →
$\operatorname{Var}\hat\beta_1=\frac{\sigma^2}{S_{xx}}$, $\operatorname{Var}\hat\beta_0=\sigma^2\left(\frac1n+\frac{\bar x^2}{S_{xx}}\right)$, $\operatorname{Cov}=-\frac{\bar x\sigma^2}{S_{xx}}$.
One-way ANOVA§ 6.2 →
$\text{SST}=\text{SSB}+\text{SSW}$, $\text{SSB}=\sum n_i(\bar y_i-\bar y)^2$; $F=\frac{\text{SSB}/(k-1)}{\text{SSW}/(N-k)}\sim F_{k-1,N-k}$ under equal means; $E[\text{MSB}]=\sigma^2+\frac{\sum n_i(\mu_i-\bar\mu)^2}{k-1}$.
Contrasts and multiple comparisons§ 6.2 →
$\hat\psi=\sum c_i\bar y_i$ ($\sum c_i=0$), $\operatorname{Var}=\sigma^2\sum\frac{c_i^2}{n_i}$, $\text{SS}=\hat\psi^2/\sum\frac{c_i^2}{n_i}$. Scheffé multiplier $\sqrt{(k-1)F_{k-1,N-k,\alpha}}$; Bonferroni $t_{N-k,\alpha/(2m)}$; Tukey $q_{k,N-k,\alpha}/\sqrt2$ per pair.
Two-way layout, n per cell§ 6.2 →
df: A $a-1$, B $b-1$, AB $(a-1)(b-1)$, error $ab(n-1)$. $\text{SSAB}=n\sum(\bar y_{ij\cdot}-\bar y_{i\cdot\cdot}-\bar y_{\cdot j\cdot}+\bar y)^2$. With $n=1$ the AB line is the error line.
Random and mixed effects§ 6.2 →
One-way: $E[\text{MSA}]=\sigma^2+n\sigma_a^2$, $\hat\sigma_a^2=\frac{\text{MSA}-\text{MSE}}{n}$, $\rho=\frac{\sigma_a^2}{\sigma_a^2+\sigma^2}$. Two-way random: A, B against MSAB; mixed: fixed A against MSAB; AB against MSE.
ANCOVA (one covariate)§ 6.2 →
$\hat\beta=\frac{E_{xy}}{E_{xx}}$; adjusted means $\bar y_i-\hat\beta(\bar x_i-\bar x)$; $\text{SSE}=E_{yy}-\frac{E_{xy}^2}{E_{xx}}$ on $N-k-1$ df; $F=\frac{(\text{SSE}_0-\text{SSE})/(k-1)}{\text{SSE}/(N-k-1)}$, $\text{SSE}_0=T_{yy}-\frac{T_{xy}^2}{T_{xx}}$.
Simple linear regression§ 6.3 →
$\hat\beta_1=\frac{S_{xy}}{S_{xx}}$, $\hat\beta_0=\bar y-\hat\beta_1\bar x$, $s^2=\frac{SSE}{n-2}$, $SSE=S_{yy}-\frac{S_{xy}^2}{S_{xx}}$. Under iid $N(0,\sigma^2)$ errors: $\frac{\hat\beta_1-\beta_1}{s/\sqrt{S_{xx}}}\sim t_{n-2}$, $\operatorname{Var}(\hat\beta_0)=\sigma^2\big(\frac1n+\frac{\bar x^2}{S_{xx}}\big)$, $\operatorname{Cov}(\hat\beta_0,\hat\beta_1)=-\frac{\bar x\sigma^2}{S_{xx}}$.
Mean response and prediction§ 6.3 →
CI for $\beta_0+\beta_1x_0$: $\hat\mu_0\pm t_{n-2,\alpha/2}\,s\sqrt{\frac1n+\frac{(x_0-\bar x)^2}{S_{xx}}}$. PI for a new $Y_0$: $\hat\mu_0\pm t_{n-2,\alpha/2}\,s\sqrt{1+\frac1n+\frac{(x_0-\bar x)^2}{S_{xx}}}$. Multiple regression: replace the root by $\sqrt{x_0^\top(X^\top X)^{-1}x_0}$ (or $\sqrt{1+\cdots}$) and $n-2$ by $n-p$.
$R^2$, adjusted $R^2$ and $F$§ 6.3 →
$R^2=1-\frac{SSE}{SST}$, $\bar R^2=1-(1-R^2)\frac{n-1}{n-p}$; overall $F=\frac{R^2/k}{(1-R^2)/(n-k-1)}\sim F_{k,n-k-1}$; extra SS $F=\frac{(SSE_R-SSE_F)/q}{SSE_F/(n-p)}\sim F_{q,n-p}$. Adding regressors raises $\bar R^2$ iff this $F\gt1$ (one regressor: $|t|\gt1$).
Regression diagnostics§ 6.3 →
$H=X(X^\top X)^{-1}X^\top$, $\sum h_{ii}=p$, $\frac1n\le h_{ii}\le1$; $r_i=\frac{e_i}{s\sqrt{1-h_{ii}}}$, $t_i=r_i\sqrt{\frac{n-p-1}{n-p-r_i^2}}\sim t_{n-p-1}$; $e_{(i)}=\frac{e_i}{1-h_{ii}}$; $D_i=\frac{r_i^2}{p}\frac{h_{ii}}{1-h_{ii}}$; $\operatorname{Var}(\hat\beta_j)=\frac{\sigma^2}{S_{jj}(1-R_j^2)}$, $\operatorname{VIF}_j=\frac1{1-R_j^2}$.
Logistic regression§ 6.3 →
$\operatorname{logit}\pi=x^\top\beta$; score $X^\top(y-\mu)$, information $X^\top WX$, $W=\operatorname{diag}(m_i\pi_i(1-\pi_i))$; Newton/IRLS $\beta^{\text{new}}=\beta+(X^\top WX)^{-1}X^\top(y-\mu)$. Odds ratio CI $\exp(\hat\beta_j\pm z_{\alpha/2}\operatorname{SE})$; $2\times2$ table: $\hat\beta_1=\log\frac{ad}{bc}$, $\operatorname{SE}=\sqrt{\frac1a+\frac1b+\frac1c+\frac1d}$.

Chapter 7 · Multivariate Analysis

Multivariate normal $N_p(\mu,\Sigma)$§ 7.1 →
$X\sim N_p(\mu,\Sigma)$ iff every $a^{\mathsf T}X$ is normal. $\varphi(t)=e^{it^{\mathsf T}\mu-\frac12t^{\mathsf T}\Sigma t}$, $M(t)=e^{t^{\mathsf T}\mu+\frac12t^{\mathsf T}\Sigma t}$. If $\Sigma\gt0$: $f(x)=(2\pi)^{-p/2}|\Sigma|^{-1/2}e^{-\frac12(x-\mu)^{\mathsf T}\Sigma^{-1}(x-\mu)}$. $AX+b\sim N_q(A\mu+b,A\Sigma A^{\mathsf T})$; jointly normal blocks are independent iff $\Sigma_{12}=0$.
Conditional distributions (Σ₂₂ > 0)§ 7.1 →
$X_1\mid X_2=x_2\sim N\big(\mu_1+\Sigma_{12}\Sigma_{22}^{-1}(x_2-\mu_2),\ \Sigma_{11\cdot2}\big)$, $\Sigma_{11\cdot2}=\Sigma_{11}-\Sigma_{12}\Sigma_{22}^{-1}\Sigma_{21}$. $|\Sigma|=|\Sigma_{22}||\Sigma_{11\cdot2}|$, $(\Sigma^{-1})_{11}=\Sigma_{11\cdot2}^{-1}$.
Mahalanobis distance§ 7.1 →
$D^2=(X-\mu)^{\mathsf T}\Sigma^{-1}(X-\mu)\sim\chi^2_p$ ($\Sigma\gt0$). Ellipsoid $D^2\le\chi^2_{p,\alpha}$ has probability $1-\alpha$, axes along eigenvectors of $\Sigma$, half-lengths $\sqrt{\chi^2_{p,\alpha}\lambda_i}$.
Quadratic forms§ 7.1 →
$Z\sim N_n(\mu,I)$: $E[Z^{\mathsf T}AZ]=\operatorname{tr}A+\mu^{\mathsf T}A\mu$, $\operatorname{Var}=2\operatorname{tr}A^2+4\mu^{\mathsf T}A^2\mu$. $Z^{\mathsf T}AZ\sim\chi^2_r(\mu^{\mathsf T}A\mu)$ iff $A^2=A$, $r=\operatorname{tr}A$ (for $X\sim N(\mu,\Sigma)$: iff $A\Sigma A=A$). Independent: $LX$, $X^{\mathsf T}AX$ if $LA=0$; two forms iff $AB=0$. Cochran: $\sum A_i=I$ and $\sum r_i=n$ ⇒ independent $\chi^2_{r_i}$.
Wishart law and normal samples§ 7.1 →
$W\sim W_p(m,\Sigma)$: $E W=m\Sigma$, $\operatorname{Var}W_{jk}=m(\sigma_{jk}^2+\sigma_{jj}\sigma_{kk})$, $CWC^{\mathsf T}\sim W_q(m,C\Sigma C^{\mathsf T})$, $\frac{a^{\mathsf T}Wa}{a^{\mathsf T}\Sigma a}\sim\chi^2_m$, $\frac{a^{\mathsf T}\Sigma^{-1}a}{a^{\mathsf T}W^{-1}a}\sim\chi^2_{m-p+1}$. Sample: $\bar X\sim N_p(\mu,\Sigma/n)$, $A=(n-1)S\sim W_p(n-1,\Sigma)$, independent; MLE $\hat\Sigma=A/n$.
Partial correlation§ 7.2 →
$\rho_{12\cdot3}=\dfrac{\rho_{12}-\rho_{13}\rho_{23}}{\sqrt{(1-\rho_{13}^2)(1-\rho_{23}^2)}}$ (same recursion for higher orders); $\rho_{ij\cdot\text{rest}}=-\dfrac{\omega_{ij}}{\sqrt{\omega_{ii}\omega_{jj}}}$ with $\Omega=\Sigma^{-1}$. $|\mathcal P|=(1-\rho_{13}^2)(1-\rho_{23}^2)(1-\rho_{12\cdot3}^2)$.
Multiple correlation§ 7.2 →
$R^2_{1\cdot2\cdots p}=\dfrac{\sigma_{12}\Sigma_{22}^{-1}\sigma_{21}}{\sigma_{11}}=1-\dfrac{|\mathcal P|}{|\mathcal P_{22}|}=1-\dfrac1{\sigma_{11}\omega_{11}}$; $R^2_{1\cdot23}=\dfrac{\rho_{12}^2+\rho_{13}^2-2\rho_{12}\rho_{13}\rho_{23}}{1-\rho_{23}^2}$, $1-R^2_{1\cdot23}=(1-\rho_{12}^2)(1-\rho_{13\cdot2}^2)$.
Tests for correlations (normal data)§ 7.2 →
$\rho=0$: $\frac{r\sqrt{n-2}}{\sqrt{1-r^2}}\sim t_{n-2}$; order-$k$ partial: $\frac{r\sqrt{n-2-k}}{\sqrt{1-r^2}}\sim t_{n-2-k}$; $R=0$: $\frac{R^2/(p-1)}{(1-R^2)/(n-p)}\sim F_{p-1,n-p}$; Fisher $z=\operatorname{artanh}r\approx N(\operatorname{artanh}\rho,\frac1{n-3-k})$; independence of $p$ variables: $-(n-1-\frac{2p+5}6)\log|R|\approx\chi^2_{p(p-1)/2}$.
One-sample Hotelling $T^2$§ 7.2 →
$T^2=n(\bar X-\mu_0)^{\mathsf T}S^{-1}(\bar X-\mu_0)$, $\frac{n-p}{p(n-1)}T^2\sim F_{p,n-p}$ under $H_0$; $\Lambda^{2/n}=(1+\frac{T^2}{n-1})^{-1}$; $T^2=\max_at_a^2$. Simultaneous: $a^{\mathsf T}\bar x\pm c\sqrt{a^{\mathsf T}Sa/n}$, $c^2=\frac{p(n-1)}{n-p}F_{p,n-p,\alpha}$.
Two-sample $T^2$ and MANOVA§ 7.2 →
Common $\Sigma$: $T^2=\frac{n_1n_2}{n_1+n_2}(\bar X_1-\bar X_2)^{\mathsf T}S_{\text{pl}}^{-1}(\bar X_1-\bar X_2)$, $\frac{n_1+n_2-p-1}{p(n_1+n_2-2)}T^2\sim F_{p,n_1+n_2-p-1}$. $k$ groups: Wilks $\Lambda=\frac{|W|}{|W+B|}$, $-(N-1-\frac{p+k}2)\log\Lambda\approx\chi^2_{p(k-1)}$.
Principal components§ 7.3 →
$\Sigma=\sum_k\lambda_ke_ke_k^{\mathsf T}$, $\lambda_1\ge\dots\ge\lambda_p\ge0$: $Y_k=e_k^{\mathsf T}(X-\mu)$, $\operatorname{Var}(Y_k)=\lambda_k=\max\{a^{\mathsf T}\Sigma a:\|a\|=1,\ a\perp e_1,\dots,e_{k-1}\}$, $\operatorname{Cov}(Y_j,Y_k)=0$. Proportion of variance $\frac{\lambda_1+\dots+\lambda_k}{\operatorname{tr}\Sigma}$ ($\operatorname{tr}R=p$); $\operatorname{corr}(Y_k,X_i)=\frac{e_{ik}\sqrt{\lambda_k}}{\sqrt{\sigma_{ii}}}$; $\prod\lambda_k=\det\Sigma$. Not scale invariant.
Optimal classification (two normal populations)§ 7.3 →
Minimum ECM: allocate to $\mathcal P_1$ iff $\frac{f_1(x)}{f_2(x)}\ge k=\frac{c(1\mid2)\pi_2}{c(2\mid1)\pi_1}$. For $N_p(\mu_k,\Sigma)$: $W(x)=(\mu_1-\mu_2)^{\mathsf T}\Sigma^{-1}\big(x-\frac{\mu_1+\mu_2}2\big)\ge\log k$; $P(2\mid1)=\Phi\big(\frac{\log k-\Delta^2/2}{\Delta}\big)$, $P(1\mid2)=\Phi\big(\frac{-\log k-\Delta^2/2}{\Delta}\big)$; equal priors and costs: both $\Phi(-\Delta/2)$.
Fisher's linear discriminant§ 7.3 →
$\max_a\frac{(a^{\mathsf T}d)^2}{a^{\mathsf T}S_{\text{pl}}a}=D^2=d^{\mathsf T}S_{\text{pl}}^{-1}d$ at $\hat a=S_{\text{pl}}^{-1}d$, $d=\bar x_1-\bar x_2$. Rule: $\hat a^{\mathsf T}x_0\ge\frac12\hat a^{\mathsf T}(\bar x_1+\bar x_2)$ → $\mathcal P_1$. $T^2=\frac{n_1n_2}{n_1+n_2}D^2$. $g$ groups: eigenvectors of $W^{-1}B$, at most $\min(g-1,p)$ nonzero.
Clustering identities§ 7.3 →
$T=W+B$: $\sum_i\|x_i-\bar x\|^2=\sum_k\sum_{i\in C_k}\|x_i-\bar x_k\|^2+\sum_k|C_k|\|\bar x_k-\bar x\|^2$. Ward merge cost $\frac{|A||B|}{|A|+|B|}\|\bar x_A-\bar x_B\|^2$. Single: $\min d$; complete: $\max d$; average: mean $d$ — monotone heights; single linkage heights = minimum spanning tree edges.
Canonical correlations§ 7.3 →
$\rho_k^{*2}$ = eigenvalues of $\Sigma_{11}^{-1}\Sigma_{12}\Sigma_{22}^{-1}\Sigma_{21}$ (singular values of $\Sigma_{11}^{-1/2}\Sigma_{12}\Sigma_{22}^{-1/2}$); $b_k\propto\Sigma_{22}^{-1}\Sigma_{21}a_k$. $q=1$: $\rho^*=R_{Y\cdot X}$. Test $\Sigma_{12}=0$: $-\big(n-1-\frac{p+q+1}2\big)\sum_k\log(1-r_k^2)\approx\chi^2_{pq}$.

Chapter 8 · Sample Surveys and Design of Experiments

SRSWOR and SRSWR§ 8.1 →
SRSWOR: $E\bar y=\bar Y$, $\operatorname{Var}\bar y=\frac{(1-f)S^2}{n}$, $Es^2=S^2$, $v(\bar y)=\frac{(1-f)s^2}{n}$, $f=\frac nN$. SRSWR: $\operatorname{Var}\bar y=\frac{\sigma^2}{n}$, $Es^2=\sigma^2=\frac{N-1}NS^2$.
Proportions and sample size§ 8.1 →
$\operatorname{Var}(p)=\frac{N-n}{N-1}\frac{PQ}{n}$, $v(p)=\frac{(1-f)pq}{n-1}$. Margin $d$: $n=\frac{n_0}{1+n_0/N}$, $n_0=\left(\frac{z_{\alpha/2}S}{d}\right)^2$; for $P$: $n_0=\frac{z^2PQ}{d^2}$, $n=\frac{n_0}{1+(n_0-1)/N}$.
Stratified sampling and allocation§ 8.1 →
$\operatorname{Var}(\bar y_{\mathrm{st}})=\sum W_h^2\frac{(1-f_h)S_h^2}{n_h}$. Optimum (cost $c_0+\sum c_hn_h$): $n_h\propto\frac{N_hS_h}{\sqrt{c_h}}$; Neyman $n_h\propto N_hS_h$: $V=\frac{(\sum W_hS_h)^2}{n}-\frac{\sum W_hS_h^2}{N}$; $V_{\mathrm{prop}}-V_{\mathrm{Ney}}=\frac1n\sum W_h(S_h-\bar S)^2$.
Systematic sampling ($N=nk$)§ 8.1 →
$V_{\mathrm{sys}}=\frac{N-1}NS^2-\frac{k(n-1)}NS^2_{\mathrm{wsy}}=\frac{\sigma^2}n[1+(n-1)\rho_w]$; better than SRS iff $S^2_{\mathrm{wsy}}\gt S^2$. Trend $a+bi$: $V_{\mathrm{sys}}=\frac{b^2(k^2-1)}{12}$, $V_{\mathrm{ran}}=\frac{b^2(k-1)(N+1)}{12}$.
PPS with replacement (Hansen–Hurwitz)§ 8.2 →
$p_i=\frac{X_i}{X}$: $\hat Y_{\mathrm{HH}}=\frac1n\sum\frac{y_j}{p_j}$, $\operatorname{Var}=\frac1n\left(\sum\frac{Y_i^2}{p_i}-Y^2\right)$, $v=\frac{\sum(z_j-\bar z)^2}{n(n-1)}$. Lahiri: accept $i$ if $R\le X_i$, $M\ge\max X_i$; mean trials $\frac{NM}{X}$.
Horvitz–Thompson estimator§ 8.2 →
$\hat Y_{\mathrm{HT}}=\sum_{s}\frac{y_i}{\pi_i}$ (all $\pi_i\gt0$). Fixed size: $\sum\pi_i=n$, $\sum_{j\ne i}\pi_{ij}=(n-1)\pi_i$, $\operatorname{Var}=\frac12\sum_{i\ne j}(\pi_i\pi_j-\pi_{ij})\left(\frac{Y_i}{\pi_i}-\frac{Y_j}{\pi_j}\right)^2$.
Ratio estimator§ 8.2 →
$\bar y_R=\frac{\bar y}{\bar x}\bar X$; $\operatorname{MSE}\approx\frac{1-f}n(S_y^2+R^2S_x^2-2R\rho S_xS_y)$; bias $\approx\frac{1-f}n\bar Y(C_x^2-\rho C_xC_y)$; $\frac{|\text{bias}|}{\text{SD}}\le\operatorname{CV}(\bar x)$; beats $\bar y$ iff $\rho\gt\frac{C_x}{2C_y}$.
Regression estimator§ 8.2 →
$\bar y_{lr}=\bar y+b(\bar X-\bar x)$: $\operatorname{MSE}\approx\frac{1-f}nS_y^2(1-\rho^2)$; $\operatorname{MSE}(\bar y_R)-V_{lr}\approx\frac{1-f}n(RS_x-\rho S_y)^2\ge0$; fixed $b_0$: unbiased, $\operatorname{Var}=\frac{1-f}n(S_y^2-2b_0S_{xy}+b_0^2S_x^2)$.
Degrees of freedom of the three classical designs§ 8.3 →
CRD: treatments $v-1$, error $N-v$. RBD ($b$ blocks): blocks $b-1$, treatments $v-1$, error $(v-1)(b-1)$. LSD (order $v$): rows, columns, treatments $v-1$ each, error $(v-1)(v-2)$. Subtract one error df per missing value.
Randomisation analysis of the CRD ($N=vr$)§ 8.3 →
Under unit–treatment additivity: $E[\bar y_i-\bar y_{i'}]=\tau_i-\tau_{i'}$, $\operatorname{Var}(\bar y_i-\bar y_{i'})=2\sigma^2/r$, $E[\text{MSE}]=\sigma^2$, $E[\text{MSTr}]=\sigma^2+r\sum(\tau_i-\bar\tau)^2/(v-1)$, with $\sigma^2=S_u^2+\sigma_e^2$, $S_u^2$ the variance (divisor $N-1$) of the unit values and $\sigma_e^2$ the technical-error variance.
Relative efficiency§ 8.3 →
RBD vs CRD: $\frac{(b-1)M_b+b(v-1)M_e}{(bv-1)M_e}$ ($\gt1\iff M_b\gt M_e$). LSD vs CRD: $\frac{M_r+M_c+(v-1)M_e}{(v+1)M_e}$. LSD vs RBD with rows as blocks: $\frac{M_c+(v-1)M_e}{vM_e}$.
One missing value§ 8.3 →
RBD: $\hat x=\frac{vT+bB-G}{(v-1)(b-1)}$, bias of treatment SS $\frac{[B-(v-1)\hat x]^2}{v(v-1)}$, $\operatorname{Var}(\hat\tau_i-\hat\tau_{i'})=\sigma^2\bigl[\frac2b+\frac{v}{b(b-1)(v-1)}\bigr]$. LSD: $\hat x=\frac{v(R+C+T)-2G}{(v-1)(v-2)}$, bias $\frac{[G-R-C-(v-1)T]^2}{(v-1)^2(v-2)^2}$.
Control versus $m$ test treatments§ 8.3 →
$\operatorname{Var}(\bar y_0-\bar y_i)=\sigma^2\bigl(\frac1{r_0}+\frac1r\bigr)$ is minimised for fixed $N$ by $r_0=r\sqrt m$.
Block designs: reduced normal equations§ 8.4 →
$C=R-NK^{-1}N^{\mathsf T}$, $Q=T-NK^{-1}B$, $C\hat\tau=Q$; $E[Q]=C\tau$, $\operatorname{Cov}(Q)=\sigma^2C$, $C\mathbf 1=0$. Connected $\iff\operatorname{rank}C=v-1$. Orthogonal $\iff n_{ij}=r_ik_j/n$.
BIBD$(v,b,r,k,\lambda)$§ 8.4 →
$bk=vr$, $\lambda(v-1)=r(k-1)$, $NN^{\mathsf T}=(r-\lambda)I+\lambda J$, $b\ge v$, $r\ge k$. Symmetric ($b=v$): two blocks share $\lambda$ treatments; $v$ even $\Rightarrow k-\lambda$ a perfect square.
Intrablock analysis of a BIBD§ 8.4 →
$Q_i=T_i-\frac1k\sum_{j\ni i}B_j$, $\hat\tau_i=\frac{k}{\lambda v}Q_i$, SS(adj) $=\frac{k}{\lambda v}\sum Q_i^2$ on $v-1$ df, error $bk-b-v+1$ df; $\operatorname{Var}(\hat\tau_i-\hat\tau_{i'})=\frac{2k\sigma^2}{\lambda v}$; $E=\frac{\lambda v}{rk}$.
$2^k$ factorial with $r$ replicates§ 8.4 →
Effect $\hat X=[X]/(r2^{k-1})$, $\text{SS}_X=[X]^2/(r2^k)$ (1 df), $\operatorname{Var}(\hat X)=\sigma^2/(r2^{k-2})$. $2^2$: $[A]=ab+a-b-(1)$, $[AB]=ab-a-b+(1)$. Yates: $k$ passes of pair sums then differences.
Confounding in $2^k$§ 8.4 →
Blocks by parities $L_X(t)=\sum_{\ell\in X}t_\ell\pmod 2$ of $p$ defining contrasts; $2^p$ blocks of $2^{k-p}$; confounded: the $2^p-1$ products (e.g. $ABC\cdot BCD=AD$). Partial: relative information $(r-r_c)/r$.

Chapter 9 · Reliability, Linear Programming and Queues

Reliability, hazard and cumulative hazard§ 9.1 →
$R(t)=P(T\gt t)$, $h(t)=\frac{f(t)}{R(t)}=-\frac{d}{dt}\log R(t)$, $H(t)=\int_0^th$: $R=e^{-H}$, $f=he^{-H}$ (continuous density); $\int_0^\infty h=\infty$; $H(T)\sim\operatorname{Exp}(1)$; $P(T\gt t+x\mid T\gt t)=e^{-(H(t+x)-H(t))}$.
Mean life and mean residual life§ 9.1 →
$\mu=\int_0^\infty R(t)\,dt$, $E[T^r]=r\int_0^\infty t^{r-1}R(t)\,dt$; $m(t)=\frac{1}{R(t)}\int_t^\infty R(u)\,du$, $h=\frac{1+m'}{m}$. IFR ⇒ NBU and DMRL; IFR ⇒ $R(t)\ge e^{-t/\mu}$ for $t\lt\mu$.
Standard lifetimes§ 9.1 →
$\operatorname{Exp}(\lambda)$: $h\equiv\lambda$, $\mu=\frac1\lambda$. $\operatorname{Weib}(k,\lambda)$: $H=(t/\lambda)^k$, $h=\frac k\lambda(t/\lambda)^{k-1}$, $\mu=\lambda\Gamma(1+\frac1k)$, IFR iff $k\ge1$, DFR iff $k\le1$. $\operatorname{Gamma}(\alpha,\lambda)$: IFR for $\alpha\ge1$, DFR for $\alpha\le1$, $h\to\lambda$. Mixtures of exponentials are DFR.
Systems of independent components§ 9.1 →
Series $R_S=\prod R_i$, $h_S=\sum h_i$; parallel $R_S=1-\prod(1-R_i)$; $k$-out-of-$n$ (iid) $R_S=\sum_{j=k}^n\binom nj R^j(1-R)^{n-j}$. Exponential parts: series $\operatorname{Exp}(\sum\lambda_i)$; $k$-out-of-$n$ MTTF $\frac1\lambda\sum_{j=k}^n\frac1j$; cold standby of $n$: $\operatorname{Gamma}(n,\lambda)$, MTTF $\frac n\lambda$.
Exponential life tests (mean θ)§ 9.1 →
Type II: $T_r=\sum_{i\le r}X_{(i)}+(n-r)X_{(r)}$, $\hat\theta=\frac{T_r}{r}$ (UMVUE), $\frac{2T_r}\theta\sim\chi^2_{2r}$, $\operatorname{Var}\hat\theta=\frac{\theta^2}r$, CI $\left[\frac{2T_r}{\chi^2_{2r,\alpha/2}},\frac{2T_r}{\chi^2_{2r,1-\alpha/2}}\right]$. Type I: $\hat\theta=\frac{T}{D}$ ($D\ge1$); with replacement $T=nt_0$, $D\sim\operatorname{Poi}(nt_0/\theta)$.
BFS = extreme point; optimum at a corner§ 9.2 →
For $Ax=b$, $x\ge0$ ($A$ is $m\times n$ of rank $m$): $x$ is an extreme point iff its positive components have independent columns; basic solution $x_{\mathcal B}=B^{-1}b$, $x_{\mathcal N}=0$; at most $\binom nm$ BFSs. If an optimum exists, some BFS is optimal.
Simplex method (maximisation)§ 9.2 →
$y_j=B^{-1}a_j$, $\Delta_j=z_j-c_j=c_{\mathcal B}^{\mathsf T}y_j-c_j$; $\ c^{\mathsf T}x=z_0-\sum_{j\in\mathcal N}\Delta_jx_j$. Optimal if all $\Delta_j\ge0$; else enter $k$ with $\Delta_k\lt0$; unbounded if $y_k\le0$; otherwise leave by $\min\{\bar x_{B_i}/y_{ik}:y_{ik}\gt0\}$; new value $z_0-\theta\Delta_k$.
Duality and complementary slackness§ 9.2 →
(P) $\max c^{\mathsf T}x$, $Ax\le b$, $x\ge0$ ↔ (D) $\min b^{\mathsf T}y$, $A^{\mathsf T}y\ge c$, $y\ge0$. Weak: $c^{\mathsf T}x\le b^{\mathsf T}y$. Strong: equal optima, $y^{\mathsf T}=c_{\mathcal B}^{\mathsf T}B^{-1}$. Optimal iff $y_i(b-Ax)_i=0$ and $x_j(A^{\mathsf T}y-c)_j=0$. Rules (max primal): $\le$ ↔ $y_i\ge0$, $\ge$ ↔ $y_i\le0$, $=$ ↔ free.
Two-phase and Big-M§ 9.2 →
Phase I: $\min\sum a_i$ s.t. $Ax+a=b$, $x,a\ge0$ ($b\ge0$); feasible iff $w^*=0$. Big-M: $\max c^{\mathsf T}x-M\sum a_i$, compare $\alpha M+\beta$ by $\alpha$ first.
Transportation and assignment§ 9.2 →
Balanced TP: rank $m+n-1$, basic cells form a tree, integral data ⇒ integral BFS. Optimal if $d_{ij}=c_{ij}-u_i-v_j\ge0$ with $u_i+v_j=c_{ij}$ on basic cells. Assignment: subtracting a constant from a row or column leaves optimal assignments unchanged (Hungarian method).
Little's law and PASTA§ 9.3 →
$L=\lambda_{\text{eff}}W$, $L_q=\lambda_{\text{eff}}W_q$, busy servers $=\lambda_{\text{eff}}E[S]$ (any distributions, any discipline). Poisson arrivals find state $n$ with probability $\pi_n$.
M/M/1 (ρ = λ/μ < 1)§ 9.3 →
$\pi_n=(1-\rho)\rho^n$; $L=\frac{\rho}{1-\rho}$, $L_q=\frac{\rho^2}{1-\rho}$, $W=\frac1{\mu-\lambda}$, $W_q=\frac{\rho}{\mu-\lambda}$; sojourn $\sim\operatorname{Exp}(\mu-\lambda)$, $P(W_q\gt t)=\rho e^{-(\mu-\lambda)t}$.
M/M/1/K (any ρ)§ 9.3 →
$\pi_n=\frac{(1-\rho)\rho^n}{1-\rho^{K+1}}$ ($\rho\ne1$), $\frac1{K+1}$ ($\rho=1$); blocking $\pi_K$; $\lambda_{\text{eff}}=\lambda(1-\pi_K)=\mu(1-\pi_0)$; $L=\frac{\rho}{1-\rho}-\frac{(K+1)\rho^{K+1}}{1-\rho^{K+1}}$.
M/M/c and the Erlang formulas§ 9.3 →
$a=\lambda/\mu$, $\rho=a/c\lt1$: $C(c,a)=\pi_0\frac{a^c}{c!(1-\rho)}$, $L_q=C\frac{\rho}{1-\rho}$, $W_q=\frac{C}{c\mu-\lambda}$, $L=L_q+a$. Loss system: $B(c,a)=\frac{a^c/c!}{\sum_{k\le c}a^k/k!}$, $B(c,a)=\frac{aB(c-1,a)}{c+aB(c-1,a)}$, $C=\frac{B}{1-\rho(1-B)}$.
Pollaczek–Khinchine (M/G/1)§ 9.3 →
$\rho=\lambda E[S]\lt1$: $W_q=\frac{\lambda E[S^2]}{2(1-\rho)}$, $L_q=\frac{\lambda^2\sigma^2+\rho^2}{2(1-\rho)}$, $L=L_q+\rho$, $W=W_q+E[S]$. M/D/1: $L_q=\frac{\rho^2}{2(1-\rho)}$.
Economic order quantity§ 9.3 →
$Q^*=\sqrt{2KD/h}$, cost $\sqrt{2KDh}$, $\frac{C(Q)}{C^*}=\frac12\left(\frac Q{Q^*}+\frac{Q^*}Q\right)$. Shortages (cost $p$): replace $h$ by $\frac{hp}{h+p}$, $S^*=\frac{p}{h+p}Q^*$. Finite production rate $P$: replace $h$ by $h(1-D/P)$.