Reference

Glossary

Every technical word in the course and what it means in plain language, with flashcards for practice.

Every technical term in the course, with a plain-language meaning. On every page these words are marked with a dotted underline: hover or tap one to see its meaning. The “Used in” links point to the sections where each term appears.

Flashcards — term ⇄ meaning
Ch 1 · Probability
TermMeaningUsed in
almost surelyWith probability $1$. An event of probability $0$ may still occur, so 'almost surely' is weaker than 'surely'.1.2 1.3 1.4 2.2
Bayes' theoremFor a partition $\{B_i\}$, $P(B_j\mid A)=P(A\mid B_j)P(B_j)/\sum_iP(A\mid B_i)P(B_i)$: posterior $\propto$ likelihood $\times$ prior.1.2 1.4 2.1 4.3
bivariate normal distributionThe joint law with elliptical density determined by $\mu_1,\mu_2,\sigma_1,\sigma_2,\rho$; marginals and conditionals are normal, $E[Y\mid X]$ is linear, and $\rho=0$ gives independence.1.4 4.1 5.2 5.3
Borel–Cantelli lemmaIf $\sum P(A_n)\lt\infty$ then a.s. only finitely many $A_n$ occur; if the $A_n$ are independent and $\sum P(A_n)=\infty$ then a.s. infinitely many occur.1.2 2.3
coefficient of variation$\mathrm{CV}=s/\bar x$ for ratio-scale data with positive mean: a unit-free measure of relative spread, often given as a percentage.1.1 6.3 8.1 8.2
conditional expectation$E[X\mid Y=y]$ is the mean of the conditional law of $X$ given $Y=y$; $E[X\mid Y]=m(Y)$ is a random variable with $E[E[X\mid Y]]=E[X]$ and is the best mean-square predictor of $X$ from $Y$.1.4
conditional probability$P(A\mid B)=P(A\cap B)/P(B)$ for $P(B)\gt0$: the probability of $A$ once $B$ is known to have occurred. As a function of $A$ it is again a probability measure.1.2 2.1 3.3 4.2
convolutionThe law of a sum of independent variables: $f_{X+Y}(z)=\int f_X(x)f_Y(z-x)\,dx$, or $\sum_kp_X(k)p_Y(n-k)$; pgfs and mgfs multiply.1.4 2.1
correlation coefficientPearson's $r=s_{xy}/(s_xs_y)$, the cosine of the angle between the centred data vectors. It lies in $[-1,1]$ and equals $\pm1$ iff the points are collinear; it measures linear association only.1.1 1.4 2.3 5.3
covariance$\operatorname{Cov}(X,Y)=E[(X-\mu_X)(Y-\mu_Y)]=E[XY]-E[X]E[Y]$; zero for independent variables, but zero covariance does not imply independence.1.1 1.4 2.1 2.2
distribution function$F(x)=P(X\le x)$: nondecreasing, right-continuous, with limits $0$ and $1$; its jump at $x$ is $P(X=x)$. Every such function is the cdf of some random variable.1.2 1.3 1.4 2.1
exchangeableA sequence of events or variables whose joint probabilities do not change when the order is permuted, e.g. the colours drawn from Pólya's urn. Exchangeable variables need not be independent.1.2 2.1 4.1 4.2
inclusion–exclusion$P(\bigcup A_i)=S_1-S_2+S_3-\cdots$, where $S_k$ sums the probabilities of all $k$-fold intersections; truncations give the Bonferroni bounds.1.2 1.4 8.4
interquartile range$\mathrm{IQR}=Q_3-Q_1$, the spread of the middle half of the data. Unlike the range or the SD, it is not affected by a few extreme values.1.1 1.3
JacobianFor a change of variables $(x,y)=(x(u,v),y(u,v))$, the determinant $J=\partial(x,y)/\partial(u,v)$; the new density is $f(x(u,v),y(u,v))\,|J|$.1.4 2.1 3.3 4.1
joint distributionThe law of a random vector, given by $F(x,y)=P(X\le x,\,Y\le y)$, a joint pmf $p(x,y)$ or a joint density $f(x,y)$; it determines the marginals but not conversely.1.4 2.1 3.3 4.1
kurtosis$\beta_2=m_4/m_2^2$, a measure of tail weight; it equals $3$ for the normal distribution and always satisfies $\beta_2\ge1+\beta_1$.1.1 1.3 2.1 2.3
law of total variance$\operatorname{Var}X=E[\operatorname{Var}(X\mid Y)]+\operatorname{Var}(E[X\mid Y])$: average within-group variance plus variance of the group means.1.4 2.1 3.3 4.2
marginal distributionThe law of one component of a random vector, obtained by summing or integrating the other variables out: $f_X(x)=\int f(x,y)\,dy$.1.4 4.3
mixed distributionA law whose cdf is $\alpha F_d+(1-\alpha)F_c$ with $0\lt\alpha\lt1$, $F_d$ discrete (jumps) and $F_c$ with a density, e.g. a lifetime with an atom at $0$.1.3
moment generating function$M_X(t)=E[e^{tX}]$. If it is finite near $0$, all moments exist, $E[X^r]=M_X^{(r)}(0)$, and $M_X$ determines the distribution.1.3 2.2 7.1
mutual independenceEvents are mutually independent if the probability of every finite intersection of distinct events is the product of their probabilities; pairwise independence (products of pairs only) is weaker.1.2 1.4 4.1 6.2
outlierAn observation far from the bulk of the data; in a box plot, any value below $Q_1-1.5\,\mathrm{IQR}$ or above $Q_3+1.5\,\mathrm{IQR}$.1.1 5.3 6.3 7.1
probability density functionA function $f\ge0$ with $F(x)=\int_{-\infty}^xf(t)\,dt$; probabilities are integrals of $f$, so $f$ itself may exceed $1$.1.3
probability generating function$G_X(s)=E[s^X]=\sum_kP(X=k)s^k$ for $X$ in $\{0,1,2,\dots\}$; $P(X=k)=G^{(k)}(0)/k!$, $E[X]=G'(1)$, $\operatorname{Var}X=G''(1)+G'(1)-G'(1)^2$.1.3 5.3 9.3
probability integral transformIf $X$ has a continuous cdf $F$, then $F(X)$ is uniform on $(0,1)$. It fails when $F$ has jumps.1.3 4.1 4.3 5.1
probability spaceA triple $(\Omega,\mathcal{F},P)$: a sample space, a σ-field of events, and a countably additive $P$ with $P\ge0$ and $P(\Omega)=1$ (Kolmogorov's axioms).1.2 1.3 1.4 2.1
quantile function$Q(u)=\inf\{x:F(x)\ge u\}$ for $0\lt u\lt1$. It satisfies $Q(u)\le x\iff u\le F(x)$, and $Q(U)$ has cdf $F$ when $U$ is uniform on $(0,1)$.1.3 2.3 4.1
random variableA function $X:\Omega\to\mathbb{R}$ on a probability space with $\{X\le x\}$ an event for every $x$; then $\{X\in B\}$ is an event for every Borel set $B$.1.3 1.4 2.1 2.2
regression coefficientThe slope $b_{yx}=s_{xy}/s_x^2$ of the least-squares line of $y$ on $x$ (or $b_{xy}=s_{xy}/s_y^2$ for $x$ on $y$); $b_{yx}b_{xy}=r^2$.1.1 1.4 6.2 7.1
skewnessAsymmetry of a distribution. The moment coefficient is $\gamma_1=m_3/m_2^{3/2}$, which is positive when the right tail is longer; Pearson's and Bowley's coefficients are alternatives.1.1 1.3 2.1 2.3
trimmed meanThe mean after discarding the $k=\lfloor n\alpha\rfloor$ smallest and $k$ largest observations. It lies between the mean ($\alpha=0$) and the median in robustness.1.1
σ-fieldA collection of subsets of $\Omega$ that contains $\Omega$ and is closed under complements and countable unions; its members are the events. A σ-field with $k$ atoms has $2^k$ sets.1.2 1.3
Ch 2 · Distributions and Limit Theorems
TermMeaningUsed in
almost-sure convergence$X_n\xrightarrow{a.s.}X$ if $P(X_n\to X)=1$: almost every sample path converges. Equivalent to $P(\sup_{m\ge n}|X_m-X|\gt\varepsilon)\to0$ for every $\varepsilon\gt0$.1.2 2.3
beta distributionThe law $\operatorname{Beta}(a,b)$ on $(0,1)$ with density $x^{a-1}(1-x)^{b-1}/B(a,b)$ and mean $a/(a+b)$. It models random proportions; $X/(X+Y)$ is beta when $X,Y$ are independent gamma variables with a common rate.2.1 4.1 7.1 7.2
beta–binomial distributionThe law of a binomial count whose success probability is itself random with a beta law: $P(X=k)=\binom nkB(a+k,b+n-k)/B(a,b)$. It has the binomial mean $na/(a+b)$ but a larger variance.2.1 4.1 4.3
binomial distributionThe law $\operatorname{Bin}(n,p)$ of the number of successes in $n$ independent trials, each a success with probability $p$: $P(X=k)=\binom nkp^k(1-p)^{n-k}$, with mean $np$ and variance $np(1-p)$.1.3 2.1 2.2 3.3
Cantelli's inequalityThe one-sided Chebyshev inequality $P(X-\mu\ge\lambda)\le\frac{\sigma^2}{\sigma^2+\lambda^2}$ for $\lambda\gt0$; sharp for a two-point law.2.2
Cauchy–Schwarz inequality$(E[XY])^2\le E[X^2]E[Y^2]$, with equality iff $aX=bY$ a.s. for some $(a,b)\ne(0,0)$; it implies $|\rho|\le1$. Hölder's inequality $E|XY|\le\|X\|_p\|Y\|_q$ ($\frac1p+\frac1q=1$) generalises it.1.1 1.4 2.1 2.2
central limit theoremFor iid $X_j$ with mean $\mu$ and finite variance $\sigma^2\gt0$, $\sqrt n(\bar X_n-\mu)/\sigma\xrightarrow{d}N(0,1)$ (Lindeberg–Lévy); Lyapunov's and Lindeberg's conditions extend it to non-identical summands.2.1 2.2 2.3 3.3
characteristic functionThe function $\varphi_X(t)=E[e^{itX}]=E[\cos tX]+iE[\sin tX]$, $t\in\mathbb{R}$. It exists for every random variable, determines the law, turns sums of independent variables into products, and its derivatives at $0$ give the moments: $\varphi^{(k)}(0)=i^kE[X^k]$.1.3 2.1 2.2 2.3
Chebyshev's inequalityFor $X$ with mean $\mu$ and variance $\sigma^2$, $P(|X-\mu|\ge k\sigma)\le1/k^2$; sharp for a three-point law, useless for $k\le1$.2.2 2.3 5.3
continuity correctionWhen approximating an integer-valued law by a normal law, $P(X\le b)$ is replaced by the normal area up to $b+\frac12$ and $P(X\ge a)$ by the area from $a-\frac12$.2.2 2.3 5.2 5.3
convergence in distribution$X_n\xrightarrow{d}X$ if $F_{X_n}(x)\to F_X(x)$ at every point where $F_X$ is continuous; only the laws matter. The weakest of the standard modes of convergence.1.3 2.2 2.3 4.1
convergence in mean$X_n\to X$ in $r$-th mean (in $L^r$) if $E|X_n-X|^r\to0$; for $r=2$, mean-square convergence. It implies convergence in probability but neither implies nor is implied by almost-sure convergence.2.3
convergence in probability$X_n\xrightarrow{P}X$ if $P(|X_n-X|\gt\varepsilon)\to0$ for every $\varepsilon\gt0$. Implied by almost-sure and by $r$-th mean convergence; implies convergence in distribution.2.2 2.3 5.2
delta methodIf $\sqrt n(T_n-\theta)\xrightarrow{d}N(0,\tau^2)$ and $g$ is differentiable at $\theta$, then $\sqrt n(g(T_n)-g(\theta))\xrightarrow{d}N(0,g'(\theta)^2\tau^2)$; when $g'(\theta)=0$ a second-order version gives a $\chi^2_1$ limit at scale $n$.2.3 4.1 4.2 4.3
gamma distributionThe law $\operatorname{Gamma}(\alpha,\lambda)$ with density $\lambda^\alpha x^{\alpha-1}e^{-\lambda x}/\Gamma(\alpha)$ on $(0,\infty)$; mean $\alpha/\lambda$, variance $\alpha/\lambda^2$. For integer $\alpha=n$ it is the law of a sum of $n$ independent $\operatorname{Exp}(\lambda)$ waiting times; $\chi^2_n=\operatorname{Gamma}(n/2,1/2)$.1.3 1.4 2.1 4.1
geometric distributionThe law of the number of independent trials up to the first success, $P(T=k)=(1-p)^{k-1}p$ on $\{1,2,\dots\}$ (or of the number of failures before it, on $\{0,1,\dots\}$). It is the only memoryless law on the positive integers.1.4 2.1 2.2 3.2
hypergeometric distributionThe law of the number of successes in a sample of $n$ drawn without replacement from $N$ items of which $K$ are successes: $P(X=k)=\binom Kk\binom{N-K}{n-k}/\binom Nn$. Mean $np$ and variance $np(1-p)\frac{N-n}{N-1}$ with $p=K/N$.2.1 4.2 5.2 5.3
inversion formulaThe formula recovering a distribution from its characteristic function; if $\int|\varphi|\lt\infty$, the density is $f(x)=\frac1{2\pi}\int e^{-itx}\varphi(t)\,dt$. It implies that the characteristic function determines the law (uniqueness).1.3 2.2 9.1
Jensen's inequalityFor convex $g$ and integrable $X$, $g(E[X])\le E[g(X)]$ (reversed for concave $g$), with equality for strictly convex $g$ only when $X$ is constant. Gives $E[X^2]\ge(EX)^2$, $E[\log X]\le\log E[X]$ and Lyapunov's inequality.1.3 2.2 4.2 4.3
Lévy's continuity theorem$X_n\xrightarrow{d}X$ iff $\varphi_{X_n}(t)\to\varphi_X(t)$ for every $t$; and if $\varphi_{X_n}(t)\to\psi(t)$ pointwise with $\psi$ continuous at $0$, then $\psi$ is a characteristic function and the $X_n$ converge in distribution to its law.2.2 2.3
Markov's inequalityFor $X\ge0$ and $a\gt0$, $P(X\ge a)\le E[X]/a$; equality holds only for laws on $\{0,a\}$.2.2 2.3 3.3
memoryless propertyThe property $P(X\gt s+t\mid X\gt s)=P(X\gt t)$ for all $s,t\ge0$: the remaining waiting time does not depend on how long one has already waited. Among positive laws only the exponential has it, and among laws on $\{1,2,\dots\}$ only the geometric.1.3 1.4 2.1 3.3
negative binomial distributionThe law $\operatorname{NB}(r,p)$ of the number of failures before the $r$-th success in independent trials: $P(Y=k)=\binom{k+r-1}{k}p^r(1-p)^k$. A sum of $r$ independent geometric failure counts; its variance exceeds its mean.2.1 2.2 3.3 5.1
Poisson distributionThe law $\operatorname{Poi}(\lambda)$ with $P(X=k)=e^{-\lambda}\lambda^k/k!$, $k=0,1,2,\dots$; its mean and variance both equal $\lambda$. It is the limit of $\operatorname{Bin}(n,p)$ as $n\to\infty$ with $np\to\lambda$, the law of counts of rare events.1.2 2.1 2.3 3.3
reproductive propertyA family of distributions has the reproductive property if a sum of independent members is again a member, e.g. $\operatorname{Poi}(\lambda)+\operatorname{Poi}(\mu)=\operatorname{Poi}(\lambda+\mu)$, $\operatorname{Bin}(n,p)+\operatorname{Bin}(m,p)=\operatorname{Bin}(n+m,p)$ and gamma laws with a common rate.2.1 4.1 4.3 7.1
Slutsky's theoremIf $X_n\xrightarrow{d}X$ and $Y_n\xrightarrow{P}c$ (a constant), then $X_n+Y_n\xrightarrow{d}X+c$, $Y_nX_n\xrightarrow{d}cX$ and $X_n/Y_n\xrightarrow{d}X/c$ for $c\ne0$.2.3 4.1 4.2 4.3
strong law of large numbersKolmogorov's theorem: for iid $X_j$, $\bar X_n\to E[X_1]$ almost surely if and only if $E|X_1|\lt\infty$.2.3 3.2 3.3
variance-stabilising transformationA function $g$ with $g'(\theta)^2\tau^2(\theta)$ constant, so that the delta-method variance of $g(T_n)$ does not depend on $\theta$; e.g. $\sqrt x$ for Poisson means and $\log x$ for exponential means.2.3 4.3 6.3
weak law of large numbersA theorem asserting $\bar X_n-E[\bar X_n]\xrightarrow{P}0$; Chebyshev's version needs $\operatorname{Var}(S_n)/n^2\to0$, Khintchine's needs iid variables with a finite mean.2.2 2.3 4.2 5.3
Ch 3 · Markov Chains and Stochastic Processes
TermMeaningUsed in
absorbing stateA state $i$ with $p_{ii}=1$: once entered, it is never left.3.1
birth–death processA continuous-time Markov chain on $\{0,1,2,\dots\}$ that moves only $n\to n+1$ (rate $\lambda_n$) or $n\to n-1$ (rate $\mu_n$). Its steady state is $\pi_n\propto\lambda_0\cdots\lambda_{n-1}/(\mu_1\cdots\mu_n)$ when this is summable (and the process is recurrent). The Yule process is the pure birth process with $\lambda_n=n\lambda$.3.3 9.3
Chapman–Kolmogorov equations$p_{ij}^{(m+n)}=\sum_kp_{ik}^{(m)}p_{kj}^{(n)}$: to go from $i$ to $j$ in $m+n$ steps, pass through some state $k$ at time $m$. Equivalently $P^{(m+n)}=P^mP^n$.3.1 3.3
communicating classA maximal set of states each accessible from every other. A chain with a single communicating class is irreducible.3.1 3.2 9.3
compound Poisson process$X(t)=\sum_{i\le N(t)}Y_i$ with $N$ a Poisson process and iid jumps $Y_i$ independent of $N$; $E X(t)=\lambda tE[Y]$ and $\operatorname{Var}X(t)=\lambda tE[Y^2]$.2.2 3.3
continuous-time Markov chainA process on a countable state space with the Markov property in continuous time: it stays in state $i$ for an $\operatorname{Exp}(q_i)$ time and then jumps to $j$ with probability $q_{ij}/q_i$.3.3
counting processA process $N(t)$ with $N(0)=0$, integer values and nondecreasing right-continuous paths that increase by jumps of size $1$: $N(t)$ counts the events in $[0,t]$.3.3
couplingConstructing two processes on one probability space so that they eventually coincide; then $d_{TV}$ of their laws at time $n$ is at most $P(T\gt n)$, where $T$ is the meeting time.2.1 3.2
detailed balance$\pi_ip_{ij}=\pi_jp_{ji}$ for all $i,j$: in equilibrium the flow from $i$ to $j$ equals the flow back. It implies stationarity; a chain satisfying it is reversible (it looks the same run backwards).3.2 3.3
doubly stochastic matrixA transition matrix whose columns, as well as its rows, sum to $1$. On a finite state space this is exactly the condition for the uniform distribution to be stationary.3.2
ergodic theoremFor an irreducible chain, the fraction of time spent in state $j$ up to time $n$ converges almost surely to $1/\mu_j$ ($=\pi_j$ when positive recurrent): time averages equal space averages.3.2 3.3 9.3
fundamental matrixFor an absorbing chain with transient block $Q$, $N=(I-Q)^{-1}=\sum_nQ^n$; $N_{ij}$ is the expected number of visits to $j$ from $i$, $NR$ gives absorption probabilities and $N\mathbf 1$ expected absorption times.3.1
generator matrixThe matrix $Q=P'(0)$ of jump rates of a continuous-time Markov chain: $q_{ij}\ge0$ for $i\ne j$, rows summing to $0$, $q_i=-q_{ii}$ the rate of leaving $i$. For finite chains $P(t)=e^{tQ}$ and stationary laws solve $\pi Q=0$.3.3
independent incrementsA process has independent increments if its changes over disjoint time intervals are independent, and stationary increments if the law of $N(t+s)-N(s)$ depends only on the length $t$.3.3
jump chainThe discrete-time chain of states visited by a continuous-time Markov chain, with transition probabilities $q_{ij}/q_i$. Its stationary law is proportional to $\pi_jq_j$, not to $\pi_j$.3.3
Kolmogorov forward equationsThe differential equations $P'(t)=P(t)Q$ (forward: condition on the last short interval) and $P'(t)=QP(t)$ (backward: condition on the first) for the transition probabilities of a continuous-time Markov chain.3.3
Markov chainA sequence of random states $X_0,X_1,\dots$ in a countable set $S$ such that, given the present state $X_n=i$, the next state has law $(p_{ij})_j$ whatever the past: $P(X_{n+1}=j\mid X_0,\dots,X_{n-1},X_n=i)=p_{ij}$.3.1 3.2 3.3 7.2
mean return time$\mu_i=E_i(T_i)$, the expected time for a chain started at $i$ to come back to $i$.3.2 3.3
period of a state$d(i)=\gcd\{n\ge1:p_{ii}^{(n)}\gt0\}$; the state is aperiodic if $d(i)=1$. The period is the same for all states of a communicating class.3.1 3.2 3.3 9.3
Poisson processA counting process with independent increments and $N(t+s)-N(s)\sim\operatorname{Poi}(\lambda t)$; equivalently, one with iid $\operatorname{Exp}(\lambda)$ gaps between events. Given $N(t)=n$, the event times are sorted iid uniform points on $(0,t)$.1.4 2.1 3.3 9.1
positive recurrentA recurrent state is positive recurrent if its mean return time is finite and null recurrent if it is infinite (as for the symmetric random walk on $\mathbb{Z}$).3.1 3.2 3.3 9.3
recurrent stateState $i$ is recurrent if the chain started at $i$ returns to $i$ with probability one ($f_{ii}=1$), equivalently $\sum_np_{ii}^{(n)}=\infty$; otherwise it is transient and is visited only finitely often.3.1 3.2 3.3
stationary distributionA probability vector $\pi$ with $\pi P=\pi$: a chain started from $\pi$ keeps distribution $\pi$ for ever. For an irreducible chain it exists iff the chain is positive recurrent, and then $\pi_i=1/\mu_i$.3.1 3.2 3.3
stopping timeA random time $T$ such that whether $T=n$ can be decided from $X_0,\dots,X_n$ alone, such as a first passage time. The strong Markov property says the chain starts afresh at a stopping time.3.1 3.3
thinningKeeping each event of a process independently with probability $p$. Thinning a Poisson process of rate $\lambda$ gives independent Poisson processes of rates $\lambda p$ (kept) and $\lambda(1-p)$ (removed).2.1 3.3 9.3
total variation distance$d_{TV}(\mu,\nu)=\frac12\sum_j|\mu_j-\nu_j|=\max_A|\mu(A)-\nu(A)|$, the standard measure of distance from equilibrium for Markov chains.2.1 3.2
transition matrixThe matrix $P=(p_{ij})$ of one-step probabilities of a Markov chain: nonnegative entries, every row summing to $1$. The $n$-step probabilities are the entries of $P^n$.3.1 3.2 3.3
Ch 4 · Sampling Distributions and Estimation
TermMeaningUsed in
asymptotic relative efficiencyFor two asymptotically normal estimators of the same parameter, the ratio of their asymptotic variances; e.g. the sample median relative to $\bar X$ for normal data has efficiency $2/\pi$.4.1 5.3
Bayes estimatorAn estimator minimising the Bayes risk $\int R(\theta,\delta)\pi(\theta)d\theta$; it is the posterior mean under squared error loss and a posterior median under absolute error loss.4.3
chi-square distribution$\chi^2_k=\operatorname{Gamma}(\frac k2,\frac12)$, the law of $Z_1^2+\dots+Z_k^2$ for iid $N(0,1)$ variables; mean $k$, variance $2k$. For normal samples $(n-1)S^2/\sigma^2\sim\chi^2_{n-1}$.1.3 2.1 4.1 7.1
Clopper–Pearson intervalThe exact interval for a binomial proportion obtained by inverting the binomial cdf; its endpoints are beta quantiles and its coverage is at least $1-\alpha$ for every $p$.4.3
complete statistic$T$ is complete if $E_\theta[g(T)]=0$ for all $\theta$ forces $g(T)=0$ a.s.; then an unbiased function of $T$ is unique, and with sufficiency it is the UMVUE.4.2 6.1 8.4 9.1
confidence intervalA rule $[L(X),U(X)]$ computed from the data such that $P_\theta(L(X)\le\theta\le U(X))\ge1-\alpha$ for every $\theta$; the level describes the procedure over repeated samples, not a single realised interval.1.1 2.3 4.1 4.3
conjugate priorA family of priors that is closed under updating: the posterior is in the same family, e.g. beta for binomial data, gamma for Poisson data, normal for a normal mean with known variance.2.1 4.3
coverage probabilityThe probability $P_\theta(\theta\in C(X))$ that a confidence set contains the true parameter, as a function of $\theta$; its infimum over $\theta$ is the confidence coefficient.4.3
Cramér–Rao lower boundIn a regular model every unbiased estimator of $\psi(\theta)$ has $\operatorname{Var}\ge\psi'(\theta)^2/(nI(\theta))$; attained only in exponential families. It does not apply when the support depends on $\theta$.4.2 9.1
credible intervalAn interval containing the parameter with posterior probability $1-\alpha$; the highest posterior density (HPD) region $\{\theta:\pi(\theta\mid x)\ge k\}$ is the shortest one.4.3
exponential familyA family $f(x;\theta)=h(x)\exp(\sum_j\eta_j(\theta)T_j(x)-A(\theta))$ with support free of $\theta$; if the natural parameters fill an open set, $(\sum_iT_j(X_i))_j$ is complete sufficient.4.2 5.1 6.1 9.1
F distribution$F_{m,n}$ is the law of $\frac{U/m}{V/n}$ for independent $U\sim\chi^2_m$, $V\sim\chi^2_n$; mean $\frac{n}{n-2}$; $1/F_{m,n}\sim F_{n,m}$ and $t_n^2=F_{1,n}$.8.3
Fisher information$I(\theta)=E_\theta[(\partial_\theta\log f(X;\theta))^2]=-E_\theta[\partial_\theta^2\log f(X;\theta)]$ in a regular model; a sample of size $n$ carries $nI(\theta)$.4.2 4.3 5.1 5.2
Helmert transformationAn orthogonal matrix whose first row is $\frac{1}{\sqrt n}(1,\dots,1)$; applied to an iid normal sample it separates $\bar X$ from the deviations and proves $\bar X\perp S^2$ with $(n-1)S^2/\sigma^2\sim\chi^2_{n-1}$.4.1 6.1 7.1
maximum likelihood estimatorA value $\hat\theta$ maximising the likelihood $L(\theta)=\prod f(X_i;\theta)$; invariant under reparametrisation ($\widehat{\tau(\theta)}=\tau(\hat\theta)$) and asymptotically $N(\theta,1/(nI(\theta)))$ in regular models.2.1 2.3 4.1 4.2
mean squared error$\operatorname{MSE}_\theta(T)=E_\theta[(T-\psi(\theta))^2]=\operatorname{Var}_\theta(T)+\text{bias}^2$, the usual single measure of an estimator's accuracy.1.4 4.2 4.3 6.1
method of momentsEstimate $\theta$ by solving $E_\theta[X^r]=\frac1n\sum X_i^r$, $r=1,\dots,k$: equate population and sample moments.2.1 4.2
order statisticThe sorted sample values $X_{(1)}\le\dots\le X_{(n)}$; for a density $f$, $X_{(r)}$ has density $\frac{n!}{(r-1)!(n-r)!}F^{r-1}(1-F)^{n-r}f$.1.1 2.1 3.3 4.1
pivotal quantityA function $Q(X,\theta)$ of the data and the parameter whose distribution is known and the same for every $\theta$, e.g. $\sqrt n(\bar X-\mu)/S\sim t_{n-1}$ or $X_{(n)}/\theta$ for $U(0,\theta)$; inverting $a\le Q\le b$ gives a confidence set.4.3
posterior distributionThe conditional distribution of the parameter given the data, $\pi(\theta\mid x)\propto f(x\mid\theta)\pi(\theta)$: likelihood times prior, normalised.3.2 4.3
prior distributionThe distribution $\pi(\theta)$ describing beliefs about the parameter before the data are seen; an improper prior has infinite total mass, and Jeffreys' prior is $\pi(\theta)\propto\sqrt{I(\theta)}$.4.3
sample range$R=X_{(n)}-X_{(1)}$, the spread between the largest and smallest observations; for a $U(0,1)$ sample $R\sim\operatorname{Beta}(n-1,2)$.4.1
sampling distributionThe probability distribution of a statistic $T=t(X_1,\dots,X_n)$ computed from a random sample, e.g. $\bar X\sim N(\mu,\sigma^2/n)$ for a normal sample.2.3 4.1 4.3 5.2
standard errorThe standard deviation of a statistic's sampling distribution, e.g. $\operatorname{SE}(\bar X)=\sigma/\sqrt n$; with unknown parameters replaced by estimates it is the estimated standard error.2.3 4.1 4.2 4.3
Student's t distribution$t_k$ is the law of $Z/\sqrt{V/k}$ with $Z\sim N(0,1)$ independent of $V\sim\chi^2_k$; symmetric with heavier tails than $N(0,1)$, variance $k/(k-2)$ for $k\gt2$; $t_1$ is Cauchy.
sufficient statisticA statistic $T$ such that the conditional law of the sample given $T$ does not involve $\theta$; by the factorisation theorem, iff $f(x;\theta)=g(T(x);\theta)h(x)$.3.2 4.2 4.3 5.1
UMVUEUniformly minimum variance unbiased estimator: unbiased, with variance no larger than any other unbiased estimator for every $\theta$; found by Rao–Blackwell and Lehmann–Scheffé.4.2 6.1 9.1
unbiased estimatorA statistic $T$ with $E_\theta[T]=\psi(\theta)$ for every $\theta$; e.g. $S^2$ for $\sigma^2$. Unbiased estimators may fail to exist, and unbiasedness is not preserved by nonlinear maps.4.1 4.2 4.3 5.3
Wilson intervalThe large-sample interval for a binomial proportion obtained by solving $(\hat p-p)^2\le z^2p(1-p)/n$ for $p$; unlike the Wald interval $\hat p\pm z\sqrt{\hat p(1-\hat p)/n}$, it stays in $[0,1]$ and has coverage close to nominal.4.3 5.1 5.2
Ch 5 · Testing of Hypotheses
TermMeaningUsed in
contingency tableA table of counts of individuals cross-classified by two (or more) categorical attributes, with row totals $R_i$, column totals $C_j$ and grand total $n$.5.2 5.3
distribution-free testA test whose statistic has the same null distribution for every population in the null hypothesis (for example every continuous cdf), so that its size can be computed without knowing the population; typical examples use signs, ranks, runs or the empirical cdf.4.1 5.3
empirical distribution function$F_n(x)=\frac1n\#\{i:X_i\le x\}$, the cdf that puts mass $\frac1n$ on each observation; $nF_n(x)\sim\operatorname{Bin}(n,F(x))$ and $F_n\to F$ uniformly almost surely.2.3 5.3
expected frequencyThe count $E_j=n\pi_j$ predicted for a cell under $H_0$ (estimated when parameters are unknown; $R_iC_j/n$ in a contingency table). The $\chi^2$ approximation needs the expected, not the observed, counts to be reasonably large.5.1 5.2
Fisher's exact testA test for a $2\times2$ table that uses the hypergeometric law of one cell given all the margins; this law does not depend on the unknown common probability under $H_0$.5.2 5.3
goodness-of-fit testA test of whether data follow a specified distribution; Pearson's version compares observed and expected cell counts by $X^2=\sum(O-E)^2/E\approx\chi^2_{k-1-s}$, with $s$ the number of estimated parameters.2.1 5.2 6.3
index of dispersion$D=\sum(X_i-\bar X)^2/\bar X$, approximately $\chi^2_{n-1}$ for iid Poisson counts; large values indicate a variance larger than the mean (over-dispersion).2.1 3.3 5.2
Kendall's tau$\tau=\frac{C-D}{\binom n2}$, where $C$ and $D$ are the numbers of concordant and discordant pairs; under independence $E[\tau]=0$ and $\operatorname{Var}(\tau)=\frac{2(2n+5)}{9n(n-1)}$.5.3
Kolmogorov–Smirnov testGoodness-of-fit test based on $D_n=\sup_x|F_n(x)-F_0(x)|$ for a fully specified continuous $F_0$ (distribution-free), or on $\sup_x|F_m(x)-G_n(x)|$ for two samples; $\sqrt n\,D_n$ has Kolmogorov's limit law with $5\%$ point $1.358$.1.3 3.3 5.3
likelihood ratio testRejects $H_0$ when $\Lambda=\sup_{\Theta_0}L/\sup_\Theta L$ is small. By Wilks' theorem, in regular models $-2\log\Lambda\to\chi^2_r$ under $H_0$, $r=\dim\Theta-\dim\Theta_0$.4.2 5.1 5.2 6.1
Mann–Whitney testTwo-sample rank test of $F=G$ based on $U=\#\{X_i\lt Y_j\}$ or the rank sum $W=U+\frac{n(n+1)}2$ of the second sample; it detects $P(X\lt Y)\ne\frac12$, such as a shift in location.5.3
McNemar's testA test for paired yes/no data that uses only the discordant pairs: $(b-c)^2/(b+c)\approx\chi^2_1$ under marginal homogeneity.5.2
median testTwo-sample test that counts how many observations of one sample lie above the combined sample median; under $F=G$ the count is hypergeometric (Fisher's exact test on a $2\times2$ table).5.3
monotone likelihood ratioA family has MLR in $T(x)$ if $f(x;\theta')/f(x;\theta)$ is non-decreasing in $T(x)$ whenever $\theta\lt\theta'$; then one-sided UMP tests reject for large (or small) $T$.5.1 5.3
Neyman–Pearson lemmaFor simple $H_0:f_0$ against simple $H_1:f_1$, a test that rejects when $f_1(x)/f_0(x)\gt k$ (randomising on equality) and has size $\alpha$ is most powerful at level $\alpha$.1.2 4.3 5.1 7.1
null hypothesisThe claim $H_0:\theta\in\Theta_0$ that a test controls the risk of wrongly rejecting; the alternative $H_1:\theta\in\Theta_1$ is what is accepted when $H_0$ is rejected. A hypothesis is simple if it specifies a single distribution.5.1 5.2 5.3 6.2
odds ratio$\theta=\frac{p_1/(1-p_1)}{p_2/(1-p_2)}$, estimated by $ad/(bc)$; equal to $1$ exactly under independence, with $\operatorname{SE}(\log\hat\theta)\approx\sqrt{1/a+1/b+1/c+1/d}$.5.2 6.3
p-valueThe probability under $H_0$ of a result at least as extreme as the one observed; the smallest level at which the data reject $H_0$. Under a simple $H_0$ with a continuous statistic it is uniform on $(0,1)$.5.3 6.2
power function$\beta_\phi(\theta)=E_\theta\phi(X)$, the probability that the test $\phi$ rejects $H_0$ when the parameter is $\theta$. Its supremum over $\Theta_0$ is the size; on $\Theta_1$ it is the power.5.1
runs testTest of randomness of a two-symbol sequence based on the number $R$ of maximal blocks of identical symbols; with $m$ and $n$ symbols, $E[R]=1+\frac{2mn}{m+n}$ under randomness. Too few or too many runs are evidence against randomness.5.3 6.3
score testLarge-sample tests of $H_0:\theta=\theta_0$: the score test standardises the score at $\theta_0$ (for a proportion, $(\hat p-p_0)/\sqrt{p_0q_0/n}$), the Wald test standardises $\hat\theta-\theta_0$ by the estimated standard error. Both are asymptotically equivalent to the likelihood ratio test.5.1 5.2 6.3
sign testTest of a median $\theta_0$ based on $S$, the number of observations above $\theta_0$; under $H_0$, $S\sim\operatorname{Bin}(n,\tfrac12)$ for every continuous population.5.3
size of a testThe size is $\sup_{\theta\in\Theta_0}P_\theta(\text{reject }H_0)$; a test has level (significance level) $\alpha$ if its size is at most $\alpha$.5.1 5.2
Spearman's rank correlationThe Pearson correlation of the ranks, $r_s=1-\frac{6\sum d_i^2}{n(n^2-1)}$; under independence $E[r_s]=0$ and $\operatorname{Var}(r_s)=\frac1{n-1}$.5.3
test of homogeneityChi-square tests for an $r\times c$ table: independence of two attributes in one sample, or equality of the distributions of several samples with fixed sizes. Both use $\hat E_{ij}=R_iC_j/n$ and $(r-1)(c-1)$ degrees of freedom.5.2 5.3
type I errorA type I error rejects a true null hypothesis; a type II error accepts a false one. Tests fix a bound $\alpha$ on the first and minimise the probability of the second.5.1 6.2
unbiased testA test whose power is at least $\alpha$ at every alternative and at most $\alpha$ under $H_0$; the equal-tailed $z$-test is unbiased, a one-sided test of a two-sided alternative is not.5.1
uniformly most powerful testA level-$\alpha$ test whose power is at least that of every other level-$\alpha$ test at every alternative. UMP one-sided tests exist under MLR; two-sided ones usually do not.5.1
Wilcoxon signed-rank testTest of symmetry about $\theta_0$ based on $T^+$, the sum of the ranks of $|X_i-\theta_0|$ over the positive $X_i-\theta_0$; under $H_0$, $T^+$ has mean $\frac{n(n+1)}4$ and variance $\frac{n(n+1)(2n+1)}{24}$.5.3
Yates' correctionFor a $2\times2$ table, $X^2_{\mathrm Y}=n(|ad-bc|-n/2)^2/(R_1R_2C_1C_2)$: a continuity correction that brings the $\chi^2_1$ $p$-value closer to the exact one when counts are small.5.2
Ch 6 · Linear Models and Regression
TermMeaningUsed in
analysis of covarianceA one-way (or larger) ANOVA with a regression on a concomitant variable $x$ measured before treatment: $y_{ij}=\mu+\alpha_i+\beta(x_{ij}-\bar x)+\varepsilon_{ij}$. It compares adjusted means $\bar y_i-\hat\beta(\bar x_i-\bar x)$ with a reduced error variance.6.2
analysis of varianceThe partition of the total sum of squares of a linear model into parts attributed to factors and to error, with $F$ ratios of the corresponding mean squares to test the factors; e.g. one-way $\text{SST}=\text{SSB}+\text{SSW}$, $F=\text{MSB}/\text{MSW}$.1.1 1.4 6.1 6.2
best linear unbiased estimatorThe linear unbiased estimator with the smallest variance (BLUE). By the Gauss–Markov theorem it is $\lambda^{\mathsf T}\hat\beta$ for least squares $\hat\beta$, with variance $\sigma^2\lambda^{\mathsf T}(X^{\mathsf T}X)^-\lambda$.6.1 6.3
Box–Cox transformationThe family $y^{(\lambda)}=(y^\lambda-1)/\lambda$ ($\lambda\ne0$), $\log y$ ($\lambda=0$) of power transformations of a positive response, with $\lambda$ chosen by maximum likelihood to make a normal linear model fit.6.3
Cochran's theoremIf $y\sim N_n(\mu,\sigma^2I)$ and $y^{\mathsf T}y=\sum y^{\mathsf T}A_iy$ with ranks $r_i$, then the quadratic forms are independent scaled (noncentral) $\chi^2_{r_i}$ variables iff $\sum r_i=n$; the basis of every ANOVA table.6.1 6.2 7.1 8.3
coefficient of determination$R^2=1-SSE/SST$, the proportion of the variation of $y$ about its mean explained by a regression with an intercept; it equals $[\operatorname{corr}(y,\hat y)]^2$ and never decreases when a regressor is added. The adjusted version $\bar R^2=1-(1-R^2)\frac{n-1}{n-p}$ penalises extra parameters.1.1 6.3 7.2
Cook's distance$D_i=\frac{(\hat\beta_{(i)}-\hat\beta)^\top X^\top X(\hat\beta_{(i)}-\hat\beta)}{ps^2}=\frac{r_i^2}{p}\cdot\frac{h_{ii}}{1-h_{ii}}$, the change in the fit when case $i$ is deleted; values above $1$ signal an influential case.6.3
estimable functionA linear function $\lambda^{\mathsf T}\beta$ that has a linear unbiased estimator; this holds iff $\lambda$ is a linear combination of the rows of $X$. In the one-way model, contrasts $\sum c_i\alpha_i$ with $\sum c_i=0$ are estimable, single effects $\alpha_i$ are not.6.1 6.2 8.3 8.4
expected mean squaresThe expectations $E[\text{MS}]$ of the mean squares of an ANOVA table, written in terms of $\sigma^2$ and the effects or variance components. A term is tested against the mean square whose expectation equals its own when the term is absent.6.2 8.3
Gauss–Markov modelThe linear model $y=X\beta+\varepsilon$ with a known design matrix $X$, $E[\varepsilon]=0$ and $\operatorname{Cov}(\varepsilon)=\sigma^2I$: uncorrelated errors with a common variance. Adding $\varepsilon\sim N_n(0,\sigma^2I)$ gives the normal linear model.6.1
Gauss–Markov theoremIn a Gauss–Markov model (mean-zero, uncorrelated, equal-variance errors; normality not needed), the least squares estimator of every estimable function is its unique best linear unbiased estimator.6.1 6.2 6.3 8.3
generalised inverseA matrix $A^-$ with $AA^-A=A$. It exists for every matrix and is used to write solutions of the normal equations as $(X^{\mathsf T}X)^-X^{\mathsf T}y$; the Moore–Penrose inverse is the unique one satisfying three further symmetry conditions.6.1 8.4
hat matrix$H=X(X^{\mathsf T}X)^-X^{\mathsf T}$, the orthogonal projection onto the column space of $X$: symmetric, idempotent, $\operatorname{tr}H=\operatorname{rank}X$, with $\hat y=Hy$ and residuals $(I-H)y$.6.1 6.3
interaction plotA plot of the cell means $\bar y_{ij}$ against the levels of one factor, with one line per level of the other factor. Parallel lines indicate no interaction; non-parallel or crossing lines indicate that the effect of one factor depends on the level of the other.6.2
intraclass correlation$\rho=\sigma_a^2/(\sigma_a^2+\sigma^2)$, the correlation between two observations in the same group of a one-way random effects model, i.e. the share of the total variance due to groups.6.2 8.1
least squares estimatorA vector $\hat\beta$ minimising $\|y-Xb\|^2$; equivalently a solution of the normal equations. The fitted vector $X\hat\beta$ is the orthogonal projection of $y$ onto the column space of $X$.6.1 6.2 6.3 8.3
leverageThe diagonal entry $h_{ii}$ of the hat matrix $H=X(X^\top X)^{-1}X^\top$, measuring how far case $i$ lies from the centre of the design; $0\le h_{ii}\le1$, $\sum h_{ii}=p$, $\operatorname{Var}(e_i)=\sigma^2(1-h_{ii})$, and in simple regression $h_{ii}=\frac1n+\frac{(x_i-\bar x)^2}{S_{xx}}$.6.1 6.3
log-odds$\operatorname{logit}\pi=\log\frac{\pi}{1-\pi}$, a bijection from $(0,1)$ onto $\mathbb{R}$ with inverse $\pi=1/(1+e^{-\eta})$; it is the scale on which logistic regression is linear.2.3 4.3 6.3
logistic regressionThe model for binary or binomial responses in which $\operatorname{logit}\pi_i=\log\frac{\pi_i}{1-\pi_i}=x_i^\top\beta$; $e^{\beta_j}$ is the odds ratio for a unit increase in $x_j$, and the MLE is found by Newton–Raphson (iteratively reweighted least squares).6.3
mixed modelA model with both fixed and random factors; in the balanced two-way mixed model the fixed factor is tested against the interaction mean square, not against the error mean square.6.2 8.3
multicollinearityNear-linear dependence among the regressors of a linear model, so that $X^\top X$ is nearly singular; it inflates the variances of individual coefficients and makes them unstable, without harming fitted values inside the range of the data.6.3
multiple comparisonsProcedures that control the probability of any false rejection over a family of comparisons: Bonferroni (level $\alpha/m$ each), Scheffé (all contrasts, multiplier $\sqrt{(k-1)F_{k-1,\nu,\alpha}}$) and Tukey (all pairs, studentized range).6.2
normal equationsThe linear system $X^{\mathsf T}Xb=X^{\mathsf T}y$ whose solutions are exactly the least squares estimates; it is always consistent, and has a unique solution iff $X$ has full column rank.1.1 6.1 6.3 8.3
orthogonal contrastsContrasts $\sum c_i\mu_i$ and $\sum d_i\mu_i$ ($\sum c_i=\sum d_i=0$) with $\sum c_id_i/n_i=0$. Their single-degree-of-freedom sums of squares are independent under normality, and $k-1$ of them add up to the between-groups sum of squares.6.2 8.4
prediction intervalAn interval that contains a future observation $Y_0$ at $x_0$ with probability $1-\alpha$; in regression $\hat\mu_0\pm t_{n-p,\alpha/2}\,s\sqrt{1+x_0^\top(X^\top X)^{-1}x_0}$. Unlike a confidence interval for the mean response it does not shrink to zero as $n\to\infty$.6.1 6.3
random effects modelA model in which the levels of a factor are a random sample from a population of levels, so their effects are random variables (e.g. $a_i\sim N(0,\sigma_a^2)$) and inference concerns the variance $\sigma_a^2$ rather than the particular effects.6.2
studentised residualA residual divided by its estimated standard deviation: $r_i=e_i/(s\sqrt{1-h_{ii}})$ (internal) or $t_i=e_i/(s_{(i)}\sqrt{1-h_{ii}})$ with $s_{(i)}$ computed without case $i$ (external); under normal errors $t_i\sim t_{n-p-1}$.6.3
testable hypothesisA linear hypothesis $\Lambda^{\mathsf T}\beta=d$ in which every component of $\Lambda^{\mathsf T}\beta$ is estimable; it is tested by $F=\frac{(\operatorname{RSS}_0-\operatorname{RSS})/q}{\operatorname{RSS}/(n-r)}\sim F_{q,n-r}$ under normality.6.1
variance componentA variance parameter of a random effects model, such as $\sigma_a^2$ and $\sigma^2$ in $y_{ij}=\mu+a_i+\varepsilon_{ij}$; the ANOVA estimator $\hat\sigma_a^2=(\text{MSA}-\text{MSE})/n$ is unbiased but can be negative.6.2
variance inflation factor$\operatorname{VIF}_j=1/(1-R_j^2)$, where $R_j^2$ is the $R^2$ of regressing $x_j$ on the other regressors; $\operatorname{Var}(\hat\beta_j)=\frac{\sigma^2}{S_{jj}}\operatorname{VIF}_j$. With two regressors $\operatorname{VIF}=1/(1-r_{12}^2)$.6.3
Ch 7 · Multivariate Analysis
TermMeaningUsed in
canonical correlationThe maximal correlation between a linear combination of one set of variables $X$ and a linear combination of another set $Y$, and the successive maxima under uncorrelatedness constraints; their squares are the eigenvalues of $\Sigma_{11}^{-1}\Sigma_{12}\Sigma_{22}^{-1}\Sigma_{21}$.7.3
covariance matrix$\Sigma=E[(X-\mu)(X-\mu)^{\mathsf T}]$, with entries $\operatorname{Cov}(X_i,X_j)$. It is symmetric and nonnegative definite, $\operatorname{Var}(a^{\mathsf T}X)=a^{\mathsf T}\Sigma a$, and it is singular exactly when some linear combination of the components is constant.6.2 7.1 7.2 7.3
dendrogramThe tree diagram of a hierarchical clustering: each merge of two clusters is drawn as a bar at the height (dissimilarity) at which it happens; cutting the tree at a level gives a partition.7.3
expected cost of misclassificationFor a rule allocating observations to $\mathcal P_1$ or $\mathcal P_2$: $c(2\mid1)\pi_1P(2\mid1)+c(1\mid2)\pi_2P(1\mid2)$. It is minimised by allocating to $\mathcal P_1$ when $f_1(x)/f_2(x)\ge c(1\mid2)\pi_2/(c(2\mid1)\pi_1)$.7.3
Fisher's linear discriminantThe linear combination $\hat a^{\mathsf T}x$ with $\hat a=S_{\text{pl}}^{-1}(\bar x_1-\bar x_2)$ that maximises the squared difference of the two group means relative to the pooled within-group variance; the maximum is the sample Mahalanobis distance $D^2$.7.1 7.3
Gaussian graphical modelA multivariate normal model drawn as a graph with an edge between $i$ and $j$ exactly when the precision-matrix entry $\omega_{ij}\ne0$, i.e. when $X_i$ and $X_j$ are not conditionally independent given all the other variables.7.2
Hotelling's T²$T^2=n(\bar X-\mu_0)^{\mathsf T}S^{-1}(\bar X-\mu_0)$, the multivariate analogue of $t^2$; for normal data $\frac{n-p}{p(n-1)}T^2\sim F_{p,n-p}$ under $H_0:\mu=\mu_0$. It is the likelihood ratio test and the maximum of $t_a^2$ over all linear combinations.7.1 7.2
idempotent matrixA square matrix with $A^2=A$. A symmetric idempotent matrix is an orthogonal projection, has eigenvalues $0$ and $1$ only, and satisfies $\operatorname{rank}A=\operatorname{tr}A$; $Z^{\mathsf T}AZ$ is chi-square for $Z\sim N_n(0,I)$ exactly when $A$ is idempotent.6.1 6.3 7.1 7.3
k-means clusteringPartitioning data into $K$ clusters so as to minimise the within-cluster sum of squares $\sum_k\sum_{i\in C_k}\|x_i-\bar x_k\|^2$; usually computed by Lloyd's algorithm (assign to the nearest centre, move centres to cluster means), which converges to a local minimum.7.3
linkageThe rule defining the distance between two clusters in hierarchical clustering: the smallest (single), largest (complete) or average (average) pairwise dissimilarity, or the distance between cluster means (centroid). Single, complete and average linkage give monotone dendrograms; centroid linkage can invert.7.3
Mahalanobis distance$\Delta(x)=\sqrt{(x-\mu)^{\mathsf T}\Sigma^{-1}(x-\mu)}$, distance measured in units that account for variances and correlations; it is invariant under nonsingular affine changes of coordinates, and $\Delta^2(X)\sim\chi^2_p$ for $X\sim N_p(\mu,\Sigma)$.6.3 7.1 7.2 7.3
mean vectorThe vector $\mu=E[X]=(E[X_1],\dots,E[X_p])^{\mathsf T}$ of the means of the components of a random vector; for a sample, $\bar x=\frac1n\sum x_i$.6.1 7.1 7.2 7.3
multiple correlation coefficient$R_{1\cdot2\cdots p}$, the largest correlation between $X_1$ and any linear combination of $X_2,\dots,X_p$; $R^2=\sigma_{12}\Sigma_{22}^{-1}\sigma_{21}/\sigma_{11}$, and the sample version is the square root of the regression $R^2$.6.3 7.2 7.3
multivariate normal distributionThe law $N_p(\mu,\Sigma)$ of a random vector all of whose linear combinations $a^{\mathsf T}X$ are normal. It is determined by $\mu$ and $\Sigma$, has a density iff $\Sigma\gt0$, and its marginals, conditionals and linear images are again normal.7.1 7.2 7.3
noncentral chi-square distribution$\chi^2_r(\delta)$, the law of $\sum_{i=1}^rY_i^2$ with independent $Y_i\sim N(m_i,1)$ and $\delta=\sum m_i^2$; mean $r+\delta$, variance $2r+4\delta$. It is the law of test statistics under alternatives.7.1
partial correlationThe correlation of two variables after the linear effect of a set of other variables has been removed from both; it is the correlation of the two regression residuals, e.g. $\rho_{12\cdot3}=\frac{\rho_{12}-\rho_{13}\rho_{23}}{\sqrt{(1-\rho_{13}^2)(1-\rho_{23}^2)}}$. For normal data it is the conditional correlation.7.1 7.2
precision matrixThe inverse $\Omega=\Sigma^{-1}$ of a covariance matrix. Its diagonal blocks invert the conditional covariances, and for a normal vector $\omega_{ij}=0$ means that $X_i$ and $X_j$ are conditionally independent given the other variables.7.1 7.2
principal componentA linear combination $e_k^{\mathsf T}X$ of the variables, where $e_k$ is a unit eigenvector of the covariance (or correlation) matrix for its $k$-th largest eigenvalue $\lambda_k$; it has the largest variance, $\lambda_k$, among unit-length combinations uncorrelated with the earlier components.7.3
scree plotThe plot of the ordered eigenvalues $\hat\lambda_1\ge\hat\lambda_2\ge\cdots$ of a sample covariance or correlation matrix against their rank, used to judge how many principal components to keep (look for the "elbow").7.3
simultaneous confidence intervalsA family of intervals that all cover their targets together with probability at least $1-\alpha$, e.g. the $T^2$ intervals $a^{\mathsf T}\bar x\pm c\sqrt{a^{\mathsf T}Sa/n}$ for every $a$, or Bonferroni intervals for a few chosen combinations.7.2
suppressor variableA predictor that is (nearly) uncorrelated with the response but raises the multiple correlation, because it removes from another predictor a component that is irrelevant to the response.6.3 7.2
ultrametricA dissimilarity satisfying the strong triangle inequality $d(i,k)\le\max\{d(i,j),d(j,k)\}$. The cophenetic distances of a monotone dendrogram form an ultrametric.7.3
Wilks' lambda$\Lambda=|W|/|W+B|$, the ratio of the within-groups to the total generalised variance; small values reject equality of several mean vectors (MANOVA). For two groups it is a monotone function of the two-sample $T^2$.7.2
Wishart distribution$W_p(m,\Sigma)$, the law of $\sum_{i=1}^mZ_iZ_i^{\mathsf T}$ with $Z_i$ iid $N_p(0,\Sigma)$; the matrix analogue of $\sigma^2\chi^2_m$. For a normal sample, $\sum(X_i-\bar X)(X_i-\bar X)^{\mathsf T}\sim W_p(n-1,\Sigma)$, independent of $\bar X$.7.1 7.2
Ch 8 · Sample Surveys and Design of Experiments
TermMeaningUsed in
auxiliary variableA variable $x$ known for every population unit (or at least its total $X$), used to improve selection (PPS) or estimation (ratio and regression estimators) of a related study variable $y$.8.2
balanced incomplete block designAn arrangement of $v$ treatments in $b$ blocks of $k\lt v$ distinct treatments, each treatment replicated $r$ times and each pair together in $\lambda$ blocks; $bk=vr$, $\lambda(v-1)=r(k-1)$ and $b\ge v$ (Fisher's inequality).8.4
C-matrixThe matrix $C=R-NK^{-1}N^{\mathsf T}$ of the reduced normal equations $C\hat\tau=Q$ of a block design; it is symmetric, positive semidefinite, has zero row sums, and $\operatorname{rank}C=v-1$ iff the design is connected.8.4
completely randomised designA design in which the treatments, with prescribed replications $r_i$, are allotted to all $N$ units completely at random; it is analysed by one-way ANOVA with $N-v$ error degrees of freedom.6.2 8.3 8.4
confoundingArranging a factorial experiment in incomplete blocks so that chosen effects (usually high-order interactions) coincide with block differences and are sacrificed; total if the same effects are confounded in every replicate, partial if different effects are confounded in different replicates.8.4
connected designA block design in which any two treatments are linked by a chain treatment–block–treatment–… of incidences; equivalently every treatment contrast is estimable, or $\operatorname{rank}C=v-1$.8.4
cumulative total methodPPS selection by listing cumulative sizes $T_i$ and taking the unit with $T_{i-1}\lt R\le T_i$ for a random integer $R\in\{1,\dots,X\}$.8.2
efficiency factorFor a BIBD, $E=\frac{\lambda v}{rk}=\frac{v(k-1)}{k(v-1)}\lt1$: the ratio of the variance of an elementary contrast in an RBD with the same replication and error variance to that in the BIBD.8.4
experimental unitThe smallest piece of material (a plot, a patient, a batch) to which a treatment is applied independently of the others. Repeated measurements on one unit are subsamples, not replicates.6.2 8.3
finite population correctionThe factor $1-f=1-n/N$ in the variance $(1-f)S^2/n$ of the sample mean under sampling without replacement; it is negligible when the sampling fraction $n/N$ is small.2.1 8.1
Fisher's inequalityIn every BIBD the number of blocks is at least the number of treatments, $b\ge v$ (hence $r\ge k$), because $NN^{\mathsf T}=(r-\lambda)I+\lambda J$ is nonsingular.8.4
generalised interactionThe product of two effects with squared letters deleted, e.g. $ABC\cdot BCD=AD$. When two effects are confounded with blocks, their generalised interaction is confounded as well.8.4
Hansen–Hurwitz estimatorUnder PPS with replacement, $\hat Y_{\mathrm{HH}}=\frac1n\sum y_j/p_j$; unbiased for the total, with variance $\frac1n\sum p_i(Y_i/p_i-Y)^2$.8.2
Horvitz–Thompson estimator$\hat Y_{\mathrm{HT}}=\sum_{i\in s}y_i/\pi_i$, unbiased for the population total under any design in which every inclusion probability $\pi_i$ is positive.8.2
incidence matrixFor a block design, the $v\times b$ matrix $N=(n_{ij})$ with $n_{ij}$ the number of plots of block $j$ that receive treatment $i$; its row sums are the replications $r_i$ and its column sums the block sizes $k_j$.8.4
inclusion probabilityFor a sampling design, $\pi_i=P(\text{unit } i \text{ is in the sample})$; the second-order inclusion probability is $\pi_{ij}=P(i \text{ and } j \text{ both in the sample})$.8.1 8.2
Lahiri's methodPPS selection by drawing a random unit $i$ and a random integer $R\le M$ (with $M\ge\max X_i$) and accepting $i$ if $R\le X_i$, repeating until acceptance.8.2
Latin square designA design for $v$ treatments on $v^2$ units classified by rows and columns, each treatment occurring once in every row and every column; it controls two nuisance factors and leaves $(v-1)(v-2)$ error degrees of freedom.8.3
local controlGrouping experimental units into homogeneous blocks and comparing treatments within blocks, so that variation between blocks is removed from the experimental error.8.3
missing plot techniqueEstimating a lost observation by the value that minimises the error sum of squares of the completed data (for an RBD, $\hat x=\frac{vT+bB-G}{(v-1)(b-1)}$), then analysing the completed table with one error df fewer and the treatment SS corrected for its upward bias.8.3
Neyman allocationThe allocation $n_h\propto N_hS_h$ that minimises the variance of the stratified mean for fixed $n$; with unit costs $c_h$ the optimum is $n_h\propto N_hS_h/\sqrt{c_h}$.8.1
orthogonal block designA block design with proportional frequencies $n_{ij}=r_ik_j/n$; treatment and block effects are then estimated independently and adjustment for blocks changes nothing. Every RBD is orthogonal; no incomplete design is.8.4
PPS samplingSampling in which unit $i$ is selected with probability proportional to a known size $X_i$, i.e. $p_i=X_i/X$ per draw (with replacement) or $\pi_i\propto X_i$ (without replacement).8.2
proportional allocationAllocation of a stratified sample with $n_h=nN_h/N$; it makes the sample self-weighting (the stratified mean equals the plain sample mean).8.1
randomisationAllotting treatments to units by a chance mechanism, all allocations permitted by the design being equally likely. It makes treatment comparisons unbiased and the error mean square a valid estimate of the error variance, without assuming normality.5.1 8.3
randomised block designA design with $b$ blocks of $v$ units in which every treatment appears once in every block, randomised independently within blocks; analysed by the additive two-way ANOVA with $(v-1)(b-1)$ error degrees of freedom.6.2 8.3 8.4
ratio estimator$\bar y_R=(\bar y/\bar x)\bar X$; slightly biased, with first-order MSE $\frac{1-f}{n}(S_y^2+R^2S_x^2-2R\rho S_xS_y)$, better than $\bar y$ iff $\rho\gt C_x/(2C_y)$.2.3 6.3 8.2
regression estimator$\bar y_{lr}=\bar y+b(\bar X-\bar x)$ with $b$ the sample slope; to first order its MSE is $\frac{1-f}{n}S_y^2(1-\rho^2)$, never larger than that of $\bar y$ or $\bar y_R$.8.2
relative efficiency of a designThe ratio of the error variances of two designs for the same comparisons; e.g. RBD versus CRD is estimated by $\frac{(b-1)M_b+b(v-1)M_e}{(bv-1)M_e}$, which exceeds $1$ iff $M_b\gt M_e$.8.3
sampling frameThe list (or map) of population units from which a sample is actually selected; units missing from the frame can never be sampled.8.1
Sen–Yates–Grundy estimatorFor fixed-size designs, the variance estimator $\sum_{i\lt j\in s}\frac{\pi_i\pi_j-\pi_{ij}}{\pi_{ij}}\left(\frac{y_i}{\pi_i}-\frac{y_j}{\pi_j}\right)^2$ of the Horvitz–Thompson estimator; non-negative when $\pi_i\pi_j\ge\pi_{ij}$.8.2
simple random samplingDrawing $n$ units so that every subset of size $n$ is equally likely (without replacement, SRSWOR), or by $n$ independent equiprobable draws (with replacement, SRSWR).8.1 8.2 8.3
stratified samplingPartitioning the population into strata and drawing an independent random sample from each; the mean is estimated by $\bar y_{\mathrm{st}}=\sum W_h\bar y_h$ with $W_h=N_h/N$.8.1 8.2
systematic samplingSelecting every $k$-th unit of an ordered list after a random start in $\{1,\dots,k\}$; efficient under a trend, poor when $k$ matches a periodic pattern.8.1 8.2
uniformity trialAn experiment in which all units receive the same treatment, so that the observed variation is pure unit-to-unit variation; used to study field heterogeneity and the efficiency of block shapes.8.3
Yates' algorithmA method for the contrasts of a $2^k$ factorial: list the treatment totals in standard order and $k$ times replace the column by the sums of successive pairs followed by their differences; the final column holds $G$ and the contrasts $[A],[B],[AB],\dots$.8.4
Ch 9 · Reliability, Linear Programming and Queues
TermMeaningUsed in
artificial variableAn extra nonnegative variable added to a $\ge$ or $=$ constraint to supply a starting basis; the two-phase method minimises their sum first, and the Big-M method penalises each by a very large cost $M$. The original problem is feasible iff they can all be driven to $0$.9.2
assignment problemThe problem of assigning $n$ agents to $n$ tasks, one each, at least total cost; a transportation problem with all supplies and demands $1$, solved by the Hungarian method of row and column reductions.9.2
basic feasible solutionFor $Ax=b$, $x\ge0$ with $A$ of rank $m$: a solution obtained by choosing $m$ linearly independent columns (a basis), setting the other variables to $0$ and solving, which happens to be $\ge0$. It is degenerate if some basic variable is $0$.9.2
censoringIncomplete observation of a lifetime: we know only that it exceeds some time. A Type I (time-censored) test stops at a fixed time $t_0$, so the number of failures is random; a Type II (failure-censored) test stops at the $r$-th failure, so the duration is random. Censored items contribute $R(y)$ to the likelihood, failures contribute $f(y)$.4.1 9.1
complementary slacknessFeasible primal and dual solutions are both optimal iff $y_i(b-Ax)_i=0$ and $x_j(A^{\mathsf T}y-c)_j=0$ for all $i,j$: a positive dual price forces its constraint to be tight, and a positive variable forces its dual constraint to be tight.9.2
cumulative hazard$H(t)=\int_0^t h(u)\,du=-\log R(t)$. For a lifetime it tends to $\infty$, $H(T)\sim\operatorname{Exp}(1)$, and the cumulative hazards of independent components in series add.9.1
dual problemThe linear program attached to a primal LP with one variable per primal constraint: the dual of $\max c^{\mathsf T}x$, $Ax\le b$, $x\ge0$ is $\min b^{\mathsf T}y$, $A^{\mathsf T}y\ge c$, $y\ge0$. Its optimal value equals the primal one when either has an optimum (strong duality).9.2
economic order quantityThe order size $Q^*=\sqrt{2KD/h}$ that minimises ordering plus holding cost $KD/Q+hQ/2$ for steady demand $D$, ordering cost $K$ and holding cost $h$ per unit per unit time; at $Q^*$ the two costs are equal.9.3
Erlang B formulaThe fraction of customers lost in the loss system M/M/c/c: $B(c,a)=\frac{a^c/c!}{\sum_{k\le c}a^k/k!}$, computed by $B(c,a)=\frac{aB(c-1,a)}{c+aB(c-1,a)}$; it is insensitive to the service-time distribution.3.3 9.3
Erlang C formulaIn an M/M/c queue with offered load $a$ and $\rho=a/c\lt1$, the probability that an arrival must wait: $C(c,a)=\pi_0\,\frac{a^c}{c!(1-\rho)}$; then $W_q=C(c,a)/(c\mu-\lambda)$.9.3
extreme pointA point of a convex set that is not the midpoint of any segment of two different points of the set — a corner. For $\{x:Ax=b,\ x\ge0\}$ the extreme points are exactly the basic feasible solutions.7.1 9.2
hazard rateThe instantaneous failure rate of a unit that has survived to age $t$: $h(t)=f(t)/R(t)=-\frac{d}{dt}\log R(t)$, so that $P(t\lt T\le t+\Delta\mid T\gt t)\approx h(t)\Delta$. It is a rate (it may exceed $1$), and it determines the lifetime law.2.1 9.1
increasing failure rateA lifetime is IFR (DFR) if $R(t+x)/R(t)$ is nonincreasing (nondecreasing) in $t$ for every $x\gt0$ — for laws with a density, if the hazard rate is nondecreasing (nonincreasing). The exponential law is both; Weibull and gamma laws are IFR for shape $\ge1$ and DFR for shape $\le1$.9.1
k-out-of-n systemA system that works while at least $k$ of its $n$ components work: $k=n$ is a series and $k=1$ a parallel system. With iid components of reliability $p$, its reliability is $P(\operatorname{Bin}(n,p)\ge k)$, and its lifetime is the $(n-k+1)$-th order statistic of the component lifetimes.4.1 9.1
Kendall notationThe shorthand $A/B/c/K/N$ for a queue: inter-arrival law $A$, service law $B$ ($M$ = exponential, $D$ = constant, $E_k$ = Erlang, $G$ = general), $c$ servers, capacity $K$ and source population $N$ (the last two $\infty$ if omitted).9.3
linear programming problemThe problem of maximising or minimising a linear function $c^{\mathsf T}x$ over the set of $x$ satisfying finitely many linear equations and inequalities; its feasible region is a convex polyhedron.9.2
Little's law$L=\lambda W$: the long-run average number in a system equals the arrival rate (of admitted customers) times the mean time each spends there. It holds for any arrival and service laws and any discipline; likewise $L_q=\lambda W_q$.9.3
mean residual lifeThe expected remaining life of a unit that has survived to age $t$: $m(t)=E[T-t\mid T\gt t]=\frac{1}{R(t)}\int_t^\infty R(u)\,du$. It is constant only for the exponential law and nonincreasing for IFR laws.9.1
mean time to failureThe expected lifetime $\mu=E[T]=\int_0^\infty R(t)\,dt$; for a system, the integral of the system reliability function. For a $k$-out-of-$n$ system of iid $\operatorname{Exp}(\lambda)$ parts it is $\frac1\lambda\sum_{j=k}^n\frac1j$.9.1
PASTAPoisson Arrivals See Time Averages: when arrivals form a Poisson process not anticipating the future of the system, the fraction of arrivals that find the system in a state equals the fraction of time it spends in that state.3.3 9.3
Pollaczek–Khinchine formulaFor the M/G/1 queue with $\rho=\lambda E[S]\lt1$: $W_q=\frac{\lambda E[S^2]}{2(1-\rho)}$, so the mean wait grows with the variance of the service time; constant service halves the M/M/1 queue.9.3
reliability functionFor a lifetime $T$, the probability of surviving beyond age $t$: $R(t)=P(T\gt t)=1-F(t)$. It is nonincreasing, $R(0)=1$ for a lifetime with a density, and $R(t)=e^{-H(t)}$.1.3 1.4 2.1 9.1
simplex methodDantzig's method for linear programs: move from one basic feasible solution to an adjacent one with a better objective, choosing the entering variable by its net evaluation $z_j-c_j$ and the leaving variable by the minimum ratio test, until all $z_j-c_j\ge0$ (maximisation) or unboundedness is detected.9.2
structure functionThe map $\varphi:\{0,1\}^n\to\{0,1\}$ giving the state of a system from the states of its $n$ components. The system is coherent if $\varphi$ is nondecreasing in each component and every component is relevant; series is $\min_ix_i$, parallel is $\max_ix_i$.9.1
total time on testThe sum of the times all items spent on test in a life test, e.g. $T_r=\sum_{i=1}^rX_{(i)}+(n-r)X_{(r)}$ in a Type II test without replacement. For exponential lifetimes of mean $\theta$, the MLE of $\theta$ is the total time on test divided by the number of failures, and $2T_r/\theta\sim\chi^2_{2r}$.4.1 4.3 9.1
traffic intensityFor arrival rate $\lambda$, service rate $\mu$ per server and $c$ servers, the offered load is $a=\lambda/\mu$ and the traffic intensity (utilisation) is $\rho=\lambda/(c\mu)$; an infinite-capacity M/M/c queue is stable iff $\rho\lt1$.9.3
transportation problemThe LP of shipping supplies $a_i$ from $m$ sources to meet demands $b_j$ at $n$ destinations at least total cost $\sum c_{ij}x_{ij}$; when balanced it always has an optimal basic solution with at most $m+n-1$ positive shipments, integral if the data are integers.9.2