Glossary
Every technical word in the course and what it means in plain language, with flashcards for practice.
Every technical term in the course, with a plain-language meaning. On every page these words are marked with a dotted underline: hover or tap one to see its meaning. The “Used in” links point to the sections where each term appears.
Ch 1 · ProbabilityCh 2 · Distributions and Limit TheoremsCh 3 · Markov Chains and Stochastic ProcessesCh 4 · Sampling Distributions and EstimationCh 5 · Testing of HypothesesCh 6 · Linear Models and RegressionCh 7 · Multivariate AnalysisCh 8 · Sample Surveys and Design of ExperimentsCh 9 · Reliability, Linear Programming and Queues
Ch 1 · Probability
| Term | Meaning | Used in |
|---|---|---|
| almost surely | With probability $1$. An event of probability $0$ may still occur, so 'almost surely' is weaker than 'surely'. | 1.2 1.3 1.4 2.2 |
| Bayes' theorem | For a partition $\{B_i\}$, $P(B_j\mid A)=P(A\mid B_j)P(B_j)/\sum_iP(A\mid B_i)P(B_i)$: posterior $\propto$ likelihood $\times$ prior. | 1.2 1.4 2.1 4.3 |
| bivariate normal distribution | The joint law with elliptical density determined by $\mu_1,\mu_2,\sigma_1,\sigma_2,\rho$; marginals and conditionals are normal, $E[Y\mid X]$ is linear, and $\rho=0$ gives independence. | 1.4 4.1 5.2 5.3 |
| Borel–Cantelli lemma | If $\sum P(A_n)\lt\infty$ then a.s. only finitely many $A_n$ occur; if the $A_n$ are independent and $\sum P(A_n)=\infty$ then a.s. infinitely many occur. | 1.2 2.3 |
| coefficient of variation | $\mathrm{CV}=s/\bar x$ for ratio-scale data with positive mean: a unit-free measure of relative spread, often given as a percentage. | 1.1 6.3 8.1 8.2 |
| conditional expectation | $E[X\mid Y=y]$ is the mean of the conditional law of $X$ given $Y=y$; $E[X\mid Y]=m(Y)$ is a random variable with $E[E[X\mid Y]]=E[X]$ and is the best mean-square predictor of $X$ from $Y$. | 1.4 |
| conditional probability | $P(A\mid B)=P(A\cap B)/P(B)$ for $P(B)\gt0$: the probability of $A$ once $B$ is known to have occurred. As a function of $A$ it is again a probability measure. | 1.2 2.1 3.3 4.2 |
| convolution | The law of a sum of independent variables: $f_{X+Y}(z)=\int f_X(x)f_Y(z-x)\,dx$, or $\sum_kp_X(k)p_Y(n-k)$; pgfs and mgfs multiply. | 1.4 2.1 |
| correlation coefficient | Pearson's $r=s_{xy}/(s_xs_y)$, the cosine of the angle between the centred data vectors. It lies in $[-1,1]$ and equals $\pm1$ iff the points are collinear; it measures linear association only. | 1.1 1.4 2.3 5.3 |
| covariance | $\operatorname{Cov}(X,Y)=E[(X-\mu_X)(Y-\mu_Y)]=E[XY]-E[X]E[Y]$; zero for independent variables, but zero covariance does not imply independence. | 1.1 1.4 2.1 2.2 |
| distribution function | $F(x)=P(X\le x)$: nondecreasing, right-continuous, with limits $0$ and $1$; its jump at $x$ is $P(X=x)$. Every such function is the cdf of some random variable. | 1.2 1.3 1.4 2.1 |
| exchangeable | A sequence of events or variables whose joint probabilities do not change when the order is permuted, e.g. the colours drawn from Pólya's urn. Exchangeable variables need not be independent. | 1.2 2.1 4.1 4.2 |
| inclusion–exclusion | $P(\bigcup A_i)=S_1-S_2+S_3-\cdots$, where $S_k$ sums the probabilities of all $k$-fold intersections; truncations give the Bonferroni bounds. | 1.2 1.4 8.4 |
| interquartile range | $\mathrm{IQR}=Q_3-Q_1$, the spread of the middle half of the data. Unlike the range or the SD, it is not affected by a few extreme values. | 1.1 1.3 |
| Jacobian | For a change of variables $(x,y)=(x(u,v),y(u,v))$, the determinant $J=\partial(x,y)/\partial(u,v)$; the new density is $f(x(u,v),y(u,v))\,|J|$. | 1.4 2.1 3.3 4.1 |
| joint distribution | The law of a random vector, given by $F(x,y)=P(X\le x,\,Y\le y)$, a joint pmf $p(x,y)$ or a joint density $f(x,y)$; it determines the marginals but not conversely. | 1.4 2.1 3.3 4.1 |
| kurtosis | $\beta_2=m_4/m_2^2$, a measure of tail weight; it equals $3$ for the normal distribution and always satisfies $\beta_2\ge1+\beta_1$. | 1.1 1.3 2.1 2.3 |
| law of total variance | $\operatorname{Var}X=E[\operatorname{Var}(X\mid Y)]+\operatorname{Var}(E[X\mid Y])$: average within-group variance plus variance of the group means. | 1.4 2.1 3.3 4.2 |
| marginal distribution | The law of one component of a random vector, obtained by summing or integrating the other variables out: $f_X(x)=\int f(x,y)\,dy$. | 1.4 4.3 |
| mixed distribution | A law whose cdf is $\alpha F_d+(1-\alpha)F_c$ with $0\lt\alpha\lt1$, $F_d$ discrete (jumps) and $F_c$ with a density, e.g. a lifetime with an atom at $0$. | 1.3 |
| moment generating function | $M_X(t)=E[e^{tX}]$. If it is finite near $0$, all moments exist, $E[X^r]=M_X^{(r)}(0)$, and $M_X$ determines the distribution. | 1.3 2.2 7.1 |
| mutual independence | Events are mutually independent if the probability of every finite intersection of distinct events is the product of their probabilities; pairwise independence (products of pairs only) is weaker. | 1.2 1.4 4.1 6.2 |
| outlier | An observation far from the bulk of the data; in a box plot, any value below $Q_1-1.5\,\mathrm{IQR}$ or above $Q_3+1.5\,\mathrm{IQR}$. | 1.1 5.3 6.3 7.1 |
| probability density function | A function $f\ge0$ with $F(x)=\int_{-\infty}^xf(t)\,dt$; probabilities are integrals of $f$, so $f$ itself may exceed $1$. | 1.3 |
| probability generating function | $G_X(s)=E[s^X]=\sum_kP(X=k)s^k$ for $X$ in $\{0,1,2,\dots\}$; $P(X=k)=G^{(k)}(0)/k!$, $E[X]=G'(1)$, $\operatorname{Var}X=G''(1)+G'(1)-G'(1)^2$. | 1.3 5.3 9.3 |
| probability integral transform | If $X$ has a continuous cdf $F$, then $F(X)$ is uniform on $(0,1)$. It fails when $F$ has jumps. | 1.3 4.1 4.3 5.1 |
| probability space | A triple $(\Omega,\mathcal{F},P)$: a sample space, a σ-field of events, and a countably additive $P$ with $P\ge0$ and $P(\Omega)=1$ (Kolmogorov's axioms). | 1.2 1.3 1.4 2.1 |
| quantile function | $Q(u)=\inf\{x:F(x)\ge u\}$ for $0\lt u\lt1$. It satisfies $Q(u)\le x\iff u\le F(x)$, and $Q(U)$ has cdf $F$ when $U$ is uniform on $(0,1)$. | 1.3 2.3 4.1 |
| random variable | A function $X:\Omega\to\mathbb{R}$ on a probability space with $\{X\le x\}$ an event for every $x$; then $\{X\in B\}$ is an event for every Borel set $B$. | 1.3 1.4 2.1 2.2 |
| regression coefficient | The slope $b_{yx}=s_{xy}/s_x^2$ of the least-squares line of $y$ on $x$ (or $b_{xy}=s_{xy}/s_y^2$ for $x$ on $y$); $b_{yx}b_{xy}=r^2$. | 1.1 1.4 6.2 7.1 |
| skewness | Asymmetry of a distribution. The moment coefficient is $\gamma_1=m_3/m_2^{3/2}$, which is positive when the right tail is longer; Pearson's and Bowley's coefficients are alternatives. | 1.1 1.3 2.1 2.3 |
| trimmed mean | The mean after discarding the $k=\lfloor n\alpha\rfloor$ smallest and $k$ largest observations. It lies between the mean ($\alpha=0$) and the median in robustness. | 1.1 |
| σ-field | A collection of subsets of $\Omega$ that contains $\Omega$ and is closed under complements and countable unions; its members are the events. A σ-field with $k$ atoms has $2^k$ sets. | 1.2 1.3 |
Ch 2 · Distributions and Limit Theorems
| Term | Meaning | Used in |
|---|---|---|
| almost-sure convergence | $X_n\xrightarrow{a.s.}X$ if $P(X_n\to X)=1$: almost every sample path converges. Equivalent to $P(\sup_{m\ge n}|X_m-X|\gt\varepsilon)\to0$ for every $\varepsilon\gt0$. | 1.2 2.3 |
| beta distribution | The law $\operatorname{Beta}(a,b)$ on $(0,1)$ with density $x^{a-1}(1-x)^{b-1}/B(a,b)$ and mean $a/(a+b)$. It models random proportions; $X/(X+Y)$ is beta when $X,Y$ are independent gamma variables with a common rate. | 2.1 4.1 7.1 7.2 |
| beta–binomial distribution | The law of a binomial count whose success probability is itself random with a beta law: $P(X=k)=\binom nkB(a+k,b+n-k)/B(a,b)$. It has the binomial mean $na/(a+b)$ but a larger variance. | 2.1 4.1 4.3 |
| binomial distribution | The law $\operatorname{Bin}(n,p)$ of the number of successes in $n$ independent trials, each a success with probability $p$: $P(X=k)=\binom nkp^k(1-p)^{n-k}$, with mean $np$ and variance $np(1-p)$. | 1.3 2.1 2.2 3.3 |
| Cantelli's inequality | The one-sided Chebyshev inequality $P(X-\mu\ge\lambda)\le\frac{\sigma^2}{\sigma^2+\lambda^2}$ for $\lambda\gt0$; sharp for a two-point law. | 2.2 |
| Cauchy–Schwarz inequality | $(E[XY])^2\le E[X^2]E[Y^2]$, with equality iff $aX=bY$ a.s. for some $(a,b)\ne(0,0)$; it implies $|\rho|\le1$. Hölder's inequality $E|XY|\le\|X\|_p\|Y\|_q$ ($\frac1p+\frac1q=1$) generalises it. | 1.1 1.4 2.1 2.2 |
| central limit theorem | For iid $X_j$ with mean $\mu$ and finite variance $\sigma^2\gt0$, $\sqrt n(\bar X_n-\mu)/\sigma\xrightarrow{d}N(0,1)$ (Lindeberg–Lévy); Lyapunov's and Lindeberg's conditions extend it to non-identical summands. | 2.1 2.2 2.3 3.3 |
| characteristic function | The function $\varphi_X(t)=E[e^{itX}]=E[\cos tX]+iE[\sin tX]$, $t\in\mathbb{R}$. It exists for every random variable, determines the law, turns sums of independent variables into products, and its derivatives at $0$ give the moments: $\varphi^{(k)}(0)=i^kE[X^k]$. | 1.3 2.1 2.2 2.3 |
| Chebyshev's inequality | For $X$ with mean $\mu$ and variance $\sigma^2$, $P(|X-\mu|\ge k\sigma)\le1/k^2$; sharp for a three-point law, useless for $k\le1$. | 2.2 2.3 5.3 |
| continuity correction | When approximating an integer-valued law by a normal law, $P(X\le b)$ is replaced by the normal area up to $b+\frac12$ and $P(X\ge a)$ by the area from $a-\frac12$. | 2.2 2.3 5.2 5.3 |
| convergence in distribution | $X_n\xrightarrow{d}X$ if $F_{X_n}(x)\to F_X(x)$ at every point where $F_X$ is continuous; only the laws matter. The weakest of the standard modes of convergence. | 1.3 2.2 2.3 4.1 |
| convergence in mean | $X_n\to X$ in $r$-th mean (in $L^r$) if $E|X_n-X|^r\to0$; for $r=2$, mean-square convergence. It implies convergence in probability but neither implies nor is implied by almost-sure convergence. | 2.3 |
| convergence in probability | $X_n\xrightarrow{P}X$ if $P(|X_n-X|\gt\varepsilon)\to0$ for every $\varepsilon\gt0$. Implied by almost-sure and by $r$-th mean convergence; implies convergence in distribution. | 2.2 2.3 5.2 |
| delta method | If $\sqrt n(T_n-\theta)\xrightarrow{d}N(0,\tau^2)$ and $g$ is differentiable at $\theta$, then $\sqrt n(g(T_n)-g(\theta))\xrightarrow{d}N(0,g'(\theta)^2\tau^2)$; when $g'(\theta)=0$ a second-order version gives a $\chi^2_1$ limit at scale $n$. | 2.3 4.1 4.2 4.3 |
| gamma distribution | The law $\operatorname{Gamma}(\alpha,\lambda)$ with density $\lambda^\alpha x^{\alpha-1}e^{-\lambda x}/\Gamma(\alpha)$ on $(0,\infty)$; mean $\alpha/\lambda$, variance $\alpha/\lambda^2$. For integer $\alpha=n$ it is the law of a sum of $n$ independent $\operatorname{Exp}(\lambda)$ waiting times; $\chi^2_n=\operatorname{Gamma}(n/2,1/2)$. | 1.3 1.4 2.1 4.1 |
| geometric distribution | The law of the number of independent trials up to the first success, $P(T=k)=(1-p)^{k-1}p$ on $\{1,2,\dots\}$ (or of the number of failures before it, on $\{0,1,\dots\}$). It is the only memoryless law on the positive integers. | 1.4 2.1 2.2 3.2 |
| hypergeometric distribution | The law of the number of successes in a sample of $n$ drawn without replacement from $N$ items of which $K$ are successes: $P(X=k)=\binom Kk\binom{N-K}{n-k}/\binom Nn$. Mean $np$ and variance $np(1-p)\frac{N-n}{N-1}$ with $p=K/N$. | 2.1 4.2 5.2 5.3 |
| inversion formula | The formula recovering a distribution from its characteristic function; if $\int|\varphi|\lt\infty$, the density is $f(x)=\frac1{2\pi}\int e^{-itx}\varphi(t)\,dt$. It implies that the characteristic function determines the law (uniqueness). | 1.3 2.2 9.1 |
| Jensen's inequality | For convex $g$ and integrable $X$, $g(E[X])\le E[g(X)]$ (reversed for concave $g$), with equality for strictly convex $g$ only when $X$ is constant. Gives $E[X^2]\ge(EX)^2$, $E[\log X]\le\log E[X]$ and Lyapunov's inequality. | 1.3 2.2 4.2 4.3 |
| Lévy's continuity theorem | $X_n\xrightarrow{d}X$ iff $\varphi_{X_n}(t)\to\varphi_X(t)$ for every $t$; and if $\varphi_{X_n}(t)\to\psi(t)$ pointwise with $\psi$ continuous at $0$, then $\psi$ is a characteristic function and the $X_n$ converge in distribution to its law. | 2.2 2.3 |
| Markov's inequality | For $X\ge0$ and $a\gt0$, $P(X\ge a)\le E[X]/a$; equality holds only for laws on $\{0,a\}$. | 2.2 2.3 3.3 |
| memoryless property | The property $P(X\gt s+t\mid X\gt s)=P(X\gt t)$ for all $s,t\ge0$: the remaining waiting time does not depend on how long one has already waited. Among positive laws only the exponential has it, and among laws on $\{1,2,\dots\}$ only the geometric. | 1.3 1.4 2.1 3.3 |
| negative binomial distribution | The law $\operatorname{NB}(r,p)$ of the number of failures before the $r$-th success in independent trials: $P(Y=k)=\binom{k+r-1}{k}p^r(1-p)^k$. A sum of $r$ independent geometric failure counts; its variance exceeds its mean. | 2.1 2.2 3.3 5.1 |
| Poisson distribution | The law $\operatorname{Poi}(\lambda)$ with $P(X=k)=e^{-\lambda}\lambda^k/k!$, $k=0,1,2,\dots$; its mean and variance both equal $\lambda$. It is the limit of $\operatorname{Bin}(n,p)$ as $n\to\infty$ with $np\to\lambda$, the law of counts of rare events. | 1.2 2.1 2.3 3.3 |
| reproductive property | A family of distributions has the reproductive property if a sum of independent members is again a member, e.g. $\operatorname{Poi}(\lambda)+\operatorname{Poi}(\mu)=\operatorname{Poi}(\lambda+\mu)$, $\operatorname{Bin}(n,p)+\operatorname{Bin}(m,p)=\operatorname{Bin}(n+m,p)$ and gamma laws with a common rate. | 2.1 4.1 4.3 7.1 |
| Slutsky's theorem | If $X_n\xrightarrow{d}X$ and $Y_n\xrightarrow{P}c$ (a constant), then $X_n+Y_n\xrightarrow{d}X+c$, $Y_nX_n\xrightarrow{d}cX$ and $X_n/Y_n\xrightarrow{d}X/c$ for $c\ne0$. | 2.3 4.1 4.2 4.3 |
| strong law of large numbers | Kolmogorov's theorem: for iid $X_j$, $\bar X_n\to E[X_1]$ almost surely if and only if $E|X_1|\lt\infty$. | 2.3 3.2 3.3 |
| variance-stabilising transformation | A function $g$ with $g'(\theta)^2\tau^2(\theta)$ constant, so that the delta-method variance of $g(T_n)$ does not depend on $\theta$; e.g. $\sqrt x$ for Poisson means and $\log x$ for exponential means. | 2.3 4.3 6.3 |
| weak law of large numbers | A theorem asserting $\bar X_n-E[\bar X_n]\xrightarrow{P}0$; Chebyshev's version needs $\operatorname{Var}(S_n)/n^2\to0$, Khintchine's needs iid variables with a finite mean. | 2.2 2.3 4.2 5.3 |
Ch 3 · Markov Chains and Stochastic Processes
| Term | Meaning | Used in |
|---|---|---|
| absorbing state | A state $i$ with $p_{ii}=1$: once entered, it is never left. | 3.1 |
| birth–death process | A continuous-time Markov chain on $\{0,1,2,\dots\}$ that moves only $n\to n+1$ (rate $\lambda_n$) or $n\to n-1$ (rate $\mu_n$). Its steady state is $\pi_n\propto\lambda_0\cdots\lambda_{n-1}/(\mu_1\cdots\mu_n)$ when this is summable (and the process is recurrent). The Yule process is the pure birth process with $\lambda_n=n\lambda$. | 3.3 9.3 |
| Chapman–Kolmogorov equations | $p_{ij}^{(m+n)}=\sum_kp_{ik}^{(m)}p_{kj}^{(n)}$: to go from $i$ to $j$ in $m+n$ steps, pass through some state $k$ at time $m$. Equivalently $P^{(m+n)}=P^mP^n$. | 3.1 3.3 |
| communicating class | A maximal set of states each accessible from every other. A chain with a single communicating class is irreducible. | 3.1 3.2 9.3 |
| compound Poisson process | $X(t)=\sum_{i\le N(t)}Y_i$ with $N$ a Poisson process and iid jumps $Y_i$ independent of $N$; $E X(t)=\lambda tE[Y]$ and $\operatorname{Var}X(t)=\lambda tE[Y^2]$. | 2.2 3.3 |
| continuous-time Markov chain | A process on a countable state space with the Markov property in continuous time: it stays in state $i$ for an $\operatorname{Exp}(q_i)$ time and then jumps to $j$ with probability $q_{ij}/q_i$. | 3.3 |
| counting process | A process $N(t)$ with $N(0)=0$, integer values and nondecreasing right-continuous paths that increase by jumps of size $1$: $N(t)$ counts the events in $[0,t]$. | 3.3 |
| coupling | Constructing two processes on one probability space so that they eventually coincide; then $d_{TV}$ of their laws at time $n$ is at most $P(T\gt n)$, where $T$ is the meeting time. | 2.1 3.2 |
| detailed balance | $\pi_ip_{ij}=\pi_jp_{ji}$ for all $i,j$: in equilibrium the flow from $i$ to $j$ equals the flow back. It implies stationarity; a chain satisfying it is reversible (it looks the same run backwards). | 3.2 3.3 |
| doubly stochastic matrix | A transition matrix whose columns, as well as its rows, sum to $1$. On a finite state space this is exactly the condition for the uniform distribution to be stationary. | 3.2 |
| ergodic theorem | For an irreducible chain, the fraction of time spent in state $j$ up to time $n$ converges almost surely to $1/\mu_j$ ($=\pi_j$ when positive recurrent): time averages equal space averages. | 3.2 3.3 9.3 |
| fundamental matrix | For an absorbing chain with transient block $Q$, $N=(I-Q)^{-1}=\sum_nQ^n$; $N_{ij}$ is the expected number of visits to $j$ from $i$, $NR$ gives absorption probabilities and $N\mathbf 1$ expected absorption times. | 3.1 |
| generator matrix | The matrix $Q=P'(0)$ of jump rates of a continuous-time Markov chain: $q_{ij}\ge0$ for $i\ne j$, rows summing to $0$, $q_i=-q_{ii}$ the rate of leaving $i$. For finite chains $P(t)=e^{tQ}$ and stationary laws solve $\pi Q=0$. | 3.3 |
| independent increments | A process has independent increments if its changes over disjoint time intervals are independent, and stationary increments if the law of $N(t+s)-N(s)$ depends only on the length $t$. | 3.3 |
| jump chain | The discrete-time chain of states visited by a continuous-time Markov chain, with transition probabilities $q_{ij}/q_i$. Its stationary law is proportional to $\pi_jq_j$, not to $\pi_j$. | 3.3 |
| Kolmogorov forward equations | The differential equations $P'(t)=P(t)Q$ (forward: condition on the last short interval) and $P'(t)=QP(t)$ (backward: condition on the first) for the transition probabilities of a continuous-time Markov chain. | 3.3 |
| Markov chain | A sequence of random states $X_0,X_1,\dots$ in a countable set $S$ such that, given the present state $X_n=i$, the next state has law $(p_{ij})_j$ whatever the past: $P(X_{n+1}=j\mid X_0,\dots,X_{n-1},X_n=i)=p_{ij}$. | 3.1 3.2 3.3 7.2 |
| mean return time | $\mu_i=E_i(T_i)$, the expected time for a chain started at $i$ to come back to $i$. | 3.2 3.3 |
| period of a state | $d(i)=\gcd\{n\ge1:p_{ii}^{(n)}\gt0\}$; the state is aperiodic if $d(i)=1$. The period is the same for all states of a communicating class. | 3.1 3.2 3.3 9.3 |
| Poisson process | A counting process with independent increments and $N(t+s)-N(s)\sim\operatorname{Poi}(\lambda t)$; equivalently, one with iid $\operatorname{Exp}(\lambda)$ gaps between events. Given $N(t)=n$, the event times are sorted iid uniform points on $(0,t)$. | 1.4 2.1 3.3 9.1 |
| positive recurrent | A recurrent state is positive recurrent if its mean return time is finite and null recurrent if it is infinite (as for the symmetric random walk on $\mathbb{Z}$). | 3.1 3.2 3.3 9.3 |
| recurrent state | State $i$ is recurrent if the chain started at $i$ returns to $i$ with probability one ($f_{ii}=1$), equivalently $\sum_np_{ii}^{(n)}=\infty$; otherwise it is transient and is visited only finitely often. | 3.1 3.2 3.3 |
| stationary distribution | A probability vector $\pi$ with $\pi P=\pi$: a chain started from $\pi$ keeps distribution $\pi$ for ever. For an irreducible chain it exists iff the chain is positive recurrent, and then $\pi_i=1/\mu_i$. | 3.1 3.2 3.3 |
| stopping time | A random time $T$ such that whether $T=n$ can be decided from $X_0,\dots,X_n$ alone, such as a first passage time. The strong Markov property says the chain starts afresh at a stopping time. | 3.1 3.3 |
| thinning | Keeping each event of a process independently with probability $p$. Thinning a Poisson process of rate $\lambda$ gives independent Poisson processes of rates $\lambda p$ (kept) and $\lambda(1-p)$ (removed). | 2.1 3.3 9.3 |
| total variation distance | $d_{TV}(\mu,\nu)=\frac12\sum_j|\mu_j-\nu_j|=\max_A|\mu(A)-\nu(A)|$, the standard measure of distance from equilibrium for Markov chains. | 2.1 3.2 |
| transition matrix | The matrix $P=(p_{ij})$ of one-step probabilities of a Markov chain: nonnegative entries, every row summing to $1$. The $n$-step probabilities are the entries of $P^n$. | 3.1 3.2 3.3 |
Ch 4 · Sampling Distributions and Estimation
| Term | Meaning | Used in |
|---|---|---|
| asymptotic relative efficiency | For two asymptotically normal estimators of the same parameter, the ratio of their asymptotic variances; e.g. the sample median relative to $\bar X$ for normal data has efficiency $2/\pi$. | 4.1 5.3 |
| Bayes estimator | An estimator minimising the Bayes risk $\int R(\theta,\delta)\pi(\theta)d\theta$; it is the posterior mean under squared error loss and a posterior median under absolute error loss. | 4.3 |
| chi-square distribution | $\chi^2_k=\operatorname{Gamma}(\frac k2,\frac12)$, the law of $Z_1^2+\dots+Z_k^2$ for iid $N(0,1)$ variables; mean $k$, variance $2k$. For normal samples $(n-1)S^2/\sigma^2\sim\chi^2_{n-1}$. | 1.3 2.1 4.1 7.1 |
| Clopper–Pearson interval | The exact interval for a binomial proportion obtained by inverting the binomial cdf; its endpoints are beta quantiles and its coverage is at least $1-\alpha$ for every $p$. | 4.3 |
| complete statistic | $T$ is complete if $E_\theta[g(T)]=0$ for all $\theta$ forces $g(T)=0$ a.s.; then an unbiased function of $T$ is unique, and with sufficiency it is the UMVUE. | 4.2 6.1 8.4 9.1 |
| confidence interval | A rule $[L(X),U(X)]$ computed from the data such that $P_\theta(L(X)\le\theta\le U(X))\ge1-\alpha$ for every $\theta$; the level describes the procedure over repeated samples, not a single realised interval. | 1.1 2.3 4.1 4.3 |
| conjugate prior | A family of priors that is closed under updating: the posterior is in the same family, e.g. beta for binomial data, gamma for Poisson data, normal for a normal mean with known variance. | 2.1 4.3 |
| coverage probability | The probability $P_\theta(\theta\in C(X))$ that a confidence set contains the true parameter, as a function of $\theta$; its infimum over $\theta$ is the confidence coefficient. | 4.3 |
| Cramér–Rao lower bound | In a regular model every unbiased estimator of $\psi(\theta)$ has $\operatorname{Var}\ge\psi'(\theta)^2/(nI(\theta))$; attained only in exponential families. It does not apply when the support depends on $\theta$. | 4.2 9.1 |
| credible interval | An interval containing the parameter with posterior probability $1-\alpha$; the highest posterior density (HPD) region $\{\theta:\pi(\theta\mid x)\ge k\}$ is the shortest one. | 4.3 |
| exponential family | A family $f(x;\theta)=h(x)\exp(\sum_j\eta_j(\theta)T_j(x)-A(\theta))$ with support free of $\theta$; if the natural parameters fill an open set, $(\sum_iT_j(X_i))_j$ is complete sufficient. | 4.2 5.1 6.1 9.1 |
| F distribution | $F_{m,n}$ is the law of $\frac{U/m}{V/n}$ for independent $U\sim\chi^2_m$, $V\sim\chi^2_n$; mean $\frac{n}{n-2}$; $1/F_{m,n}\sim F_{n,m}$ and $t_n^2=F_{1,n}$. | 8.3 |
| Fisher information | $I(\theta)=E_\theta[(\partial_\theta\log f(X;\theta))^2]=-E_\theta[\partial_\theta^2\log f(X;\theta)]$ in a regular model; a sample of size $n$ carries $nI(\theta)$. | 4.2 4.3 5.1 5.2 |
| Helmert transformation | An orthogonal matrix whose first row is $\frac{1}{\sqrt n}(1,\dots,1)$; applied to an iid normal sample it separates $\bar X$ from the deviations and proves $\bar X\perp S^2$ with $(n-1)S^2/\sigma^2\sim\chi^2_{n-1}$. | 4.1 6.1 7.1 |
| maximum likelihood estimator | A value $\hat\theta$ maximising the likelihood $L(\theta)=\prod f(X_i;\theta)$; invariant under reparametrisation ($\widehat{\tau(\theta)}=\tau(\hat\theta)$) and asymptotically $N(\theta,1/(nI(\theta)))$ in regular models. | 2.1 2.3 4.1 4.2 |
| mean squared error | $\operatorname{MSE}_\theta(T)=E_\theta[(T-\psi(\theta))^2]=\operatorname{Var}_\theta(T)+\text{bias}^2$, the usual single measure of an estimator's accuracy. | 1.4 4.2 4.3 6.1 |
| method of moments | Estimate $\theta$ by solving $E_\theta[X^r]=\frac1n\sum X_i^r$, $r=1,\dots,k$: equate population and sample moments. | 2.1 4.2 |
| order statistic | The sorted sample values $X_{(1)}\le\dots\le X_{(n)}$; for a density $f$, $X_{(r)}$ has density $\frac{n!}{(r-1)!(n-r)!}F^{r-1}(1-F)^{n-r}f$. | 1.1 2.1 3.3 4.1 |
| pivotal quantity | A function $Q(X,\theta)$ of the data and the parameter whose distribution is known and the same for every $\theta$, e.g. $\sqrt n(\bar X-\mu)/S\sim t_{n-1}$ or $X_{(n)}/\theta$ for $U(0,\theta)$; inverting $a\le Q\le b$ gives a confidence set. | 4.3 |
| posterior distribution | The conditional distribution of the parameter given the data, $\pi(\theta\mid x)\propto f(x\mid\theta)\pi(\theta)$: likelihood times prior, normalised. | 3.2 4.3 |
| prior distribution | The distribution $\pi(\theta)$ describing beliefs about the parameter before the data are seen; an improper prior has infinite total mass, and Jeffreys' prior is $\pi(\theta)\propto\sqrt{I(\theta)}$. | 4.3 |
| sample range | $R=X_{(n)}-X_{(1)}$, the spread between the largest and smallest observations; for a $U(0,1)$ sample $R\sim\operatorname{Beta}(n-1,2)$. | 4.1 |
| sampling distribution | The probability distribution of a statistic $T=t(X_1,\dots,X_n)$ computed from a random sample, e.g. $\bar X\sim N(\mu,\sigma^2/n)$ for a normal sample. | 2.3 4.1 4.3 5.2 |
| standard error | The standard deviation of a statistic's sampling distribution, e.g. $\operatorname{SE}(\bar X)=\sigma/\sqrt n$; with unknown parameters replaced by estimates it is the estimated standard error. | 2.3 4.1 4.2 4.3 |
| Student's t distribution | $t_k$ is the law of $Z/\sqrt{V/k}$ with $Z\sim N(0,1)$ independent of $V\sim\chi^2_k$; symmetric with heavier tails than $N(0,1)$, variance $k/(k-2)$ for $k\gt2$; $t_1$ is Cauchy. | |
| sufficient statistic | A statistic $T$ such that the conditional law of the sample given $T$ does not involve $\theta$; by the factorisation theorem, iff $f(x;\theta)=g(T(x);\theta)h(x)$. | 3.2 4.2 4.3 5.1 |
| UMVUE | Uniformly minimum variance unbiased estimator: unbiased, with variance no larger than any other unbiased estimator for every $\theta$; found by Rao–Blackwell and Lehmann–Scheffé. | 4.2 6.1 9.1 |
| unbiased estimator | A statistic $T$ with $E_\theta[T]=\psi(\theta)$ for every $\theta$; e.g. $S^2$ for $\sigma^2$. Unbiased estimators may fail to exist, and unbiasedness is not preserved by nonlinear maps. | 4.1 4.2 4.3 5.3 |
| Wilson interval | The large-sample interval for a binomial proportion obtained by solving $(\hat p-p)^2\le z^2p(1-p)/n$ for $p$; unlike the Wald interval $\hat p\pm z\sqrt{\hat p(1-\hat p)/n}$, it stays in $[0,1]$ and has coverage close to nominal. | 4.3 5.1 5.2 |
Ch 5 · Testing of Hypotheses
| Term | Meaning | Used in |
|---|---|---|
| contingency table | A table of counts of individuals cross-classified by two (or more) categorical attributes, with row totals $R_i$, column totals $C_j$ and grand total $n$. | 5.2 5.3 |
| distribution-free test | A test whose statistic has the same null distribution for every population in the null hypothesis (for example every continuous cdf), so that its size can be computed without knowing the population; typical examples use signs, ranks, runs or the empirical cdf. | 4.1 5.3 |
| empirical distribution function | $F_n(x)=\frac1n\#\{i:X_i\le x\}$, the cdf that puts mass $\frac1n$ on each observation; $nF_n(x)\sim\operatorname{Bin}(n,F(x))$ and $F_n\to F$ uniformly almost surely. | 2.3 5.3 |
| expected frequency | The count $E_j=n\pi_j$ predicted for a cell under $H_0$ (estimated when parameters are unknown; $R_iC_j/n$ in a contingency table). The $\chi^2$ approximation needs the expected, not the observed, counts to be reasonably large. | 5.1 5.2 |
| Fisher's exact test | A test for a $2\times2$ table that uses the hypergeometric law of one cell given all the margins; this law does not depend on the unknown common probability under $H_0$. | 5.2 5.3 |
| goodness-of-fit test | A test of whether data follow a specified distribution; Pearson's version compares observed and expected cell counts by $X^2=\sum(O-E)^2/E\approx\chi^2_{k-1-s}$, with $s$ the number of estimated parameters. | 2.1 5.2 6.3 |
| index of dispersion | $D=\sum(X_i-\bar X)^2/\bar X$, approximately $\chi^2_{n-1}$ for iid Poisson counts; large values indicate a variance larger than the mean (over-dispersion). | 2.1 3.3 5.2 |
| Kendall's tau | $\tau=\frac{C-D}{\binom n2}$, where $C$ and $D$ are the numbers of concordant and discordant pairs; under independence $E[\tau]=0$ and $\operatorname{Var}(\tau)=\frac{2(2n+5)}{9n(n-1)}$. | 5.3 |
| Kolmogorov–Smirnov test | Goodness-of-fit test based on $D_n=\sup_x|F_n(x)-F_0(x)|$ for a fully specified continuous $F_0$ (distribution-free), or on $\sup_x|F_m(x)-G_n(x)|$ for two samples; $\sqrt n\,D_n$ has Kolmogorov's limit law with $5\%$ point $1.358$. | 1.3 3.3 5.3 |
| likelihood ratio test | Rejects $H_0$ when $\Lambda=\sup_{\Theta_0}L/\sup_\Theta L$ is small. By Wilks' theorem, in regular models $-2\log\Lambda\to\chi^2_r$ under $H_0$, $r=\dim\Theta-\dim\Theta_0$. | 4.2 5.1 5.2 6.1 |
| Mann–Whitney test | Two-sample rank test of $F=G$ based on $U=\#\{X_i\lt Y_j\}$ or the rank sum $W=U+\frac{n(n+1)}2$ of the second sample; it detects $P(X\lt Y)\ne\frac12$, such as a shift in location. | 5.3 |
| McNemar's test | A test for paired yes/no data that uses only the discordant pairs: $(b-c)^2/(b+c)\approx\chi^2_1$ under marginal homogeneity. | 5.2 |
| median test | Two-sample test that counts how many observations of one sample lie above the combined sample median; under $F=G$ the count is hypergeometric (Fisher's exact test on a $2\times2$ table). | 5.3 |
| monotone likelihood ratio | A family has MLR in $T(x)$ if $f(x;\theta')/f(x;\theta)$ is non-decreasing in $T(x)$ whenever $\theta\lt\theta'$; then one-sided UMP tests reject for large (or small) $T$. | 5.1 5.3 |
| Neyman–Pearson lemma | For simple $H_0:f_0$ against simple $H_1:f_1$, a test that rejects when $f_1(x)/f_0(x)\gt k$ (randomising on equality) and has size $\alpha$ is most powerful at level $\alpha$. | 1.2 4.3 5.1 7.1 |
| null hypothesis | The claim $H_0:\theta\in\Theta_0$ that a test controls the risk of wrongly rejecting; the alternative $H_1:\theta\in\Theta_1$ is what is accepted when $H_0$ is rejected. A hypothesis is simple if it specifies a single distribution. | 5.1 5.2 5.3 6.2 |
| odds ratio | $\theta=\frac{p_1/(1-p_1)}{p_2/(1-p_2)}$, estimated by $ad/(bc)$; equal to $1$ exactly under independence, with $\operatorname{SE}(\log\hat\theta)\approx\sqrt{1/a+1/b+1/c+1/d}$. | 5.2 6.3 |
| p-value | The probability under $H_0$ of a result at least as extreme as the one observed; the smallest level at which the data reject $H_0$. Under a simple $H_0$ with a continuous statistic it is uniform on $(0,1)$. | 5.3 6.2 |
| power function | $\beta_\phi(\theta)=E_\theta\phi(X)$, the probability that the test $\phi$ rejects $H_0$ when the parameter is $\theta$. Its supremum over $\Theta_0$ is the size; on $\Theta_1$ it is the power. | 5.1 |
| runs test | Test of randomness of a two-symbol sequence based on the number $R$ of maximal blocks of identical symbols; with $m$ and $n$ symbols, $E[R]=1+\frac{2mn}{m+n}$ under randomness. Too few or too many runs are evidence against randomness. | 5.3 6.3 |
| score test | Large-sample tests of $H_0:\theta=\theta_0$: the score test standardises the score at $\theta_0$ (for a proportion, $(\hat p-p_0)/\sqrt{p_0q_0/n}$), the Wald test standardises $\hat\theta-\theta_0$ by the estimated standard error. Both are asymptotically equivalent to the likelihood ratio test. | 5.1 5.2 6.3 |
| sign test | Test of a median $\theta_0$ based on $S$, the number of observations above $\theta_0$; under $H_0$, $S\sim\operatorname{Bin}(n,\tfrac12)$ for every continuous population. | 5.3 |
| size of a test | The size is $\sup_{\theta\in\Theta_0}P_\theta(\text{reject }H_0)$; a test has level (significance level) $\alpha$ if its size is at most $\alpha$. | 5.1 5.2 |
| Spearman's rank correlation | The Pearson correlation of the ranks, $r_s=1-\frac{6\sum d_i^2}{n(n^2-1)}$; under independence $E[r_s]=0$ and $\operatorname{Var}(r_s)=\frac1{n-1}$. | 5.3 |
| test of homogeneity | Chi-square tests for an $r\times c$ table: independence of two attributes in one sample, or equality of the distributions of several samples with fixed sizes. Both use $\hat E_{ij}=R_iC_j/n$ and $(r-1)(c-1)$ degrees of freedom. | 5.2 5.3 |
| type I error | A type I error rejects a true null hypothesis; a type II error accepts a false one. Tests fix a bound $\alpha$ on the first and minimise the probability of the second. | 5.1 6.2 |
| unbiased test | A test whose power is at least $\alpha$ at every alternative and at most $\alpha$ under $H_0$; the equal-tailed $z$-test is unbiased, a one-sided test of a two-sided alternative is not. | 5.1 |
| uniformly most powerful test | A level-$\alpha$ test whose power is at least that of every other level-$\alpha$ test at every alternative. UMP one-sided tests exist under MLR; two-sided ones usually do not. | 5.1 |
| Wilcoxon signed-rank test | Test of symmetry about $\theta_0$ based on $T^+$, the sum of the ranks of $|X_i-\theta_0|$ over the positive $X_i-\theta_0$; under $H_0$, $T^+$ has mean $\frac{n(n+1)}4$ and variance $\frac{n(n+1)(2n+1)}{24}$. | 5.3 |
| Yates' correction | For a $2\times2$ table, $X^2_{\mathrm Y}=n(|ad-bc|-n/2)^2/(R_1R_2C_1C_2)$: a continuity correction that brings the $\chi^2_1$ $p$-value closer to the exact one when counts are small. | 5.2 |
Ch 6 · Linear Models and Regression
| Term | Meaning | Used in |
|---|---|---|
| analysis of covariance | A one-way (or larger) ANOVA with a regression on a concomitant variable $x$ measured before treatment: $y_{ij}=\mu+\alpha_i+\beta(x_{ij}-\bar x)+\varepsilon_{ij}$. It compares adjusted means $\bar y_i-\hat\beta(\bar x_i-\bar x)$ with a reduced error variance. | 6.2 |
| analysis of variance | The partition of the total sum of squares of a linear model into parts attributed to factors and to error, with $F$ ratios of the corresponding mean squares to test the factors; e.g. one-way $\text{SST}=\text{SSB}+\text{SSW}$, $F=\text{MSB}/\text{MSW}$. | 1.1 1.4 6.1 6.2 |
| best linear unbiased estimator | The linear unbiased estimator with the smallest variance (BLUE). By the Gauss–Markov theorem it is $\lambda^{\mathsf T}\hat\beta$ for least squares $\hat\beta$, with variance $\sigma^2\lambda^{\mathsf T}(X^{\mathsf T}X)^-\lambda$. | 6.1 6.3 |
| Box–Cox transformation | The family $y^{(\lambda)}=(y^\lambda-1)/\lambda$ ($\lambda\ne0$), $\log y$ ($\lambda=0$) of power transformations of a positive response, with $\lambda$ chosen by maximum likelihood to make a normal linear model fit. | 6.3 |
| Cochran's theorem | If $y\sim N_n(\mu,\sigma^2I)$ and $y^{\mathsf T}y=\sum y^{\mathsf T}A_iy$ with ranks $r_i$, then the quadratic forms are independent scaled (noncentral) $\chi^2_{r_i}$ variables iff $\sum r_i=n$; the basis of every ANOVA table. | 6.1 6.2 7.1 8.3 |
| coefficient of determination | $R^2=1-SSE/SST$, the proportion of the variation of $y$ about its mean explained by a regression with an intercept; it equals $[\operatorname{corr}(y,\hat y)]^2$ and never decreases when a regressor is added. The adjusted version $\bar R^2=1-(1-R^2)\frac{n-1}{n-p}$ penalises extra parameters. | 1.1 6.3 7.2 |
| Cook's distance | $D_i=\frac{(\hat\beta_{(i)}-\hat\beta)^\top X^\top X(\hat\beta_{(i)}-\hat\beta)}{ps^2}=\frac{r_i^2}{p}\cdot\frac{h_{ii}}{1-h_{ii}}$, the change in the fit when case $i$ is deleted; values above $1$ signal an influential case. | 6.3 |
| estimable function | A linear function $\lambda^{\mathsf T}\beta$ that has a linear unbiased estimator; this holds iff $\lambda$ is a linear combination of the rows of $X$. In the one-way model, contrasts $\sum c_i\alpha_i$ with $\sum c_i=0$ are estimable, single effects $\alpha_i$ are not. | 6.1 6.2 8.3 8.4 |
| expected mean squares | The expectations $E[\text{MS}]$ of the mean squares of an ANOVA table, written in terms of $\sigma^2$ and the effects or variance components. A term is tested against the mean square whose expectation equals its own when the term is absent. | 6.2 8.3 |
| Gauss–Markov model | The linear model $y=X\beta+\varepsilon$ with a known design matrix $X$, $E[\varepsilon]=0$ and $\operatorname{Cov}(\varepsilon)=\sigma^2I$: uncorrelated errors with a common variance. Adding $\varepsilon\sim N_n(0,\sigma^2I)$ gives the normal linear model. | 6.1 |
| Gauss–Markov theorem | In a Gauss–Markov model (mean-zero, uncorrelated, equal-variance errors; normality not needed), the least squares estimator of every estimable function is its unique best linear unbiased estimator. | 6.1 6.2 6.3 8.3 |
| generalised inverse | A matrix $A^-$ with $AA^-A=A$. It exists for every matrix and is used to write solutions of the normal equations as $(X^{\mathsf T}X)^-X^{\mathsf T}y$; the Moore–Penrose inverse is the unique one satisfying three further symmetry conditions. | 6.1 8.4 |
| hat matrix | $H=X(X^{\mathsf T}X)^-X^{\mathsf T}$, the orthogonal projection onto the column space of $X$: symmetric, idempotent, $\operatorname{tr}H=\operatorname{rank}X$, with $\hat y=Hy$ and residuals $(I-H)y$. | 6.1 6.3 |
| interaction plot | A plot of the cell means $\bar y_{ij}$ against the levels of one factor, with one line per level of the other factor. Parallel lines indicate no interaction; non-parallel or crossing lines indicate that the effect of one factor depends on the level of the other. | 6.2 |
| intraclass correlation | $\rho=\sigma_a^2/(\sigma_a^2+\sigma^2)$, the correlation between two observations in the same group of a one-way random effects model, i.e. the share of the total variance due to groups. | 6.2 8.1 |
| least squares estimator | A vector $\hat\beta$ minimising $\|y-Xb\|^2$; equivalently a solution of the normal equations. The fitted vector $X\hat\beta$ is the orthogonal projection of $y$ onto the column space of $X$. | 6.1 6.2 6.3 8.3 |
| leverage | The diagonal entry $h_{ii}$ of the hat matrix $H=X(X^\top X)^{-1}X^\top$, measuring how far case $i$ lies from the centre of the design; $0\le h_{ii}\le1$, $\sum h_{ii}=p$, $\operatorname{Var}(e_i)=\sigma^2(1-h_{ii})$, and in simple regression $h_{ii}=\frac1n+\frac{(x_i-\bar x)^2}{S_{xx}}$. | 6.1 6.3 |
| log-odds | $\operatorname{logit}\pi=\log\frac{\pi}{1-\pi}$, a bijection from $(0,1)$ onto $\mathbb{R}$ with inverse $\pi=1/(1+e^{-\eta})$; it is the scale on which logistic regression is linear. | 2.3 4.3 6.3 |
| logistic regression | The model for binary or binomial responses in which $\operatorname{logit}\pi_i=\log\frac{\pi_i}{1-\pi_i}=x_i^\top\beta$; $e^{\beta_j}$ is the odds ratio for a unit increase in $x_j$, and the MLE is found by Newton–Raphson (iteratively reweighted least squares). | 6.3 |
| mixed model | A model with both fixed and random factors; in the balanced two-way mixed model the fixed factor is tested against the interaction mean square, not against the error mean square. | 6.2 8.3 |
| multicollinearity | Near-linear dependence among the regressors of a linear model, so that $X^\top X$ is nearly singular; it inflates the variances of individual coefficients and makes them unstable, without harming fitted values inside the range of the data. | 6.3 |
| multiple comparisons | Procedures that control the probability of any false rejection over a family of comparisons: Bonferroni (level $\alpha/m$ each), Scheffé (all contrasts, multiplier $\sqrt{(k-1)F_{k-1,\nu,\alpha}}$) and Tukey (all pairs, studentized range). | 6.2 |
| normal equations | The linear system $X^{\mathsf T}Xb=X^{\mathsf T}y$ whose solutions are exactly the least squares estimates; it is always consistent, and has a unique solution iff $X$ has full column rank. | 1.1 6.1 6.3 8.3 |
| orthogonal contrasts | Contrasts $\sum c_i\mu_i$ and $\sum d_i\mu_i$ ($\sum c_i=\sum d_i=0$) with $\sum c_id_i/n_i=0$. Their single-degree-of-freedom sums of squares are independent under normality, and $k-1$ of them add up to the between-groups sum of squares. | 6.2 8.4 |
| prediction interval | An interval that contains a future observation $Y_0$ at $x_0$ with probability $1-\alpha$; in regression $\hat\mu_0\pm t_{n-p,\alpha/2}\,s\sqrt{1+x_0^\top(X^\top X)^{-1}x_0}$. Unlike a confidence interval for the mean response it does not shrink to zero as $n\to\infty$. | 6.1 6.3 |
| random effects model | A model in which the levels of a factor are a random sample from a population of levels, so their effects are random variables (e.g. $a_i\sim N(0,\sigma_a^2)$) and inference concerns the variance $\sigma_a^2$ rather than the particular effects. | 6.2 |
| studentised residual | A residual divided by its estimated standard deviation: $r_i=e_i/(s\sqrt{1-h_{ii}})$ (internal) or $t_i=e_i/(s_{(i)}\sqrt{1-h_{ii}})$ with $s_{(i)}$ computed without case $i$ (external); under normal errors $t_i\sim t_{n-p-1}$. | 6.3 |
| testable hypothesis | A linear hypothesis $\Lambda^{\mathsf T}\beta=d$ in which every component of $\Lambda^{\mathsf T}\beta$ is estimable; it is tested by $F=\frac{(\operatorname{RSS}_0-\operatorname{RSS})/q}{\operatorname{RSS}/(n-r)}\sim F_{q,n-r}$ under normality. | 6.1 |
| variance component | A variance parameter of a random effects model, such as $\sigma_a^2$ and $\sigma^2$ in $y_{ij}=\mu+a_i+\varepsilon_{ij}$; the ANOVA estimator $\hat\sigma_a^2=(\text{MSA}-\text{MSE})/n$ is unbiased but can be negative. | 6.2 |
| variance inflation factor | $\operatorname{VIF}_j=1/(1-R_j^2)$, where $R_j^2$ is the $R^2$ of regressing $x_j$ on the other regressors; $\operatorname{Var}(\hat\beta_j)=\frac{\sigma^2}{S_{jj}}\operatorname{VIF}_j$. With two regressors $\operatorname{VIF}=1/(1-r_{12}^2)$. | 6.3 |
Ch 7 · Multivariate Analysis
| Term | Meaning | Used in |
|---|---|---|
| canonical correlation | The maximal correlation between a linear combination of one set of variables $X$ and a linear combination of another set $Y$, and the successive maxima under uncorrelatedness constraints; their squares are the eigenvalues of $\Sigma_{11}^{-1}\Sigma_{12}\Sigma_{22}^{-1}\Sigma_{21}$. | 7.3 |
| covariance matrix | $\Sigma=E[(X-\mu)(X-\mu)^{\mathsf T}]$, with entries $\operatorname{Cov}(X_i,X_j)$. It is symmetric and nonnegative definite, $\operatorname{Var}(a^{\mathsf T}X)=a^{\mathsf T}\Sigma a$, and it is singular exactly when some linear combination of the components is constant. | 6.2 7.1 7.2 7.3 |
| dendrogram | The tree diagram of a hierarchical clustering: each merge of two clusters is drawn as a bar at the height (dissimilarity) at which it happens; cutting the tree at a level gives a partition. | 7.3 |
| expected cost of misclassification | For a rule allocating observations to $\mathcal P_1$ or $\mathcal P_2$: $c(2\mid1)\pi_1P(2\mid1)+c(1\mid2)\pi_2P(1\mid2)$. It is minimised by allocating to $\mathcal P_1$ when $f_1(x)/f_2(x)\ge c(1\mid2)\pi_2/(c(2\mid1)\pi_1)$. | 7.3 |
| Fisher's linear discriminant | The linear combination $\hat a^{\mathsf T}x$ with $\hat a=S_{\text{pl}}^{-1}(\bar x_1-\bar x_2)$ that maximises the squared difference of the two group means relative to the pooled within-group variance; the maximum is the sample Mahalanobis distance $D^2$. | 7.1 7.3 |
| Gaussian graphical model | A multivariate normal model drawn as a graph with an edge between $i$ and $j$ exactly when the precision-matrix entry $\omega_{ij}\ne0$, i.e. when $X_i$ and $X_j$ are not conditionally independent given all the other variables. | 7.2 |
| Hotelling's T² | $T^2=n(\bar X-\mu_0)^{\mathsf T}S^{-1}(\bar X-\mu_0)$, the multivariate analogue of $t^2$; for normal data $\frac{n-p}{p(n-1)}T^2\sim F_{p,n-p}$ under $H_0:\mu=\mu_0$. It is the likelihood ratio test and the maximum of $t_a^2$ over all linear combinations. | 7.1 7.2 |
| idempotent matrix | A square matrix with $A^2=A$. A symmetric idempotent matrix is an orthogonal projection, has eigenvalues $0$ and $1$ only, and satisfies $\operatorname{rank}A=\operatorname{tr}A$; $Z^{\mathsf T}AZ$ is chi-square for $Z\sim N_n(0,I)$ exactly when $A$ is idempotent. | 6.1 6.3 7.1 7.3 |
| k-means clustering | Partitioning data into $K$ clusters so as to minimise the within-cluster sum of squares $\sum_k\sum_{i\in C_k}\|x_i-\bar x_k\|^2$; usually computed by Lloyd's algorithm (assign to the nearest centre, move centres to cluster means), which converges to a local minimum. | 7.3 |
| linkage | The rule defining the distance between two clusters in hierarchical clustering: the smallest (single), largest (complete) or average (average) pairwise dissimilarity, or the distance between cluster means (centroid). Single, complete and average linkage give monotone dendrograms; centroid linkage can invert. | 7.3 |
| Mahalanobis distance | $\Delta(x)=\sqrt{(x-\mu)^{\mathsf T}\Sigma^{-1}(x-\mu)}$, distance measured in units that account for variances and correlations; it is invariant under nonsingular affine changes of coordinates, and $\Delta^2(X)\sim\chi^2_p$ for $X\sim N_p(\mu,\Sigma)$. | 6.3 7.1 7.2 7.3 |
| mean vector | The vector $\mu=E[X]=(E[X_1],\dots,E[X_p])^{\mathsf T}$ of the means of the components of a random vector; for a sample, $\bar x=\frac1n\sum x_i$. | 6.1 7.1 7.2 7.3 |
| multiple correlation coefficient | $R_{1\cdot2\cdots p}$, the largest correlation between $X_1$ and any linear combination of $X_2,\dots,X_p$; $R^2=\sigma_{12}\Sigma_{22}^{-1}\sigma_{21}/\sigma_{11}$, and the sample version is the square root of the regression $R^2$. | 6.3 7.2 7.3 |
| multivariate normal distribution | The law $N_p(\mu,\Sigma)$ of a random vector all of whose linear combinations $a^{\mathsf T}X$ are normal. It is determined by $\mu$ and $\Sigma$, has a density iff $\Sigma\gt0$, and its marginals, conditionals and linear images are again normal. | 7.1 7.2 7.3 |
| noncentral chi-square distribution | $\chi^2_r(\delta)$, the law of $\sum_{i=1}^rY_i^2$ with independent $Y_i\sim N(m_i,1)$ and $\delta=\sum m_i^2$; mean $r+\delta$, variance $2r+4\delta$. It is the law of test statistics under alternatives. | 7.1 |
| partial correlation | The correlation of two variables after the linear effect of a set of other variables has been removed from both; it is the correlation of the two regression residuals, e.g. $\rho_{12\cdot3}=\frac{\rho_{12}-\rho_{13}\rho_{23}}{\sqrt{(1-\rho_{13}^2)(1-\rho_{23}^2)}}$. For normal data it is the conditional correlation. | 7.1 7.2 |
| precision matrix | The inverse $\Omega=\Sigma^{-1}$ of a covariance matrix. Its diagonal blocks invert the conditional covariances, and for a normal vector $\omega_{ij}=0$ means that $X_i$ and $X_j$ are conditionally independent given the other variables. | 7.1 7.2 |
| principal component | A linear combination $e_k^{\mathsf T}X$ of the variables, where $e_k$ is a unit eigenvector of the covariance (or correlation) matrix for its $k$-th largest eigenvalue $\lambda_k$; it has the largest variance, $\lambda_k$, among unit-length combinations uncorrelated with the earlier components. | 7.3 |
| scree plot | The plot of the ordered eigenvalues $\hat\lambda_1\ge\hat\lambda_2\ge\cdots$ of a sample covariance or correlation matrix against their rank, used to judge how many principal components to keep (look for the "elbow"). | 7.3 |
| simultaneous confidence intervals | A family of intervals that all cover their targets together with probability at least $1-\alpha$, e.g. the $T^2$ intervals $a^{\mathsf T}\bar x\pm c\sqrt{a^{\mathsf T}Sa/n}$ for every $a$, or Bonferroni intervals for a few chosen combinations. | 7.2 |
| suppressor variable | A predictor that is (nearly) uncorrelated with the response but raises the multiple correlation, because it removes from another predictor a component that is irrelevant to the response. | 6.3 7.2 |
| ultrametric | A dissimilarity satisfying the strong triangle inequality $d(i,k)\le\max\{d(i,j),d(j,k)\}$. The cophenetic distances of a monotone dendrogram form an ultrametric. | 7.3 |
| Wilks' lambda | $\Lambda=|W|/|W+B|$, the ratio of the within-groups to the total generalised variance; small values reject equality of several mean vectors (MANOVA). For two groups it is a monotone function of the two-sample $T^2$. | 7.2 |
| Wishart distribution | $W_p(m,\Sigma)$, the law of $\sum_{i=1}^mZ_iZ_i^{\mathsf T}$ with $Z_i$ iid $N_p(0,\Sigma)$; the matrix analogue of $\sigma^2\chi^2_m$. For a normal sample, $\sum(X_i-\bar X)(X_i-\bar X)^{\mathsf T}\sim W_p(n-1,\Sigma)$, independent of $\bar X$. | 7.1 7.2 |
Ch 8 · Sample Surveys and Design of Experiments
| Term | Meaning | Used in |
|---|---|---|
| auxiliary variable | A variable $x$ known for every population unit (or at least its total $X$), used to improve selection (PPS) or estimation (ratio and regression estimators) of a related study variable $y$. | 8.2 |
| balanced incomplete block design | An arrangement of $v$ treatments in $b$ blocks of $k\lt v$ distinct treatments, each treatment replicated $r$ times and each pair together in $\lambda$ blocks; $bk=vr$, $\lambda(v-1)=r(k-1)$ and $b\ge v$ (Fisher's inequality). | 8.4 |
| C-matrix | The matrix $C=R-NK^{-1}N^{\mathsf T}$ of the reduced normal equations $C\hat\tau=Q$ of a block design; it is symmetric, positive semidefinite, has zero row sums, and $\operatorname{rank}C=v-1$ iff the design is connected. | 8.4 |
| completely randomised design | A design in which the treatments, with prescribed replications $r_i$, are allotted to all $N$ units completely at random; it is analysed by one-way ANOVA with $N-v$ error degrees of freedom. | 6.2 8.3 8.4 |
| confounding | Arranging a factorial experiment in incomplete blocks so that chosen effects (usually high-order interactions) coincide with block differences and are sacrificed; total if the same effects are confounded in every replicate, partial if different effects are confounded in different replicates. | 8.4 |
| connected design | A block design in which any two treatments are linked by a chain treatment–block–treatment–… of incidences; equivalently every treatment contrast is estimable, or $\operatorname{rank}C=v-1$. | 8.4 |
| cumulative total method | PPS selection by listing cumulative sizes $T_i$ and taking the unit with $T_{i-1}\lt R\le T_i$ for a random integer $R\in\{1,\dots,X\}$. | 8.2 |
| efficiency factor | For a BIBD, $E=\frac{\lambda v}{rk}=\frac{v(k-1)}{k(v-1)}\lt1$: the ratio of the variance of an elementary contrast in an RBD with the same replication and error variance to that in the BIBD. | 8.4 |
| experimental unit | The smallest piece of material (a plot, a patient, a batch) to which a treatment is applied independently of the others. Repeated measurements on one unit are subsamples, not replicates. | 6.2 8.3 |
| finite population correction | The factor $1-f=1-n/N$ in the variance $(1-f)S^2/n$ of the sample mean under sampling without replacement; it is negligible when the sampling fraction $n/N$ is small. | 2.1 8.1 |
| Fisher's inequality | In every BIBD the number of blocks is at least the number of treatments, $b\ge v$ (hence $r\ge k$), because $NN^{\mathsf T}=(r-\lambda)I+\lambda J$ is nonsingular. | 8.4 |
| generalised interaction | The product of two effects with squared letters deleted, e.g. $ABC\cdot BCD=AD$. When two effects are confounded with blocks, their generalised interaction is confounded as well. | 8.4 |
| Hansen–Hurwitz estimator | Under PPS with replacement, $\hat Y_{\mathrm{HH}}=\frac1n\sum y_j/p_j$; unbiased for the total, with variance $\frac1n\sum p_i(Y_i/p_i-Y)^2$. | 8.2 |
| Horvitz–Thompson estimator | $\hat Y_{\mathrm{HT}}=\sum_{i\in s}y_i/\pi_i$, unbiased for the population total under any design in which every inclusion probability $\pi_i$ is positive. | 8.2 |
| incidence matrix | For a block design, the $v\times b$ matrix $N=(n_{ij})$ with $n_{ij}$ the number of plots of block $j$ that receive treatment $i$; its row sums are the replications $r_i$ and its column sums the block sizes $k_j$. | 8.4 |
| inclusion probability | For a sampling design, $\pi_i=P(\text{unit } i \text{ is in the sample})$; the second-order inclusion probability is $\pi_{ij}=P(i \text{ and } j \text{ both in the sample})$. | 8.1 8.2 |
| Lahiri's method | PPS selection by drawing a random unit $i$ and a random integer $R\le M$ (with $M\ge\max X_i$) and accepting $i$ if $R\le X_i$, repeating until acceptance. | 8.2 |
| Latin square design | A design for $v$ treatments on $v^2$ units classified by rows and columns, each treatment occurring once in every row and every column; it controls two nuisance factors and leaves $(v-1)(v-2)$ error degrees of freedom. | 8.3 |
| local control | Grouping experimental units into homogeneous blocks and comparing treatments within blocks, so that variation between blocks is removed from the experimental error. | 8.3 |
| missing plot technique | Estimating a lost observation by the value that minimises the error sum of squares of the completed data (for an RBD, $\hat x=\frac{vT+bB-G}{(v-1)(b-1)}$), then analysing the completed table with one error df fewer and the treatment SS corrected for its upward bias. | 8.3 |
| Neyman allocation | The allocation $n_h\propto N_hS_h$ that minimises the variance of the stratified mean for fixed $n$; with unit costs $c_h$ the optimum is $n_h\propto N_hS_h/\sqrt{c_h}$. | 8.1 |
| orthogonal block design | A block design with proportional frequencies $n_{ij}=r_ik_j/n$; treatment and block effects are then estimated independently and adjustment for blocks changes nothing. Every RBD is orthogonal; no incomplete design is. | 8.4 |
| PPS sampling | Sampling in which unit $i$ is selected with probability proportional to a known size $X_i$, i.e. $p_i=X_i/X$ per draw (with replacement) or $\pi_i\propto X_i$ (without replacement). | 8.2 |
| proportional allocation | Allocation of a stratified sample with $n_h=nN_h/N$; it makes the sample self-weighting (the stratified mean equals the plain sample mean). | 8.1 |
| randomisation | Allotting treatments to units by a chance mechanism, all allocations permitted by the design being equally likely. It makes treatment comparisons unbiased and the error mean square a valid estimate of the error variance, without assuming normality. | 5.1 8.3 |
| randomised block design | A design with $b$ blocks of $v$ units in which every treatment appears once in every block, randomised independently within blocks; analysed by the additive two-way ANOVA with $(v-1)(b-1)$ error degrees of freedom. | 6.2 8.3 8.4 |
| ratio estimator | $\bar y_R=(\bar y/\bar x)\bar X$; slightly biased, with first-order MSE $\frac{1-f}{n}(S_y^2+R^2S_x^2-2R\rho S_xS_y)$, better than $\bar y$ iff $\rho\gt C_x/(2C_y)$. | 2.3 6.3 8.2 |
| regression estimator | $\bar y_{lr}=\bar y+b(\bar X-\bar x)$ with $b$ the sample slope; to first order its MSE is $\frac{1-f}{n}S_y^2(1-\rho^2)$, never larger than that of $\bar y$ or $\bar y_R$. | 8.2 |
| relative efficiency of a design | The ratio of the error variances of two designs for the same comparisons; e.g. RBD versus CRD is estimated by $\frac{(b-1)M_b+b(v-1)M_e}{(bv-1)M_e}$, which exceeds $1$ iff $M_b\gt M_e$. | 8.3 |
| sampling frame | The list (or map) of population units from which a sample is actually selected; units missing from the frame can never be sampled. | 8.1 |
| Sen–Yates–Grundy estimator | For fixed-size designs, the variance estimator $\sum_{i\lt j\in s}\frac{\pi_i\pi_j-\pi_{ij}}{\pi_{ij}}\left(\frac{y_i}{\pi_i}-\frac{y_j}{\pi_j}\right)^2$ of the Horvitz–Thompson estimator; non-negative when $\pi_i\pi_j\ge\pi_{ij}$. | 8.2 |
| simple random sampling | Drawing $n$ units so that every subset of size $n$ is equally likely (without replacement, SRSWOR), or by $n$ independent equiprobable draws (with replacement, SRSWR). | 8.1 8.2 8.3 |
| stratified sampling | Partitioning the population into strata and drawing an independent random sample from each; the mean is estimated by $\bar y_{\mathrm{st}}=\sum W_h\bar y_h$ with $W_h=N_h/N$. | 8.1 8.2 |
| systematic sampling | Selecting every $k$-th unit of an ordered list after a random start in $\{1,\dots,k\}$; efficient under a trend, poor when $k$ matches a periodic pattern. | 8.1 8.2 |
| uniformity trial | An experiment in which all units receive the same treatment, so that the observed variation is pure unit-to-unit variation; used to study field heterogeneity and the efficiency of block shapes. | 8.3 |
| Yates' algorithm | A method for the contrasts of a $2^k$ factorial: list the treatment totals in standard order and $k$ times replace the column by the sums of successive pairs followed by their differences; the final column holds $G$ and the contrasts $[A],[B],[AB],\dots$. | 8.4 |
Ch 9 · Reliability, Linear Programming and Queues
| Term | Meaning | Used in |
|---|---|---|
| artificial variable | An extra nonnegative variable added to a $\ge$ or $=$ constraint to supply a starting basis; the two-phase method minimises their sum first, and the Big-M method penalises each by a very large cost $M$. The original problem is feasible iff they can all be driven to $0$. | 9.2 |
| assignment problem | The problem of assigning $n$ agents to $n$ tasks, one each, at least total cost; a transportation problem with all supplies and demands $1$, solved by the Hungarian method of row and column reductions. | 9.2 |
| basic feasible solution | For $Ax=b$, $x\ge0$ with $A$ of rank $m$: a solution obtained by choosing $m$ linearly independent columns (a basis), setting the other variables to $0$ and solving, which happens to be $\ge0$. It is degenerate if some basic variable is $0$. | 9.2 |
| censoring | Incomplete observation of a lifetime: we know only that it exceeds some time. A Type I (time-censored) test stops at a fixed time $t_0$, so the number of failures is random; a Type II (failure-censored) test stops at the $r$-th failure, so the duration is random. Censored items contribute $R(y)$ to the likelihood, failures contribute $f(y)$. | 4.1 9.1 |
| complementary slackness | Feasible primal and dual solutions are both optimal iff $y_i(b-Ax)_i=0$ and $x_j(A^{\mathsf T}y-c)_j=0$ for all $i,j$: a positive dual price forces its constraint to be tight, and a positive variable forces its dual constraint to be tight. | 9.2 |
| cumulative hazard | $H(t)=\int_0^t h(u)\,du=-\log R(t)$. For a lifetime it tends to $\infty$, $H(T)\sim\operatorname{Exp}(1)$, and the cumulative hazards of independent components in series add. | 9.1 |
| dual problem | The linear program attached to a primal LP with one variable per primal constraint: the dual of $\max c^{\mathsf T}x$, $Ax\le b$, $x\ge0$ is $\min b^{\mathsf T}y$, $A^{\mathsf T}y\ge c$, $y\ge0$. Its optimal value equals the primal one when either has an optimum (strong duality). | 9.2 |
| economic order quantity | The order size $Q^*=\sqrt{2KD/h}$ that minimises ordering plus holding cost $KD/Q+hQ/2$ for steady demand $D$, ordering cost $K$ and holding cost $h$ per unit per unit time; at $Q^*$ the two costs are equal. | 9.3 |
| Erlang B formula | The fraction of customers lost in the loss system M/M/c/c: $B(c,a)=\frac{a^c/c!}{\sum_{k\le c}a^k/k!}$, computed by $B(c,a)=\frac{aB(c-1,a)}{c+aB(c-1,a)}$; it is insensitive to the service-time distribution. | 3.3 9.3 |
| Erlang C formula | In an M/M/c queue with offered load $a$ and $\rho=a/c\lt1$, the probability that an arrival must wait: $C(c,a)=\pi_0\,\frac{a^c}{c!(1-\rho)}$; then $W_q=C(c,a)/(c\mu-\lambda)$. | 9.3 |
| extreme point | A point of a convex set that is not the midpoint of any segment of two different points of the set — a corner. For $\{x:Ax=b,\ x\ge0\}$ the extreme points are exactly the basic feasible solutions. | 7.1 9.2 |
| hazard rate | The instantaneous failure rate of a unit that has survived to age $t$: $h(t)=f(t)/R(t)=-\frac{d}{dt}\log R(t)$, so that $P(t\lt T\le t+\Delta\mid T\gt t)\approx h(t)\Delta$. It is a rate (it may exceed $1$), and it determines the lifetime law. | 2.1 9.1 |
| increasing failure rate | A lifetime is IFR (DFR) if $R(t+x)/R(t)$ is nonincreasing (nondecreasing) in $t$ for every $x\gt0$ — for laws with a density, if the hazard rate is nondecreasing (nonincreasing). The exponential law is both; Weibull and gamma laws are IFR for shape $\ge1$ and DFR for shape $\le1$. | 9.1 |
| k-out-of-n system | A system that works while at least $k$ of its $n$ components work: $k=n$ is a series and $k=1$ a parallel system. With iid components of reliability $p$, its reliability is $P(\operatorname{Bin}(n,p)\ge k)$, and its lifetime is the $(n-k+1)$-th order statistic of the component lifetimes. | 4.1 9.1 |
| Kendall notation | The shorthand $A/B/c/K/N$ for a queue: inter-arrival law $A$, service law $B$ ($M$ = exponential, $D$ = constant, $E_k$ = Erlang, $G$ = general), $c$ servers, capacity $K$ and source population $N$ (the last two $\infty$ if omitted). | 9.3 |
| linear programming problem | The problem of maximising or minimising a linear function $c^{\mathsf T}x$ over the set of $x$ satisfying finitely many linear equations and inequalities; its feasible region is a convex polyhedron. | 9.2 |
| Little's law | $L=\lambda W$: the long-run average number in a system equals the arrival rate (of admitted customers) times the mean time each spends there. It holds for any arrival and service laws and any discipline; likewise $L_q=\lambda W_q$. | 9.3 |
| mean residual life | The expected remaining life of a unit that has survived to age $t$: $m(t)=E[T-t\mid T\gt t]=\frac{1}{R(t)}\int_t^\infty R(u)\,du$. It is constant only for the exponential law and nonincreasing for IFR laws. | 9.1 |
| mean time to failure | The expected lifetime $\mu=E[T]=\int_0^\infty R(t)\,dt$; for a system, the integral of the system reliability function. For a $k$-out-of-$n$ system of iid $\operatorname{Exp}(\lambda)$ parts it is $\frac1\lambda\sum_{j=k}^n\frac1j$. | 9.1 |
| PASTA | Poisson Arrivals See Time Averages: when arrivals form a Poisson process not anticipating the future of the system, the fraction of arrivals that find the system in a state equals the fraction of time it spends in that state. | 3.3 9.3 |
| Pollaczek–Khinchine formula | For the M/G/1 queue with $\rho=\lambda E[S]\lt1$: $W_q=\frac{\lambda E[S^2]}{2(1-\rho)}$, so the mean wait grows with the variance of the service time; constant service halves the M/M/1 queue. | 9.3 |
| reliability function | For a lifetime $T$, the probability of surviving beyond age $t$: $R(t)=P(T\gt t)=1-F(t)$. It is nonincreasing, $R(0)=1$ for a lifetime with a density, and $R(t)=e^{-H(t)}$. | 1.3 1.4 2.1 9.1 |
| simplex method | Dantzig's method for linear programs: move from one basic feasible solution to an adjacent one with a better objective, choosing the entering variable by its net evaluation $z_j-c_j$ and the leaving variable by the minimum ratio test, until all $z_j-c_j\ge0$ (maximisation) or unboundedness is detected. | 9.2 |
| structure function | The map $\varphi:\{0,1\}^n\to\{0,1\}$ giving the state of a system from the states of its $n$ components. The system is coherent if $\varphi$ is nondecreasing in each component and every component is relevant; series is $\min_ix_i$, parallel is $\max_ix_i$. | 9.1 |
| total time on test | The sum of the times all items spent on test in a life test, e.g. $T_r=\sum_{i=1}^rX_{(i)}+(n-r)X_{(r)}$ in a Type II test without replacement. For exponential lifetimes of mean $\theta$, the MLE of $\theta$ is the total time on test divided by the number of failures, and $2T_r/\theta\sim\chi^2_{2r}$. | 4.1 4.3 9.1 |
| traffic intensity | For arrival rate $\lambda$, service rate $\mu$ per server and $c$ servers, the offered load is $a=\lambda/\mu$ and the traffic intensity (utilisation) is $\rho=\lambda/(c\mu)$; an infinite-capacity M/M/c queue is stable iff $\rho\lt1$. | 9.3 |
| transportation problem | The LP of shipping supplies $a_i$ from $m$ sources to meet demands $b_j$ at $n$ destinations at least total cost $\sum c_{ij}x_{ij}$; when balanced it always has an optimal basic solution with at most $m+n-1$ positive shipments, integral if the data are integers. | 9.2 |