7.1 The Multivariate Normal and Wishart Distributions
Mean vectors and covariance matrices; the multivariate normal law defined through linear combinations, with its density, marginals and conditionals; the Mahalanobis distance; when a quadratic form is chi-square (Cochran's theorem); and the Wishart law of the sample covariance matrix.
Statistics with one variable runs on a few facts about the normal law: linear combinations of independent normals are normal, $\bar X$ and $S^2$ are independent, and $(n-1)S^2/\sigma^2$ is $\chi^2_{n-1}$. Almost every multivariate method rests on the matrix versions of these facts. The variance becomes a covariance matrix $\Sigma$; the normal law becomes the multivariate normal law $N_p(\mu,\Sigma)$, best defined by the requirement that every linear combination of the coordinates is normal; the statistic $(n-1)S^2$ becomes the matrix of sums of squares and products, whose law is the Wishart law. The same machinery explains the analysis of variance: a sum of squares is a quadratic form $X^{\mathsf T}AX$, and Cochran's theorem says exactly when such forms are independent chi-square variables. Section 6.1 quoted that theorem; here we prove it.
The section is also a catalogue of traps, and CSIR NET uses them. Two normal variables need not be jointly normal; uncorrelated normal variables need not be independent unless they are jointly normal; a quadratic form with the right mean need not be chi-square; and the sum of two Wishart matrices need not be Wishart. Each trap fails for a reason that the theorems below make precise, and the ⚠️ box near the end collects them.
Random Vectors, Mean Vectors and Covariance Matrices
A random vector $X=(X_1,\dots,X_p)^{\mathsf T}$ is a column of random variables defined on one probability space. Expectations of vectors and matrices are taken entry by entry.
Let $X$ be a random vector with $E[X_i^2]\lt\infty$ for every $i$.
- The mean vector is $\mu=E[X]=(E[X_1],\dots,E[X_p])^{\mathsf T}$.
- The covariance matrix is $\Sigma=\operatorname{Cov}(X)=E\big[(X-\mu)(X-\mu)^{\mathsf T}\big]$, the $p\times p$ matrix with entries $\sigma_{ij}=\operatorname{Cov}(X_i,X_j)$; in particular $\sigma_{ii}=\operatorname{Var}(X_i)$.
- For a second random vector $Y$ with mean $\nu$, the cross-covariance is $\operatorname{Cov}(X,Y)=E\big[(X-\mu)(Y-\nu)^{\mathsf T}\big]$.
- If every $\sigma_{ii}\gt0$, the correlation matrix is $\mathcal P=D^{-1/2}\Sigma D^{-1/2}$ with $D=\operatorname{diag}(\sigma_{11},\dots,\sigma_{pp})$; its entries are $\rho_{ij}=\sigma_{ij}/\sqrt{\sigma_{ii}\sigma_{jj}}$ and its diagonal is all $1$s.
For a symmetric matrix $A$ we write $A\ge0$ if $a^{\mathsf T}Aa\ge0$ for all $a$ (nonnegative definite) and $A\gt0$ if $a^{\mathsf T}Aa\gt0$ for all $a\ne0$ (positive definite).
Let $X$ have mean $\mu$ and covariance matrix $\Sigma$, and let $A$ ($q\times p$), $B$ ($r\times p$) and $b\in\mathbb R^q$ be constant.
- $E[AX+b]=A\mu+b$, $\ \operatorname{Cov}(AX+b)=A\Sigma A^{\mathsf T}$ and $\operatorname{Cov}(AX,BX)=A\Sigma B^{\mathsf T}$. In particular $\operatorname{Var}(a^{\mathsf T}X)=a^{\mathsf T}\Sigma a$ and $\operatorname{Cov}(a^{\mathsf T}X,c^{\mathsf T}X)=a^{\mathsf T}\Sigma c$.
- $\Sigma$ is symmetric and $\Sigma\ge0$.
- $\Sigma$ is singular if and only if some nontrivial linear combination is constant: $a^{\mathsf T}X=a^{\mathsf T}\mu$ with probability $1$ for some $a\ne0$. More precisely, $X-\mu$ lies in the column space $\mathcal C(\Sigma)$ with probability $1$.
- For every $p\times p$ matrix $A$, $E[X^{\mathsf T}AX]=\operatorname{tr}(A\Sigma)+\mu^{\mathsf T}A\mu$.
Proof. (1) Each entry of $AX+b$ is $\sum_ja_{ij}X_j+b_i$, whose mean is $\sum_ja_{ij}\mu_j+b_i$ by linearity. Then $AX+b-(A\mu+b)=A(X-\mu)$, so $$\operatorname{Cov}(AX+b)=E\big[A(X-\mu)(X-\mu)^{\mathsf T}A^{\mathsf T}\big]=A\,E\big[(X-\mu)(X-\mu)^{\mathsf T}\big]A^{\mathsf T}=A\Sigma A^{\mathsf T},$$ because a constant matrix can be taken out of an expectation on either side (again by linearity, entry by entry). The cross-covariance is identical with $B$ on the right.
(2) $\sigma_{ij}=\sigma_{ji}$, and by (1), $a^{\mathsf T}\Sigma a=\operatorname{Var}(a^{\mathsf T}X)\ge0$.
(3) $\Sigma$ is singular iff $\Sigma a=0$ for some $a\ne0$. For symmetric $\Sigma\ge0$, $\Sigma a=0$ iff $a^{\mathsf T}\Sigma a=0$ (if $a^{\mathsf T}\Sigma a=0$, write $\Sigma=\Sigma^{1/2}\Sigma^{1/2}$; then $\|\Sigma^{1/2}a\|^2=0$, so $\Sigma a=\Sigma^{1/2}(\Sigma^{1/2}a)=0$). And $a^{\mathsf T}\Sigma a=\operatorname{Var}(a^{\mathsf T}X)=0$ iff $a^{\mathsf T}X=a^{\mathsf T}\mu$ almost surely. For the last claim, the null space of $\Sigma$ is $\mathcal C(\Sigma)^\perp$; pick a basis $a_1,\dots,a_k$ of it. Each event $\{a_j^{\mathsf T}(X-\mu)=0\}$ has probability $1$, so their intersection does, and on it $X-\mu\perp\mathcal C(\Sigma)^\perp$, i.e. $X-\mu\in\mathcal C(\Sigma)$.
(4) $X^{\mathsf T}AX$ is a scalar, so it equals its own trace, and $\operatorname{tr}(X^{\mathsf T}AX)=\operatorname{tr}(AXX^{\mathsf T})$. Since $E[XX^{\mathsf T}]=\Sigma+\mu\mu^{\mathsf T}$, we get $E[X^{\mathsf T}AX]=\operatorname{tr}(A\Sigma)+\operatorname{tr}(A\mu\mu^{\mathsf T})$, and $\operatorname{tr}(A\mu\mu^{\mathsf T})=\mu^{\mathsf T}A\mu$. ∎
Conversely, every symmetric $\Sigma\ge0$ is a covariance matrix: if $Z$ has independent standard components and $AA^{\mathsf T}=\Sigma$ (for instance $A=\Sigma^{1/2}$), then $\operatorname{Cov}(AZ)=AIA^{\mathsf T}=\Sigma$ by part (1). So "is this matrix a covariance matrix?" means exactly "is it symmetric and nonnegative definite?", and for a correlation matrix one also needs unit diagonal.
The Multivariate Normal Distribution
There are several equivalent definitions. The one below needs no density, covers singular covariance matrices, and makes the key properties one-line consequences.
A random vector $X$ in $\mathbb R^p$ is multivariate normal (its components are jointly normal) if $a^{\mathsf T}X$ has a univariate normal distribution for every $a\in\mathbb R^p$. Here a constant $c$ counts as the degenerate normal law $N(c,0)$. We shall see that the law of $X$ is then determined by $\mu=E[X]$ and $\Sigma=\operatorname{Cov}(X)$, and we write $X\sim N_p(\mu,\Sigma)$.
- If $X$ is multivariate normal with mean $\mu$ and covariance $\Sigma$, its characteristic function is $$\varphi_X(t)=E\big[e^{it^{\mathsf T}X}\big]=\exp\left(it^{\mathsf T}\mu-\tfrac12t^{\mathsf T}\Sigma t\right),\qquad t\in\mathbb R^p,$$ and its moment generating function is $M_X(t)=\exp\left(t^{\mathsf T}\mu+\tfrac12t^{\mathsf T}\Sigma t\right)$. Hence the law of $X$ depends only on $(\mu,\Sigma)$.
- Conversely, for every $\mu\in\mathbb R^p$ and every symmetric $\Sigma\ge0$ there is such a vector: if $Z=(Z_1,\dots,Z_k)^{\mathsf T}$ has independent $N(0,1)$ components and $A$ is $p\times k$ with $AA^{\mathsf T}=\Sigma$, then $X=\mu+AZ\sim N_p(\mu,\Sigma)$.
- Every $X\sim N_p(\mu,\Sigma)$ has the same law as $\mu+\Sigma^{1/2}Z$ with $Z\sim N_p(0,I_p)$.
Proof. (1) Fix $t$. The variable $Y=t^{\mathsf T}X$ is normal by definition, with mean $t^{\mathsf T}\mu$ and variance $t^{\mathsf T}\Sigma t$ (Theorem 7.1). The characteristic function of $N(m,s^2)$ at the point $1$ is $e^{im-s^2/2}$ (§2.2), so $\varphi_X(t)=E[e^{iY}]=\exp\left(it^{\mathsf T}\mu-\tfrac12t^{\mathsf T}\Sigma t\right)$. The mgf is computed in the same way from $E[e^{Y}]=e^{m+s^2/2}$. A probability law on $\mathbb R^p$ is determined by its characteristic function (the uniqueness theorem of §2.2, which holds in $\mathbb R^p$ with the same proof), so the law of $X$ is determined by $(\mu,\Sigma)$.
(2) For $a\in\mathbb R^p$, $a^{\mathsf T}X=a^{\mathsf T}\mu+(A^{\mathsf T}a)^{\mathsf T}Z=a^{\mathsf T}\mu+\sum_jc_jZ_j$ with $c=A^{\mathsf T}a$. A linear combination of independent normal variables is normal (the reproductive property, §2.1), so $a^{\mathsf T}X$ is normal and $X$ is multivariate normal. By Theorem 7.1, $E[X]=\mu$ and $\operatorname{Cov}(X)=AA^{\mathsf T}=\Sigma$. A suitable $A$ always exists: by the spectral theorem $\Sigma=Q\Lambda Q^{\mathsf T}$ with $Q$ orthogonal and $\Lambda=\operatorname{diag}(\lambda_i)$, $\lambda_i\ge0$, and $A=\Sigma^{1/2}=Q\Lambda^{1/2}Q^{\mathsf T}$ works; so does the Cholesky factor when $\Sigma\gt0$.
(3) $\mu+\Sigma^{1/2}Z$ is multivariate normal with mean $\mu$ and covariance $\Sigma$ by (2), so by (1) it has the same law as $X$. ∎
Part (1) contains the Cramér–Wold device: the law of a random vector is determined by the laws of all its linear combinations $a^{\mathsf T}X$, since these give $\varphi_X(t)$ for every $t$. Part (2) shows that many different matrices $A$ give the same law — Interactive 1 lets you watch this happen.
If $\Sigma\gt0$, then $X\sim N_p(\mu,\Sigma)$ has the density $$f(x)=\frac{1}{(2\pi)^{p/2}|\Sigma|^{1/2}}\exp\left(-\tfrac12(x-\mu)^{\mathsf T}\Sigma^{-1}(x-\mu)\right),\qquad x\in\mathbb R^p.$$ If $\Sigma$ is singular of rank $r\lt p$, $X$ has no density on $\mathbb R^p$: with probability $1$ it lies in the $r$-dimensional affine subspace $\mu+\mathcal C(\Sigma)$.
Proof. By Theorem 7.2(3) we may take $X=\mu+\Sigma^{1/2}Z$ with $Z\sim N_p(0,I)$. The components of $Z$ are independent, so $Z$ has density $\prod_i\phi(z_i)=(2\pi)^{-p/2}e^{-z^{\mathsf T}z/2}$. The map $z\mapsto x=\mu+\Sigma^{1/2}z$ is a bijection of $\mathbb R^p$ with inverse $z=\Sigma^{-1/2}(x-\mu)$, whose Jacobian determinant is $|\Sigma^{-1/2}|=|\Sigma|^{-1/2}$. By the change-of-variables formula (§1.4), $$f(x)=(2\pi)^{-p/2}|\Sigma|^{-1/2}\exp\left(-\tfrac12\,z^{\mathsf T}z\right),\qquad z^{\mathsf T}z=(x-\mu)^{\mathsf T}\Sigma^{-1/2}\Sigma^{-1/2}(x-\mu)=(x-\mu)^{\mathsf T}\Sigma^{-1}(x-\mu).$$ If $\Sigma$ is singular, $X-\mu\in\mathcal C(\Sigma)$ almost surely (Theorem 7.1(3)), and a proper subspace has Lebesgue measure $0$; a law that gives probability $1$ to a null set has no density. ∎
For $p=2$, $|\Sigma|=\sigma_1^2\sigma_2^2(1-\rho^2)$ and $\Sigma^{-1}=\frac{1}{1-\rho^2}\begin{pmatrix}1/\sigma_1^2&-\rho/(\sigma_1\sigma_2)\\-\rho/(\sigma_1\sigma_2)&1/\sigma_2^2\end{pmatrix}$, and the formula becomes the bivariate normal density of §1.4.
Let $X\sim N_p(\mu,\Sigma)$, $A$ a $q\times p$ matrix and $b\in\mathbb R^q$. Then $$AX+b\sim N_q\left(A\mu+b,\ A\Sigma A^{\mathsf T}\right).$$ In particular, partition $$X=\begin{pmatrix}X_1\\X_2\end{pmatrix},\quad\mu=\begin{pmatrix}\mu_1\\\mu_2\end{pmatrix},\quad\Sigma=\begin{pmatrix}\Sigma_{11}&\Sigma_{12}\\\Sigma_{21}&\Sigma_{22}\end{pmatrix}$$ with $X_1$ of length $q$. Then $X_1\sim N_q(\mu_1,\Sigma_{11})$: every subvector of a normal vector is normal. If $\Sigma\gt0$, $\ \Sigma^{-1/2}(X-\mu)\sim N_p(0,I_p)$, and for every orthogonal $Q$, $Q(X-\mu)$ has the same law as $X-\mu$ when $\Sigma=\sigma^2I$.
Proof. For $c\in\mathbb R^q$, $c^{\mathsf T}(AX+b)=(A^{\mathsf T}c)^{\mathsf T}X+c^{\mathsf T}b$ is a linear combination of $X$ plus a constant, hence normal. So $AX+b$ is multivariate normal, with the mean and covariance of Theorem 7.1. The marginal is the case $A=(I_q\ \ 0)$; standardisation is $A=\Sigma^{-1/2}$, giving covariance $\Sigma^{-1/2}\Sigma\Sigma^{-1/2}=I$; and $Q\sigma^2IQ^{\mathsf T}=\sigma^2I$. ∎
Let $X=(X_1^{\mathsf T},X_2^{\mathsf T})^{\mathsf T}\sim N_p(\mu,\Sigma)$ be partitioned as in Theorem 7.4. Then $X_1$ and $X_2$ are independent if and only if $\Sigma_{12}=0$. More generally, blocks $X_1,\dots,X_k$ of a jointly normal vector are mutually independent if and only if all cross-covariance blocks vanish; in particular the components of $X\sim N_p(\mu,\Sigma)$ are mutually independent if and only if $\Sigma$ is diagonal.
Proof. Independence always implies zero covariance (when second moments exist). Conversely, suppose $\Sigma_{12}=0$ (so $\Sigma_{21}=\Sigma_{12}^{\mathsf T}=0$). For $t=(t_1^{\mathsf T},t_2^{\mathsf T})^{\mathsf T}$, $$t^{\mathsf T}\Sigma t=t_1^{\mathsf T}\Sigma_{11}t_1+2t_1^{\mathsf T}\Sigma_{12}t_2+t_2^{\mathsf T}\Sigma_{22}t_2=t_1^{\mathsf T}\Sigma_{11}t_1+t_2^{\mathsf T}\Sigma_{22}t_2,$$ so $\varphi_X(t)=\varphi_{X_1}(t_1)\varphi_{X_2}(t_2)$, using Theorem 7.2(1) for $X$ and, by Theorem 7.4, for $X_1$ and $X_2$. The right side is the characteristic function of the law under which $X_1$ and $X_2$ are independent with their given marginals; by uniqueness that is the law of $X$. The case of $k$ blocks is identical. ∎
The theorem is about joint normality. Example 2 shows that it fails for variables that are only marginally normal. A useful corollary: for jointly normal variables, pairwise independence of the components already implies mutual independence, because it makes $\Sigma$ diagonal.
Conditional Distributions and the Mahalanobis Distance
Example 3(c) already contains the idea of this subsection: subtract from $X_1$ the right multiple of $X_2$ and what is left is independent of $X_2$. Conditioning on $X_2$ then only shifts the mean.
Let $X\sim N_p(\mu,\Sigma)$ be partitioned as in Theorem 7.4, with $\Sigma_{22}\gt0$. Put $$\Sigma_{11\cdot2}=\Sigma_{11}-\Sigma_{12}\Sigma_{22}^{-1}\Sigma_{21}.$$ Then the conditional law of $X_1$ given $X_2=x_2$ is $$X_1\mid X_2=x_2\ \sim\ N_q\left(\mu_1+\Sigma_{12}\Sigma_{22}^{-1}(x_2-\mu_2),\ \Sigma_{11\cdot2}\right).$$ Thus (i) the regression $E[X_1\mid X_2=x_2]$ is linear in $x_2$; (ii) the conditional covariance $\Sigma_{11\cdot2}$ does not depend on $x_2$; (iii) $\Sigma_{11\cdot2}\le\Sigma_{11}$, i.e. $\Sigma_{11}-\Sigma_{11\cdot2}\ge0$: conditioning never increases variances.
Proof. Let $B=\Sigma_{12}\Sigma_{22}^{-1}$ ($q\times(p-q)$) and $U=X_1-BX_2$. The vector $(U^{\mathsf T},X_2^{\mathsf T})^{\mathsf T}$ is a linear function of $X$, hence jointly normal (Theorem 7.4), and $$\operatorname{Cov}(U,X_2)=\Sigma_{12}-B\Sigma_{22}=0.$$ By Theorem 7.5, $U$ is independent of $X_2$. Its mean is $\mu_1-B\mu_2$ and its covariance is $$\operatorname{Cov}(U)=\Sigma_{11}-B\Sigma_{21}-\Sigma_{12}B^{\mathsf T}+B\Sigma_{22}B^{\mathsf T}=\Sigma_{11}-\Sigma_{12}\Sigma_{22}^{-1}\Sigma_{21}=\Sigma_{11\cdot2},$$ since $B\Sigma_{21}=\Sigma_{12}B^{\mathsf T}=B\Sigma_{22}B^{\mathsf T}=\Sigma_{12}\Sigma_{22}^{-1}\Sigma_{21}$. Now $X_1=U+BX_2$. Given $X_2=x_2$, the independent part $U$ keeps its law $N_q(\mu_1-B\mu_2,\Sigma_{11\cdot2})$, and $BX_2=Bx_2$ is a constant; so $X_1\mid X_2=x_2\sim N_q(\mu_1+B(x_2-\mu_2),\Sigma_{11\cdot2})$. Finally $\Sigma_{11}=\operatorname{Cov}(U+BX_2)=\Sigma_{11\cdot2}+B\Sigma_{22}B^{\mathsf T}$ and $B\Sigma_{22}B^{\mathsf T}\ge0$. ∎
Let $\Sigma\gt0$ be partitioned as above and $\Omega=\Sigma^{-1}$ (the precision matrix), partitioned in the same way. Then $\Sigma_{11\cdot2}\gt0$ and
- $|\Sigma|=|\Sigma_{22}|\,|\Sigma_{11\cdot2}|$;
- $\Omega_{11}=\Sigma_{11\cdot2}^{-1}$ and $\Omega_{12}=-\Sigma_{11\cdot2}^{-1}\Sigma_{12}\Sigma_{22}^{-1}$;
- consequently the conditional covariance is $\Sigma_{11\cdot2}=\Omega_{11}^{-1}$ and the regression matrix is $\Sigma_{12}\Sigma_{22}^{-1}=-\Omega_{11}^{-1}\Omega_{12}$.
Proof. With $B=\Sigma_{12}\Sigma_{22}^{-1}$ and $T=\begin{pmatrix}I&-B\\0&I\end{pmatrix}$, block multiplication gives $$T\,\Sigma\,T^{\mathsf T}=\begin{pmatrix}\Sigma_{11\cdot2}&0\\0&\Sigma_{22}\end{pmatrix}.$$ (This is the covariance matrix of $(U,X_2)$ from the last proof.) Since $|T|=1$, taking determinants gives (1), and $\Sigma_{11\cdot2}\gt0$ because $T\Sigma T^{\mathsf T}\gt0$. Inverting, $T^{-\mathsf T}\Sigma^{-1}T^{-1}=\operatorname{diag}(\Sigma_{11\cdot2}^{-1},\Sigma_{22}^{-1})$, so $$\Sigma^{-1}=T^{\mathsf T}\begin{pmatrix}\Sigma_{11\cdot2}^{-1}&0\\0&\Sigma_{22}^{-1}\end{pmatrix}T=\begin{pmatrix}\Sigma_{11\cdot2}^{-1}&-\Sigma_{11\cdot2}^{-1}B\\-B^{\mathsf T}\Sigma_{11\cdot2}^{-1}&\Sigma_{22}^{-1}+B^{\mathsf T}\Sigma_{11\cdot2}^{-1}B\end{pmatrix},$$ which gives (2); (3) follows by solving (2) for $\Sigma_{11\cdot2}$ and $B$. ∎
Part (3) is the bridge to partial correlation in §7.2: the entries of the precision matrix describe each variable given all the others. It also shows that the density factorises as $f(x)=f_{X_2}(x_2)\,f_{X_1\mid X_2}(x_1\mid x_2)$ with the two normal densities of Theorems 7.4 and 7.6, as it must.
Let $\Sigma\gt0$. The Mahalanobis distance of a point $x$ from $\mu$ (relative to $\Sigma$) is $$\Delta(x)=\sqrt{(x-\mu)^{\mathsf T}\Sigma^{-1}(x-\mu)},$$ and the Mahalanobis distance between two populations with means $\mu_1,\mu_2$ and common covariance $\Sigma$ is $\Delta=\sqrt{(\mu_1-\mu_2)^{\mathsf T}\Sigma^{-1}(\mu_1-\mu_2)}$. For $p=1$ it is $|x-\mu|/\sigma$, the number of standard deviations.
Let $X\sim N_p(\mu,\Sigma)$ with $\Sigma\gt0$, and let $\Sigma=\sum_{i=1}^p\lambda_ie_ie_i^{\mathsf T}$ be its spectral decomposition ($\lambda_i\gt0$, $e_i$ orthonormal).
- $D^2=(X-\mu)^{\mathsf T}\Sigma^{-1}(X-\mu)\sim\chi^2_p$.
- The density is constant on each ellipsoid $\{x:(x-\mu)^{\mathsf T}\Sigma^{-1}(x-\mu)=c\}$, whose axes point along $e_1,\dots,e_p$ with half-lengths $\sqrt{c\lambda_i}$; its volume is $|\Sigma|^{1/2}c^{p/2}$ times the volume of the unit ball.
- The solid ellipsoid $\mathcal E_\alpha=\{x:(x-\mu)^{\mathsf T}\Sigma^{-1}(x-\mu)\le\chi^2_{p,\alpha}\}$ has probability $1-\alpha$, and no other region with probability $1-\alpha$ has smaller volume.
- $\Delta$ is invariant under nonsingular affine changes of coordinates: if $y=Cx+d$ with $C$ nonsingular, then the distance of $y$ from $C\mu+d$ relative to $C\Sigma C^{\mathsf T}$ equals the distance of $x$ from $\mu$ relative to $\Sigma$.
Proof. (1) $Z=\Sigma^{-1/2}(X-\mu)\sim N_p(0,I)$ (Theorem 7.4), and $D^2=Z^{\mathsf T}Z=\sum Z_i^2$ is a sum of $p$ squares of independent $N(0,1)$ variables.
(2) The density depends on $x$ only through $(x-\mu)^{\mathsf T}\Sigma^{-1}(x-\mu)$ (Theorem 7.3). With $y_i=e_i^{\mathsf T}(x-\mu)$, the coordinates of $x-\mu$ in the orthonormal basis $e_i$, and $\Sigma^{-1}=\sum\lambda_i^{-1}e_ie_i^{\mathsf T}$, the equation becomes $\sum_iy_i^2/\lambda_i=c$: an ellipsoid with half-axes $\sqrt{c\lambda_i}$ along the $e_i$. Its volume is $\prod_i\sqrt{c\lambda_i}$ times the volume of the unit ball, and $\prod\lambda_i=|\Sigma|$.
(3) $P(X\in\mathcal E_\alpha)=P(D^2\le\chi^2_{p,\alpha})=1-\alpha$ by (1). Let $R$ be any region with $P(X\in R)=1-\alpha$, and let $k$ be the value of the density on the boundary of $\mathcal E_\alpha$, so $f\ge k$ on $\mathcal E_\alpha$ and $f\le k$ outside. Since $P(\mathcal E_\alpha)=P(R)$, also $P(\mathcal E_\alpha\setminus R)=P(R\setminus\mathcal E_\alpha)$. Then $$k\operatorname{vol}(\mathcal E_\alpha\setminus R)\le P(\mathcal E_\alpha\setminus R)=P(R\setminus\mathcal E_\alpha)\le k\operatorname{vol}(R\setminus\mathcal E_\alpha),$$ so $\operatorname{vol}(\mathcal E_\alpha)\le\operatorname{vol}(R)$ — the same argument as the Neyman–Pearson lemma (§5.1).
(4) With $y-C\mu-d=C(x-\mu)$, $$\begin{aligned}&(y-C\mu-d)^{\mathsf T}(C\Sigma C^{\mathsf T})^{-1}(y-C\mu-d)\\&\quad=(x-\mu)^{\mathsf T}C^{\mathsf T}C^{-\mathsf T}\Sigma^{-1}C^{-1}C(x-\mu)\\&\quad=(x-\mu)^{\mathsf T}\Sigma^{-1}(x-\mu).\ ∎\end{aligned}$$
Part (4) is why the Mahalanobis distance, and every statistic built from it (Hotelling's $T^2$, Fisher's discriminant), does not care about units: measuring height in centimetres instead of metres, or adding a constant, changes nothing. The Euclidean distance has neither property.
Quadratic Forms and Cochran's Theorem
Sums of squares are quadratic forms. In a normal sample, $\sum(X_i-\bar X)^2=X^{\mathsf T}(I-\tfrac1nJ)X$, and in the linear model of §6.1, $\operatorname{RSS}=Y^{\mathsf T}(I-H)Y$. When is such a form chi-square, and when are two of them independent?
If $Y\sim N_r(m,I_r)$, the law of $Y^{\mathsf T}Y=\sum_{i=1}^rY_i^2$ is the noncentral chi-square distribution with $r$ degrees of freedom and noncentrality $\delta=m^{\mathsf T}m$, written $\chi^2_r(\delta)$; $\chi^2_r(0)=\chi^2_r$. It depends on $m$ only through $\delta$, and $$E\big[\chi^2_r(\delta)\big]=r+\delta,\qquad\operatorname{Var}\big[\chi^2_r(\delta)\big]=2r+4\delta.$$ (Some books use $\delta/2$ as the noncentrality parameter; we follow §6.1.)
If $m$ and $m'$ have the same length, there is an orthogonal $Q$ with $Qm=m'$, and $QY\sim N_r(m',I)$ has the same squared length as $Y$ (Theorem 7.4); so the law depends on $m$ only through $\|m\|^2=\delta$. Taking $m=(\sqrt\delta,0,\dots,0)$: $E[Y_1^2]=1+\delta$, and for $Y_1=Z+a$ with $Z\sim N(0,1)$, $E[Y_1^4]=a^4+6a^2+3$, so $\operatorname{Var}(Y_1^2)=a^4+6a^2+3-(1+a^2)^2=2+4a^2$. Adding the $r-1$ central terms, each of variance $2$, gives $2r+4\delta$.
Let $Z\sim N_n(\mu,I_n)$ and let $A$ be symmetric with eigenvalues $\lambda_1,\dots,\lambda_n$. Then
- $E[Z^{\mathsf T}AZ]=\operatorname{tr}A+\mu^{\mathsf T}A\mu$ and $\operatorname{Var}(Z^{\mathsf T}AZ)=2\operatorname{tr}(A^2)+4\mu^{\mathsf T}A^2\mu$;
- if $\mu=0$, then $Z^{\mathsf T}AZ$ has the law of $\sum_i\lambda_iW_i$ with $W_1,\dots,W_n$ iid $\chi^2_1$, and its mgf is $\prod_i(1-2t\lambda_i)^{-1/2}$ for $|t|$ small.
Proof. Write $A=Q\Lambda Q^{\mathsf T}$ with $Q$ orthogonal, and put $V=Q^{\mathsf T}Z\sim N_n(\nu,I)$, $\nu=Q^{\mathsf T}\mu$ (Theorem 7.4). Then $Z^{\mathsf T}AZ=V^{\mathsf T}\Lambda V=\sum_i\lambda_iV_i^2$ with independent $V_i^2\sim\chi^2_1(\nu_i^2)$. By Definition 7.4, $E=\sum\lambda_i(1+\nu_i^2)=\operatorname{tr}A+\nu^{\mathsf T}\Lambda\nu$ and $\operatorname{Var}=\sum\lambda_i^2(2+4\nu_i^2)=2\operatorname{tr}A^2+4\nu^{\mathsf T}\Lambda^2\nu$; finally $\nu^{\mathsf T}\Lambda\nu=\mu^{\mathsf T}A\mu$ and $\nu^{\mathsf T}\Lambda^2\nu=\mu^{\mathsf T}A^2\mu$. If $\mu=0$, $V_i^2\sim\chi^2_1$ with mgf $(1-2t)^{-1/2}$, and the mgf of $\lambda_iV_i^2$ is $(1-2t\lambda_i)^{-1/2}$; independence multiplies them. ∎
- Let $Z\sim N_n(\mu,I_n)$ and $A$ symmetric. Then $Z^{\mathsf T}AZ$ has a (noncentral) chi-square law if and only if $A$ is idempotent ($A^2=A$). In that case $$Z^{\mathsf T}AZ\sim\chi^2_r(\mu^{\mathsf T}A\mu),\qquad r=\operatorname{rank}A=\operatorname{tr}A.$$
- Let $X\sim N_n(\mu,\Sigma)$ with $\Sigma\gt0$ and $A$ symmetric. Then $X^{\mathsf T}AX\sim\chi^2_r(\mu^{\mathsf T}A\mu)$ for some $r$ if and only if $A\Sigma A=A$ (equivalently, $A\Sigma$ is idempotent), and then $r=\operatorname{rank}A=\operatorname{tr}(A\Sigma)$.
Proof. (1) If. A symmetric idempotent matrix has eigenvalues $\lambda$ with $\lambda^2=\lambda$, so $\lambda\in\{0,1\}$; with $r$ unit eigenvalues and orthonormal eigenvectors $q_1,\dots,q_r$, $A=\sum_{i\le r}q_iq_i^{\mathsf T}$, $\operatorname{rank}A=r=\operatorname{tr}A$, and $Z^{\mathsf T}AZ=\sum_{i\le r}(q_i^{\mathsf T}Z)^2$. The variables $q_i^{\mathsf T}Z$ are jointly normal with covariance $q_i^{\mathsf T}q_j=\delta_{ij}$, hence independent $N(q_i^{\mathsf T}\mu,1)$ (Theorem 7.5), and the sum is $\chi^2_r(\delta)$ with $\delta=\sum(q_i^{\mathsf T}\mu)^2=\mu^{\mathsf T}A\mu$.
Only if (central case $\mu=0$; the noncentral case uses the noncentral mgf in the same way). Suppose $Z^{\mathsf T}AZ\sim\chi^2_r$. Comparing mgfs (Theorem 7.9), $\prod_i(1-2t\lambda_i)^{-1/2}=(1-2t)^{-r/2}$ for all $t$ near $0$. Squaring and inverting, $\prod_{\lambda_i\ne0}(1-2t\lambda_i)=(1-2t)^r$ for all $t$ in an interval. Both sides are polynomials in $t$, so they are identical, and comparing their roots, every nonzero $\lambda_i$ satisfies $1/(2\lambda_i)=1/2$, i.e. $\lambda_i=1$. So every eigenvalue is $0$ or $1$, and a symmetric matrix with such eigenvalues is idempotent.
(2) Put $Y=\Sigma^{-1/2}X\sim N_n(\Sigma^{-1/2}\mu,I)$. Then $X^{\mathsf T}AX=Y^{\mathsf T}MY$ with $M=\Sigma^{1/2}A\Sigma^{1/2}$ symmetric. By (1) it is chi-square iff $M^2=M$, i.e. $\Sigma^{1/2}A\Sigma A\Sigma^{1/2}=\Sigma^{1/2}A\Sigma^{1/2}$, i.e. (cancelling the nonsingular $\Sigma^{1/2}$ on both sides) $A\Sigma A=A$. The degrees of freedom are $\operatorname{tr}M=\operatorname{tr}(A\Sigma)$ and $\operatorname{rank}M=\operatorname{rank}A$, and the noncentrality is $(\Sigma^{-1/2}\mu)^{\mathsf T}M(\Sigma^{-1/2}\mu)=\mu^{\mathsf T}A\mu$. ∎
Two familiar cases: $A=\Sigma^{-1}$ satisfies $A\Sigma A=A$, which is Theorem 7.8(1) again; and for $X\sim N_n(\theta\mathbf 1,\sigma^2I)$, $C=I-\frac1nJ$ is idempotent of trace $n-1$ with $C\mathbf 1=0$, so $\sum(X_i-\bar X)^2/\sigma^2\sim\chi^2_{n-1}$ (the result of §4.1).
Let $X\sim N_n(\mu,\sigma^2I_n)$, let $A$, $B$ be symmetric $n\times n$ and $L$ a $k\times n$ matrix.
- If $LA=0$, then $LX$ and $X^{\mathsf T}AX$ are independent.
- $X^{\mathsf T}AX$ and $X^{\mathsf T}BX$ are independent if and only if $AB=0$.
For $X\sim N_n(\mu,\Sigma)$ the conditions become $L\Sigma A=0$ and $A\Sigma B=0$.
Proof. (1) Let $A=Q_1\Lambda_1Q_1^{\mathsf T}$, where the columns of $Q_1$ are orthonormal eigenvectors for the nonzero eigenvalues and $\Lambda_1$ is nonsingular. Then $X^{\mathsf T}AX=(Q_1^{\mathsf T}X)^{\mathsf T}\Lambda_1(Q_1^{\mathsf T}X)$ is a function of $Q_1^{\mathsf T}X$. From $AQ_1=Q_1\Lambda_1$ we get $LQ_1=LAQ_1\Lambda_1^{-1}=0$. The vectors $LX$ and $Q_1^{\mathsf T}X$ are jointly normal with $\operatorname{Cov}(LX,Q_1^{\mathsf T}X)=\sigma^2LQ_1=0$, so they are independent (Theorem 7.5), and so are functions of them.
(2) If. Write $B=Q_2\Lambda_2Q_2^{\mathsf T}$ similarly. From $AB=0$, multiplying by $Q_1^{\mathsf T}$ on the left and by $Q_2$ on the right, $\Lambda_1Q_1^{\mathsf T}Q_2\Lambda_2=0$, so $Q_1^{\mathsf T}Q_2=0$. Then $Q_1^{\mathsf T}X$ and $Q_2^{\mathsf T}X$ are jointly normal and uncorrelated, hence independent, and the two forms are functions of them. Only if (sketch). For $\mu=0$, independence makes the joint mgf factor near $(0,0)$: $$|I-2sA-2tB|^{-1/2}=|I-2sA|^{-1/2}\,|I-2tB|^{-1/2}.$$ Comparing the coefficients of $st$ after taking logarithms gives $\operatorname{tr}(AB)=0$, and an argument on the full polynomial identity (Craig's theorem, completed by later authors) gives $AB=0$. For idempotent $A$, $B$ the converse is easy: Exercise III.7 shows $\operatorname{Cov}(Z^{\mathsf T}AZ,Z^{\mathsf T}BZ)=2\operatorname{tr}(AB)=2\|AB\|_F^2$. The general-$\Sigma$ form follows by $X=\Sigma^{1/2}Y$. ∎
With $L=\frac1n\mathbf 1^{\mathsf T}$ and $A=I-\frac1nJ$ we have $LA=0$: so $\bar X$ and $\sum(X_i-\bar X)^2$ are independent — the fundamental theorem for normal samples (§4.1) in two lines. In the linear model, $H(I-H)=0$ gives the independence of the fitted values and $\operatorname{RSS}$ used in §6.1.
Let $A_1,\dots,A_k$ be symmetric $n\times n$ matrices with $A_1+\cdots+A_k=I_n$, and let $r_i=\operatorname{rank}A_i$. The following are equivalent:
- $r_1+\cdots+r_k=n$;
- every $A_i$ is idempotent;
- $A_iA_j=0$ for all $i\ne j$.
If they hold and $X\sim N_n(\mu,\sigma^2I_n)$, then the forms $Q_i=X^{\mathsf T}A_iX$ (which add up to $X^{\mathsf T}X$) are independent, and $Q_i/\sigma^2\sim\chi^2_{r_i}(\mu^{\mathsf T}A_i\mu/\sigma^2)$.
Proof. (1) ⇒ (2). Fix $i$. Since $I=A_i+(I-A_i)$ and $I-A_i=\sum_{j\ne i}A_j$, rank subadditivity gives $$n=\operatorname{rank}I\le\operatorname{rank}A_i+\operatorname{rank}(I-A_i)\le r_i+\sum_{j\ne i}r_j=n.$$ Hence $\operatorname{rank}(I-A_i)=n-r_i$, so the null space of $I-A_i$ — the eigenspace of $A_i$ for the eigenvalue $1$ — has dimension $r_i$. But $A_i$ has exactly $r_i$ nonzero eigenvalues counted with multiplicity, so all of them equal $1$. A symmetric matrix with eigenvalues in $\{0,1\}$ is idempotent.
(2) ⇒ (1). For idempotent $A_i$, $\operatorname{rank}A_i=\operatorname{tr}A_i$, so $\sum r_i=\operatorname{tr}\sum A_i=\operatorname{tr}I=n$.
(1) and (2) ⇒ (3). Fix $i\ne j$ and let $C=A_i+A_j$. Then $I=C+\sum_{l\ne i,j}A_l$, and the argument of the first step, applied to $C$ (whose rank is at most $r_i+r_j$), shows that all nonzero eigenvalues of $C$ equal $1$: $C^2=C$. Expanding, $A_i^2+A_iA_j+A_jA_i+A_j^2=A_i+A_j$, so $A_iA_j+A_jA_i=0$. Multiplying this on the left by $A_i$ gives $A_iA_j+A_iA_jA_i=0$; multiplying it on the right by $A_i$ gives $A_iA_jA_i+A_jA_i=0$. Subtracting, $A_iA_j=A_jA_i$, and with $A_iA_j+A_jA_i=0$ this gives $A_iA_j=0$.
(3) ⇒ (2). $A_i=A_iI=A_i\sum_jA_j=A_i^2$.
Distribution. Write each idempotent $A_i=P_iP_i^{\mathsf T}$ with $P_i$ an $n\times r_i$ matrix of orthonormal columns. From $A_iA_j=0$, $P_i^{\mathsf T}P_j=P_i^{\mathsf T}(P_iP_i^{\mathsf T})(P_jP_j^{\mathsf T})P_j$ $=P_i^{\mathsf T}A_iA_jP_j=0$. So the $n$ columns of $P=(P_1\ \cdots\ P_k)$ are orthonormal and $P$ is orthogonal. Then $V=P^{\mathsf T}X\sim N_n(P^{\mathsf T}\mu,\sigma^2I)$ has independent coordinates, and $Q_i=\|P_i^{\mathsf T}X\|^2$ is the sum of squares of the $i$-th block of $V$. The blocks are disjoint, so the $Q_i$ are independent, and $Q_i/\sigma^2\sim\chi^2_{r_i}(\|P_i^{\mathsf T}\mu\|^2/\sigma^2)$ with $\|P_i^{\mathsf T}\mu\|^2=\mu^{\mathsf T}A_i\mu$. ∎
This is the theorem quoted in §6.1, now proved. In practice condition (1) is the one checked: if the degrees of freedom of the pieces of a sum of squares add up to the total, the pieces are independent chi-square variables, and ratios of them divided by their degrees of freedom are $F$ variables.
The Wishart Distribution and the Sample Mean and Covariance
For a univariate normal sample, $\sum(X_i-\bar X)^2\sim\sigma^2\chi^2_{n-1}$, and $\sigma^2\chi^2_m$ is the law of $\sum_{i=1}^mZ_i^2$ with $Z_i$ iid $N(0,\sigma^2)$. Replacing each square $Z_i^2$ by the outer product $Z_iZ_i^{\mathsf T}$ gives the matrix version.
Let $Z_1,\dots,Z_m$ be iid $N_p(0,\Sigma)$ and let $\mathbf Z$ be the $m\times p$ matrix with rows $Z_i^{\mathsf T}$. The law of the $p\times p$ random matrix $$W=\sum_{i=1}^mZ_iZ_i^{\mathsf T}=\mathbf Z^{\mathsf T}\mathbf Z$$ is the Wishart distribution with $m$ degrees of freedom and scale matrix $\Sigma$, written $W_p(m,\Sigma)$. For $p=1$, $W_1(m,\sigma^2)$ is the law of $\sigma^2\chi^2_m$. Its entries are $W_{jk}=\sum_iZ_{ij}Z_{ik}$, the sums of squares and products of the columns of $\mathbf Z$.
Let $W\sim W_p(m,\Sigma)$.
- (Moments) $E[W]=m\Sigma$ and $\operatorname{Var}(W_{jk})=m(\sigma_{jk}^2+\sigma_{jj}\sigma_{kk})$.
- (Additivity) If $W_1\sim W_p(m_1,\Sigma)$ and $W_2\sim W_p(m_2,\Sigma)$ are independent, then $W_1+W_2\sim W_p(m_1+m_2,\Sigma)$.
- (Linear maps) For a $q\times p$ matrix $C$, $CWC^{\mathsf T}\sim W_q(m,C\Sigma C^{\mathsf T})$. In particular every diagonal block $W_{11}\sim W_q(m,\Sigma_{11})$, $W_{jj}/\sigma_{jj}\sim\chi^2_m$, and $a^{\mathsf T}Wa/a^{\mathsf T}\Sigma a\sim\chi^2_m$ for every fixed $a$ with $a^{\mathsf T}\Sigma a\gt0$; also $\operatorname{tr}(\Sigma^{-1}W)\sim\chi^2_{mp}$ when $\Sigma\gt0$.
- (Random directions) If $a$ is a random vector independent of $W$ with $a^{\mathsf T}\Sigma a\gt0$ almost surely, then $a^{\mathsf T}Wa/a^{\mathsf T}\Sigma a\sim\chi^2_m$ and is independent of $a$.
- (Inverse) Let $\Sigma\gt0$. If $m\lt p$, $W$ is singular; if $m\ge p$, $W\gt0$ with probability $1$, and for every fixed $a\ne0$, $$\frac{a^{\mathsf T}\Sigma^{-1}a}{a^{\mathsf T}W^{-1}a}\sim\chi^2_{m-p+1};$$ as in (4), the same holds for a random $a$ independent of $W$, and the ratio is then independent of $a$.
- (Generalised variance) For $m\ge p$ and $\Sigma\gt0$, $|W|/|\Sigma|$ has the law of a product $\prod_{j=1}^pU_j$ of independent $U_j\sim\chi^2_{m-j+1}$; so $E|W|=m(m-1)\cdots(m-p+1)|\Sigma|$.
- (Density, statement) For $m\ge p$ and $\Sigma\gt0$, $W$ has the density $$f(w)=\frac{|w|^{(m-p-1)/2}\exp\left(-\tfrac12\operatorname{tr}(\Sigma^{-1}w)\right)}{2^{mp/2}|\Sigma|^{m/2}\Gamma_p(m/2)}$$ on positive definite $w$, where $\Gamma_p(a)=\pi^{p(p-1)/4}\prod_{j=1}^p\Gamma\left(a-\tfrac{j-1}2\right)$.
Proof. (1) $E[Z_iZ_i^{\mathsf T}]=\Sigma$ for each of the $m$ terms. For the variance, $W_{jk}$ is a sum of $m$ iid copies of $Z_jZ_k$ (dropping the index $i$), and for jointly normal centred variables $E[Z_j^2Z_k^2]=\sigma_{jj}\sigma_{kk}+2\sigma_{jk}^2$ (expand $Z_k=\beta Z_j+U$ with $U$ independent of $Z_j$, or use Isserlis' formula); so $\operatorname{Var}(Z_jZ_k)=\sigma_{jj}\sigma_{kk}+\sigma_{jk}^2$.
(2) Stack the two independent samples: $W_1+W_2$ is the sum of $m_1+m_2$ iid outer products. (3) $CWC^{\mathsf T}=\sum_i(CZ_i)(CZ_i)^{\mathsf T}$ and the $CZ_i$ are iid $N_q(0,C\Sigma C^{\mathsf T})$. With $C=a^{\mathsf T}$, $a^{\mathsf T}Wa=\sum_i(a^{\mathsf T}Z_i)^2$ with $a^{\mathsf T}Z_i$ iid $N(0,a^{\mathsf T}\Sigma a)$, which is $a^{\mathsf T}\Sigma a$ times a $\chi^2_m$ variable. For the trace, $\operatorname{tr}(\Sigma^{-1}W)=\sum_iZ_i^{\mathsf T}\Sigma^{-1}Z_i$, a sum of $m$ independent $\chi^2_p$ variables (Theorem 7.8). (4) Conditionally on $a$, by independence and (3), the ratio is $\chi^2_m$. This conditional law does not depend on $a$, so it is also the unconditional law, and the ratio is independent of $a$.
(5) $\operatorname{rank}W=\operatorname{rank}\mathbf Z\le m$, so $W$ is singular when $m\lt p$. Reduce to $\Sigma=I$ and $a=e_1$: choose $C$ nonsingular with $C\Sigma C^{\mathsf T}=I$ (so $\Sigma^{-1}=C^{\mathsf T}C$), let $b=Ca$, choose an orthogonal $Q$ with $Qb=\|b\|e_1$, and put $W^*=QCWC^{\mathsf T}Q^{\mathsf T}\sim W_p(m,I)$ by (3). Since $W^{-1}=C^{\mathsf T}Q^{\mathsf T}W^{*-1}QC$, we get $$a^{\mathsf T}W^{-1}a=(QCa)^{\mathsf T}W^{*-1}(QCa)=\|b\|^2(W^{*-1})_{11},$$ while $a^{\mathsf T}\Sigma^{-1}a=\|Ca\|^2=\|b\|^2$; so the ratio is $1/(W^{*-1})_{11}$. By Theorem 7.7(2) applied to $W^*$, $1/(W^{*-1})_{11}=W^*_{11}-W^*_{12}(W^*_{22})^{-1}W^*_{21}$. Write $\mathbf Z=(z_1\ \ \mathbf Z_2)$ for the $m\times p$ data matrix of $W^*$ (columns iid $N_m(0,I)$ because $\Sigma=I$). Then this Schur complement equals $z_1^{\mathsf T}(I-H_2)z_1$, where $H_2$ is the orthogonal projection onto the column space of $\mathbf Z_2$ — it is the residual sum of squares of the least-squares regression of $z_1$ on the other $p-1$ columns (§6.1). For $m\ge p$, $\mathbf Z_2$ has rank $p-1$ almost surely, so $I-H_2$ is idempotent of rank $m-p+1$; conditionally on $\mathbf Z_2$, $z_1\sim N_m(0,I)$ is independent of it, so $z_1^{\mathsf T}(I-H_2)z_1\sim\chi^2_{m-p+1}$ (Theorem 7.10). The conditional law is free of $\mathbf Z_2$, so it is the unconditional law. Finally $W^*=\mathbf Z^{\mathsf T}\mathbf Z\gt0$ as soon as $\mathbf Z$ has rank $p$, and this happens with probability $1$: the rows have a density, so each of the first $p$ rows falls into the span of the earlier ones (a proper subspace, of Lebesgue measure $0$) with probability $0$. The random-$a$ statement follows as in (4).
(6) (Sketch, with $\Sigma=I$ after the reduction in (5), which multiplies $|W|$ by $|C|^2=1/|\Sigma|$.) Gram–Schmidt on the columns $z_1,\dots,z_p$ of $\mathbf Z$ gives $|\mathbf Z^{\mathsf T}\mathbf Z|=\prod_{j=1}^pR_j$, where $R_j$ is the residual sum of squares of $z_j$ regressed on $z_1,\dots,z_{j-1}$. By the argument of (5), conditionally on $z_1,\dots,z_{j-1}$, $R_j\sim\chi^2_{m-j+1}$; since these conditional laws do not depend on the conditioning columns, $R_1,\dots,R_p$ are independent. The expectation is $\prod_j(m-j+1)$. (This product representation is Bartlett's decomposition.) (7) is obtained from the same decomposition by a change of variables; we omit the calculation. ∎
Let $X_1,\dots,X_n$ be iid $N_p(\mu,\Sigma)$ with $n\ge2$, and let $$\bar X=\frac1n\sum_{i=1}^nX_i,\qquad A=\sum_{i=1}^n(X_i-\bar X)(X_i-\bar X)^{\mathsf T},\qquad S=\frac{A}{n-1}.$$
- $\bar X\sim N_p(\mu,\Sigma/n)$;
- $A\sim W_p(n-1,\Sigma)$; in particular $E[S]=\Sigma$;
- $\bar X$ and $A$ (hence $\bar X$ and $S$) are independent;
- if $\Sigma\gt0$ and $n\gt p$, then $S\gt0$ almost surely, and the maximum likelihood estimators of $\mu$ and $\Sigma$ are $\hat\mu=\bar X$ and $\hat\Sigma=A/n$.
Proof. Let $\mathbf X$ be the $n\times p$ data matrix with rows $X_i^{\mathsf T}$, and let $H$ be an $n\times n$ orthogonal matrix whose first row is $\mathbf 1^{\mathsf T}/\sqrt n$ (for instance the Helmert matrix of §4.1). Put $\mathbf Y=H\mathbf X$, with rows $Y_j^{\mathsf T}$, so $Y_j=\sum_ih_{ji}X_i$.
The rows of $\mathbf Y$ are independent normal vectors. All entries of $\mathbf Y$ are linear combinations of the entries of the independent normal vectors $X_i$, so they are jointly normal. Their cross-covariances are $$\operatorname{Cov}(Y_j,Y_k)=\sum_ih_{ji}h_{ki}\Sigma=(HH^{\mathsf T})_{jk}\Sigma=\delta_{jk}\Sigma,$$ so by Theorem 7.5 the rows $Y_1,\dots,Y_n$ are independent, each with covariance $\Sigma$. Their means are $E[Y_j]=\big(\sum_ih_{ji}\big)\mu$; for $j=1$ this is $\sqrt n\mu$, and for $j\ge2$ it is $0$, because row $j$ of $H$ is orthogonal to row $1$, i.e. its entries add up to $0$.
Identification. $Y_1=\frac1{\sqrt n}\sum X_i=\sqrt n\bar X$, which gives (1). Since $H^{\mathsf T}H=I$, $\sum_jY_jY_j^{\mathsf T}=\mathbf Y^{\mathsf T}\mathbf Y=\mathbf X^{\mathsf T}\mathbf X=\sum_iX_iX_i^{\mathsf T}$. Expanding $A=\sum X_iX_i^{\mathsf T}-n\bar X\bar X^{\mathsf T}$ and using $n\bar X\bar X^{\mathsf T}=Y_1Y_1^{\mathsf T}$, $$A=\sum_{j=2}^nY_jY_j^{\mathsf T},$$ a sum of $n-1$ outer products of iid $N_p(0,\Sigma)$ vectors: $A\sim W_p(n-1,\Sigma)$, which is (2). $A$ is a function of $Y_2,\dots,Y_n$ and $\bar X$ of $Y_1$, so they are independent: (3).
(4) Positive definiteness is Theorem 7.13(5) with $m=n-1\ge p$. For the likelihood, using the identity $$\textstyle\sum_i(x_i-\mu)^{\mathsf T}\Sigma^{-1}(x_i-\mu)=\operatorname{tr}(\Sigma^{-1}A)+n(\bar x-\mu)^{\mathsf T}\Sigma^{-1}(\bar x-\mu)$$ (expand about $\bar x$; the cross terms vanish), $$\log L(\mu,\Sigma)=\text{const}-\frac n2\log|\Sigma|-\frac12\operatorname{tr}(\Sigma^{-1}A)-\frac n2(\bar x-\mu)^{\mathsf T}\Sigma^{-1}(\bar x-\mu).$$ For each $\Sigma$ the last term is maximised (made $0$) by $\mu=\bar x$. Next let $B=A^{1/2}\Sigma^{-1}A^{1/2}$, with eigenvalues $\eta_1,\dots,\eta_p\gt0$. Then $\operatorname{tr}(\Sigma^{-1}A)=\operatorname{tr}B=\sum\eta_i$ and $\log|\Sigma|=\log|A|-\log|B|$ $=\log|A|-\sum\log\eta_i$, so $$-\frac n2\log|\Sigma|-\frac12\operatorname{tr}(\Sigma^{-1}A)=-\frac n2\log|A|+\sum_{i=1}^p\left(\frac n2\log\eta_i-\frac{\eta_i}2\right).$$ Each term is maximised at $\eta_i=n$, so the maximum is attained when $B=nI$, i.e. $\Sigma=A/n$. ∎
- Normal marginals do not make a normal vector. $X\sim N(0,1)$ and $Y=SX$ with an independent random sign $S$ (Example 2); or the density $2\phi(x)\phi(y)$ on $\{xy\gt0\}$ (Exercise I.7). Both have $N(0,1)$ marginals and are not bivariate normal.
- Uncorrelated is not independent unless the variables are jointly normal. In Example 2, $\operatorname{Cov}(X,Y)=0$ but $|X|=|Y|$.
- Pairwise is enough — but only for jointly normal vectors. For $X\sim N_p$, pairwise independent components are mutually independent; for general random vectors this fails (three events from two fair coins, §1.2).
- A singular $\Sigma$ is allowed. $(X,2X+1)$ with $X\sim N(0,1)$ is bivariate normal (every $aX+b(2X+1)$ is normal) but has no density: it lives on a line.
- Entries in $[-1,1]$ do not make a correlation matrix. The $3\times3$ matrix with off-diagonal entries $-0.6$ has $\operatorname{Var}(X_1+X_2+X_3)=-0.6$; equicorrelation needs $\rho\ge-1/(p-1)$.
- The conditional variance is constant only for the normal law. For non-normal vectors $\operatorname{Var}(X_1\mid X_2=x_2)$ usually depends on $x_2$, and $E[X_1\mid X_2]$ need not be linear.
- Marginal independence is not conditional independence. In Exercise II.3, $X_1$ and $X_3$ are independent, but given $X_2$ they are negatively correlated; conversely variables can be dependent but conditionally independent (§7.2).
- Right mean, wrong law. $Z_1^2+Z_1Z_2+Z_2^2$ has mean $2$ but is not $\chi^2_2$ (variance $5$, Example 6). Chi-square needs an idempotent matrix; the trace only gives the degrees of freedom after idempotency is checked.
- With correlated variables, check $A\Sigma A=A$, not $A^2=A$. If $X\sim N_2(0,\Sigma)$ with $\rho\ne0$, then $X_1^2+X_2^2$ is not chi-square although $A=I$ is idempotent; $X^{\mathsf T}\Sigma^{-1}X$ is.
- Independence of forms needs $AB=0$ (or $A\Sigma B=0$), not $AB=BA$ and not merely zero correlation of the variables involved. $Z_1^2$ and $(Z_1+Z_2)^2$ are dependent.
- The sum of two forms is not automatically chi-square. If $P$ is idempotent, $Z^{\mathsf T}PZ+Z^{\mathsf T}PZ=2Z^{\mathsf T}PZ$ is $2\chi^2_r$, not $\chi^2_{2r}$.
- Wishart pitfalls. The degrees of freedom of $A=\sum(X_i-\bar X)(X_i-\bar X)^{\mathsf T}$ are $n-1$, not $n$; $W_p(m,\Sigma)$ is singular when $m\lt p$ (so $S$ is singular when $n\le p$); off-diagonal entries are not chi-square; and sums of independent Wisharts are Wishart only with a common $\Sigma$.
- Independence of $\bar X$ and $S$ characterises normality. For non-normal samples it fails: with $n=2$ Bernoulli$(\tfrac12)$ observations, $S^2=\tfrac12$ when $\bar X=\tfrac12$ and $S^2=0$ otherwise.
Exercises
- 1 $X$ has mean $\mu=(2,-1,0)^{\mathsf T}$ and covariance $\Sigma=\begin{pmatrix}3&1&0\\1&2&-1\\0&-1&4\end{pmatrix}$. Check that $\Sigma\gt0$ and find $E$ and $\operatorname{Var}$ of $2X_1-X_2+3X_3$, and $\operatorname{Cov}(X_1+X_3,\ X_2-X_3)$.
- 2 Prove that the equicorrelation matrix $(1-\rho)I_p+\rho J$ has eigenvalues $1-\rho$ (multiplicity $p-1$) and $1+(p-1)\rho$, and find its determinant and its inverse.
- 3 $X$ has covariance $\Sigma=\begin{pmatrix}1&1&2\\1&2&3\\2&3&5\end{pmatrix}$. Show that $\Sigma$ is singular and find a linear combination of $X_1,X_2,X_3$ that is constant with probability $1$.
- 4 Show that $E\|X-a\|^2$ is minimised over $a\in\mathbb R^p$ at $a=\mu$, with minimum $\operatorname{tr}\Sigma$.
- 5 For $X\sim N_2(\mu,\Sigma)$ with $\mu=(0,1)^{\mathsf T}$ and $\Sigma=\begin{pmatrix}1&0.5\\0.5&2\end{pmatrix}$, compute $E\big[e^{X_1+X_2}\big]$.
- 6 Let $Z\sim N_p(0,I)$ and $Q$ be orthogonal. Show $QZ\sim N_p(0,I)$; deduce that $(Z_1+Z_2)/\sqrt2$ and $(Z_1-Z_2)/\sqrt2$ are iid $N(0,1)$.
- 7 $(X,Y)$ has density $2\phi(x)\phi(y)$ on $\{xy\gt0\}$ and $0$ elsewhere. Show that $X$ and $Y$ are $N(0,1)$, compute $\operatorname{Corr}(X,Y)$, and show that $(X,Y)$ is not bivariate normal.
- 1 Find the conditional law of $X_1$ given $X_2=2$.
- 2 Find the conditional law of $X_1$ given $(X_2,X_3)=(2,2)$. Why does the value of $X_3$ matter, although $X_1$ and $X_3$ are uncorrelated?
- 3 Are $X_1$ and $X_3$ independent? Are they conditionally independent given $X_2$?
- 4 Let $X\sim N_2(\mu,\Sigma)$. Show that $X_1+X_2$ and $X_1-X_2$ are independent if and only if $\sigma_{11}=\sigma_{22}$.
- 5 For $\Sigma=\begin{pmatrix}4&-2\\-2&2\end{pmatrix}$ and $\mu=0$, find the Mahalanobis distance of $x=(2,0)$ and $P(D^2\gt D^2(x))$.
- 6 Find the directions and half-lengths of the axes of the $90\%$ probability ellipse of $N_2\left(0,\begin{pmatrix}5&2\\2&2\end{pmatrix}\right)$.
- 7 For $X\sim N_p(\mu,\Sigma)$, $\Sigma\gt0$, show $E[D^2]=p$ and $\operatorname{Var}(D^2)=2p$, and that $P(D^2\le p)\to\tfrac12$ as $p\to\infty$. What does this say about where high-dimensional normal points lie?
- 8 Derive the conditional law of $Y$ given $X=x$ for the bivariate normal law of §1.4 from Theorem 7.6.
- 1 For $n=3$, decide which forms are chi-square and give the degrees of freedom: (a) $\tfrac13(Z_1+Z_2+Z_3)^2$; (b) $\tfrac23(Z_1^2+Z_2^2+Z_3^2-Z_1Z_2-Z_1Z_3-Z_2Z_3)$; (c) $Z_1^2+2Z_1Z_2+Z_2^2+Z_3^2$; (d) $\tfrac12(Z_1^2+Z_2^2)+Z_1Z_2$.
- 2 $X\sim N_2(0,\Sigma)$ with $\Sigma=\begin{pmatrix}2&1\\1&3\end{pmatrix}$. Find $c$ such that $c(X_1-X_2)^2\sim\chi^2_1$.
- 3 $Z\sim N_2(\mu,I)$ with $\mu=(1,1)^{\mathsf T}$ and $Q=Z_1^2+Z_1Z_2+Z_2^2$. Find $E[Q]$ and $\operatorname{Var}(Q)$.
- 4 $X\sim N_n(\theta\mathbf 1,\sigma^2I)$. Use Theorems 7.10 and 7.11 to show that $\sum(X_i-\bar X)^2$ and $\bar X$ are independent, and find the law of $n\bar X^2/\sigma^2$.
- 5 For $n=4$ let $Q_1=\tfrac14(Z_1+Z_2+Z_3+Z_4)^2$, $Q_2=\tfrac12(Z_1-Z_2)^2$, $Q_3=\tfrac12(Z_3-Z_4)^2$, $Q_4=\tfrac14(Z_1+Z_2-Z_3-Z_4)^2$. Show that Cochran's theorem applies, and find the law of $\dfrac{2Q_4}{Q_2+Q_3}$.
- 6 Prove that a symmetric idempotent $A$ satisfies $\operatorname{tr}A=\operatorname{rank}A$, and that $I-A$ is symmetric idempotent of rank $n-\operatorname{rank}A$.
- 7 Show that $\operatorname{Cov}(Z^{\mathsf T}AZ,Z^{\mathsf T}BZ)=2\operatorname{tr}(AB)$ for symmetric $A,B$. Deduce that for symmetric idempotent $P,Q$ the forms $Z^{\mathsf T}PZ$ and $Z^{\mathsf T}QZ$ are independent as soon as they are uncorrelated.
- 8 $Z\sim N_3(\mu,I)$ with $\mu=(1,2,2)^{\mathsf T}$. Give the law of $Z^{\mathsf T}Z$ and its mean and variance.
- 1 $W\sim W_3(10,\Sigma)$ with $\Sigma=\begin{pmatrix}2&1&0\\1&2&1\\0&1&2\end{pmatrix}$. Find $E[W]$, $\operatorname{Var}(W_{12})$, the law of $W_{11}+2W_{12}+W_{22}$, and the number $c$ with $P(W_{11}\gt c)=0.05$.
- 2 Prove that $\operatorname{tr}(\Sigma^{-1}W)\sim\chi^2_{mp}$ for $W\sim W_p(m,\Sigma)$, $\Sigma\gt0$.
- 3 For the four observations $(1,2),(3,3),(2,5),(6,6)$ compute $\bar x$, $S$, $|S|$ and the sample correlation.
- 4 Prove that the sample covariance matrix of $n\le p$ observations in $\mathbb R^p$ is singular, whatever the population.
- 5 For $W\sim W_2(m,\Sigma)$ with $m\ge2$ and $\Sigma\gt0$, prove $E|W|=m(m-1)|\Sigma|$ by writing $|W|=W_{22}\,W_{11\cdot2}$.
- 6 Show that $E[S]=\Sigma$ for iid observations from any distribution with finite second moments, but that the Wishart law of $(n-1)S$ needs normality (consider the variance of $S_{11}$ when $p=1$).
- 7 Show that $\bar X$ and $S^2$ are dependent for a sample of size $2$ from the Bernoulli$(\tfrac12)$ law.
E1. CSIR Part B GATE ST For which real $\rho$ is $\Sigma=\begin{pmatrix}1&\rho&0\\\rho&1&\rho\\0&\rho&1\end{pmatrix}$ the covariance matrix of a random vector with a density on $\mathbb R^3$? (A) $|\rho|\lt1$; (B) $|\rho|\lt1/\sqrt2$; (C) $|\rho|\lt1/2$; (D) $0\le\rho\lt1$.
E2. CSIR Part B $(X,Y,Z)$ is trivariate normal with zero means, unit variances and all three correlations equal to $\tfrac12$. Then $\operatorname{Var}(X\mid Y,Z)$ equals (A) $\tfrac12$; (B) $\tfrac23$; (C) $\tfrac34$; (D) $\tfrac13$.
E3. CSIR Part B Let $Z\sim N_4(0,I_4)$. Which of the following has the $\chi^2_2$ distribution? (A) $\tfrac12(Z_1+Z_2)^2+\tfrac12(Z_3-Z_4)^2$; (B) $Z_1^2+2Z_2^2$; (C) $\tfrac12(Z_1+Z_2+Z_3+Z_4)^2$; (D) $Z_1Z_2+Z_3Z_4$.
E4. CSIR Part C Let $(X_1,X_2)\sim N_2\left(0,\begin{pmatrix}1&\rho\\\rho&1\end{pmatrix}\right)$ with $0\lt\rho\lt1$. Which of the following are true? (One or more options may be correct.) (A) $X_1+X_2$ and $X_1-X_2$ are independent. (B) $X_1^2+X_2^2\sim\chi^2_2$. (C) $\dfrac{X_1^2-2\rho X_1X_2+X_2^2}{1-\rho^2}\sim\chi^2_2$. (D) $P(X_1\gt0,\ X_2\gt0)=\tfrac14$.
E5. CSIR Part C GATE ST Let $X_1,\dots,X_n$ be iid $N_p(\mu,\Sigma)$ with $\Sigma\gt0$ and $n\gt p$, and let $A=\sum(X_i-\bar X)(X_i-\bar X)^{\mathsf T}$. Which of the following are true? (A) $A\sim W_p(n,\Sigma)$. (B) $A_{11}/\sigma_{11}\sim\chi^2_{n-1}$. (C) $\bar X$ and $A$ are independent. (D) $\operatorname{tr}(\Sigma^{-1}A)\sim\chi^2_{p(n-1)}$.
E6. CSIR Part C Let $Z\sim N_n(0,I_n)$, $n\ge3$, and let $P$ and $Q$ be symmetric idempotent $n\times n$ matrices of ranks $r\ge1$ and $s\ge1$. Which of the following are true? (A) If $PQ=0$, then $\dfrac{Z^{\mathsf T}PZ/r}{Z^{\mathsf T}QZ/s}\sim F_{r,s}$. (B) $Z^{\mathsf T}PZ+Z^{\mathsf T}QZ\sim\chi^2_{r+s}$ for all such $P,Q$. (C) If $P+Q=I_n$, then $Z^{\mathsf T}PZ$ and $Z^{\mathsf T}QZ$ are independent. (D) If $\operatorname{tr}(PQ)=0$, then $Z^{\mathsf T}PZ$ and $Z^{\mathsf T}QZ$ are independent.
Suggested plan — 3 lectures × 60 min.
- Random vectors and the multivariate normal. Definition 7.1 and Theorem 7.1 with Example 1 on the board (ask the class whether the $-0.6$ matrix is valid before computing). Definition 7.2 and Theorem 7.2 — stress that the definition through linear combinations makes everything else short. Project Interactive 1: change the factor $A$ and ask what stays the same. Theorems 7.3–7.5, Examples 2 and 3.
- Conditioning and distances. Theorem 7.6 by the "subtract the regression" trick, Theorem 7.7 by block elimination, Example 4 (including the precision-matrix check). Definition 7.3, Theorem 7.8 and Example 5; return to the chapter-intro teaser and explain why red points exist. Assign Exercise II.2–3 as a discussion problem.
- Quadratic forms and Wishart. Definition 7.4, Theorems 7.9–7.11, Example 6; project Interactive 2 in the order centring → $Z_1^2+Z_1Z_2+Z_2^2$ → $X_1^2+X_2^2$ → $X^{\mathsf T}\Sigma^{-1}X$. Prove Cochran's theorem (Theorem 7.12) in full. Definition 7.5, Theorems 7.13 (parts 1–4 in detail, 5–6 sketched) and 7.14 via the Helmert matrix, Example 7. Finish with E4 and E6 live.
Discussion questions. (1) Why is it better to define the multivariate normal by linear combinations than by its density? What would be lost for a singular $\Sigma$? (2) Two variables are uncorrelated and each is normal. What extra information would you need before concluding independence, and how could you check it from data? (3) A quadratic form has the right mean for $\chi^2_k$. What simple numerical check could reveal that it is not chi-square?
Misconceptions to watch for. "Normal marginals ⇒ jointly normal"; "uncorrelated ⇒ independent" without joint normality; using $A^2=A$ instead of $A\Sigma A=A$ for correlated variables; taking the degrees of freedom of a non-idempotent form to be its trace; forgetting that $A=\sum(X_i-\bar X)(X_i-\bar X)^{\mathsf T}$ has $n-1$ degrees of freedom; believing that a marginally independent pair stays independent after conditioning; writing $\Sigma_{11}-\Sigma_{21}\Sigma_{22}^{-1}\Sigma_{12}$ with the blocks in the wrong order (the dimensions do not even match unless $q=p-q$).
Suggested homework. I(1, 2, 3, 7), II(2, 3, 5, 6), III(1, 2, 5, 7), IV(1, 3, 5, 7); Exam corner E2, E5, E6.
Research question. Are real multivariate measurements close enough to normal for the methods of this chapter, and can the Mahalanobis distance pick out the observations that are not?
Why it matters. Hotelling's $T^2$, discriminant analysis and multivariate control charts all assume approximate multivariate normality, and all of them flag "unusual" observations through a Mahalanobis distance. Hospital laboratories screen blood panels this way, aircraft-engine monitoring systems flag sensor vectors whose combination is unusual even when each reading is within limits, and credit-card fraud filters work on the same principle. If the data are far from normal, the $\chi^2_p$ cut-offs are wrong and the false-alarm rate is not what the user believes.
What to do.
- Take a public multivariate data set with a few continuous variables — for example the four flower measurements of one species in the classical iris data, or a few physical measurements of athletes from a sports-science data set.
- Compute $\bar x$, $S$ and the squared distances $d_i^2=(x_i-\bar x)^{\mathsf T}S^{-1}(x_i-\bar x)$. Check numerically that $\sum d_i^2=(n-1)p$, and explain why from $\operatorname{tr}(S^{-1}A)$.
- Draw a chi-square Q–Q plot: the ordered $d_{(i)}^2$ against the $\chi^2_p$ quantiles at levels $(i-\tfrac12)/n$. A straight line through the origin with slope $1$ supports normality; an upward bend at the top points to heavy tails or outliers.
- Repeat the plot for simulated $N_p(\bar x,S)$ data of the same size, five times, to see how much a Q–Q plot wobbles when the model is exactly right.
- Compute Mardia's kurtosis $b_{2,p}=\frac1n\sum d_i^4$ (with $S$ replaced by $A/n$) and compare it with its normal-theory value $p(p+2)$.
Tools: Python (numpy, scipy, matplotlib) or R; a spreadsheet is enough for $p=2$.
Hand in: The two Q–Q plots (real and one simulated), the observations you would flag at $\chi^2_{p,0.01}$, the value of $b_{2,p}$ and a paragraph on whether the normal model is adequate.
Going further. With $\bar x$ and $S$ estimated from the same data, $\frac{n}{(n-1)^2}d_i^2$ has exactly a $\operatorname{Beta}\left(\tfrac p2,\tfrac{n-p-1}2\right)$ law under normality; compare these beta quantiles with the $\chi^2_p$ quantiles for small $n$. Read about robust alternatives (minimum covariance determinant estimates) that prevent outliers from masking themselves by inflating $S$.
🧪 Self-check quiz
Q1. A random vector $X\in\mathbb R^p$ is multivariate normal if and only if
Q2. If $(X_1,X_2)\sim N_2(\mu,\Sigma)$ and $\sigma_{12}=0$, then
Q3. For $X\sim N_p(\mu,\Sigma)$ partitioned into $(X_1,X_2)$ with $\Sigma_{22}\gt0$, the conditional covariance of $X_1$ given $X_2=x_2$ is
Q4. Let $Z\sim N_n(0,I)$ and $A$ symmetric. $Z^{\mathsf T}AZ$ is chi-square if and only if
Q5. For a sample of size $n$ from $N_p(\mu,\Sigma)$, the matrix $A=\sum(X_i-\bar X)(X_i-\bar X)^{\mathsf T}$ has the law