Chapter 7 · Multivariate Analysis

7.1 The Multivariate Normal and Wishart Distributions

Mean vectors and covariance matrices; the multivariate normal law defined through linear combinations, with its density, marginals and conditionals; the Mahalanobis distance; when a quadratic form is chi-square (Cochran's theorem); and the Wishart law of the sample covariance matrix.

Statistics with one variable runs on a few facts about the normal law: linear combinations of independent normals are normal, $\bar X$ and $S^2$ are independent, and $(n-1)S^2/\sigma^2$ is $\chi^2_{n-1}$. Almost every multivariate method rests on the matrix versions of these facts. The variance becomes a covariance matrix $\Sigma$; the normal law becomes the multivariate normal law $N_p(\mu,\Sigma)$, best defined by the requirement that every linear combination of the coordinates is normal; the statistic $(n-1)S^2$ becomes the matrix of sums of squares and products, whose law is the Wishart law. The same machinery explains the analysis of variance: a sum of squares is a quadratic form $X^{\mathsf T}AX$, and Cochran's theorem says exactly when such forms are independent chi-square variables. Section 6.1 quoted that theorem; here we prove it.

The section is also a catalogue of traps, and CSIR NET uses them. Two normal variables need not be jointly normal; uncorrelated normal variables need not be independent unless they are jointly normal; a quadratic form with the right mean need not be chi-square; and the sum of two Wishart matrices need not be Wishart. Each trap fails for a reason that the theorems below make precise, and the ⚠️ box near the end collects them.

Random Vectors, Mean Vectors and Covariance Matrices

A random vector $X=(X_1,\dots,X_p)^{\mathsf T}$ is a column of random variables defined on one probability space. Expectations of vectors and matrices are taken entry by entry.

📘 Definition 7.1 — Mean vector, covariance matrix, correlation matrix

Let $X$ be a random vector with $E[X_i^2]\lt\infty$ for every $i$.

  • The mean vector is $\mu=E[X]=(E[X_1],\dots,E[X_p])^{\mathsf T}$.
  • The covariance matrix is $\Sigma=\operatorname{Cov}(X)=E\big[(X-\mu)(X-\mu)^{\mathsf T}\big]$, the $p\times p$ matrix with entries $\sigma_{ij}=\operatorname{Cov}(X_i,X_j)$; in particular $\sigma_{ii}=\operatorname{Var}(X_i)$.
  • For a second random vector $Y$ with mean $\nu$, the cross-covariance is $\operatorname{Cov}(X,Y)=E\big[(X-\mu)(Y-\nu)^{\mathsf T}\big]$.
  • If every $\sigma_{ii}\gt0$, the correlation matrix is $\mathcal P=D^{-1/2}\Sigma D^{-1/2}$ with $D=\operatorname{diag}(\sigma_{11},\dots,\sigma_{pp})$; its entries are $\rho_{ij}=\sigma_{ij}/\sqrt{\sigma_{ii}\sigma_{jj}}$ and its diagonal is all $1$s.

For a symmetric matrix $A$ we write $A\ge0$ if $a^{\mathsf T}Aa\ge0$ for all $a$ (nonnegative definite) and $A\gt0$ if $a^{\mathsf T}Aa\gt0$ for all $a\ne0$ (positive definite).

📐 Theorem 7.1 — Moments of linear transformations

Let $X$ have mean $\mu$ and covariance matrix $\Sigma$, and let $A$ ($q\times p$), $B$ ($r\times p$) and $b\in\mathbb R^q$ be constant.

  1. $E[AX+b]=A\mu+b$, $\ \operatorname{Cov}(AX+b)=A\Sigma A^{\mathsf T}$ and $\operatorname{Cov}(AX,BX)=A\Sigma B^{\mathsf T}$. In particular $\operatorname{Var}(a^{\mathsf T}X)=a^{\mathsf T}\Sigma a$ and $\operatorname{Cov}(a^{\mathsf T}X,c^{\mathsf T}X)=a^{\mathsf T}\Sigma c$.
  2. $\Sigma$ is symmetric and $\Sigma\ge0$.
  3. $\Sigma$ is singular if and only if some nontrivial linear combination is constant: $a^{\mathsf T}X=a^{\mathsf T}\mu$ with probability $1$ for some $a\ne0$. More precisely, $X-\mu$ lies in the column space $\mathcal C(\Sigma)$ with probability $1$.
  4. For every $p\times p$ matrix $A$, $E[X^{\mathsf T}AX]=\operatorname{tr}(A\Sigma)+\mu^{\mathsf T}A\mu$.

Proof. (1) Each entry of $AX+b$ is $\sum_ja_{ij}X_j+b_i$, whose mean is $\sum_ja_{ij}\mu_j+b_i$ by linearity. Then $AX+b-(A\mu+b)=A(X-\mu)$, so $$\operatorname{Cov}(AX+b)=E\big[A(X-\mu)(X-\mu)^{\mathsf T}A^{\mathsf T}\big]=A\,E\big[(X-\mu)(X-\mu)^{\mathsf T}\big]A^{\mathsf T}=A\Sigma A^{\mathsf T},$$ because a constant matrix can be taken out of an expectation on either side (again by linearity, entry by entry). The cross-covariance is identical with $B$ on the right.

(2) $\sigma_{ij}=\sigma_{ji}$, and by (1), $a^{\mathsf T}\Sigma a=\operatorname{Var}(a^{\mathsf T}X)\ge0$.

(3) $\Sigma$ is singular iff $\Sigma a=0$ for some $a\ne0$. For symmetric $\Sigma\ge0$, $\Sigma a=0$ iff $a^{\mathsf T}\Sigma a=0$ (if $a^{\mathsf T}\Sigma a=0$, write $\Sigma=\Sigma^{1/2}\Sigma^{1/2}$; then $\|\Sigma^{1/2}a\|^2=0$, so $\Sigma a=\Sigma^{1/2}(\Sigma^{1/2}a)=0$). And $a^{\mathsf T}\Sigma a=\operatorname{Var}(a^{\mathsf T}X)=0$ iff $a^{\mathsf T}X=a^{\mathsf T}\mu$ almost surely. For the last claim, the null space of $\Sigma$ is $\mathcal C(\Sigma)^\perp$; pick a basis $a_1,\dots,a_k$ of it. Each event $\{a_j^{\mathsf T}(X-\mu)=0\}$ has probability $1$, so their intersection does, and on it $X-\mu\perp\mathcal C(\Sigma)^\perp$, i.e. $X-\mu\in\mathcal C(\Sigma)$.

(4) $X^{\mathsf T}AX$ is a scalar, so it equals its own trace, and $\operatorname{tr}(X^{\mathsf T}AX)=\operatorname{tr}(AXX^{\mathsf T})$. Since $E[XX^{\mathsf T}]=\Sigma+\mu\mu^{\mathsf T}$, we get $E[X^{\mathsf T}AX]=\operatorname{tr}(A\Sigma)+\operatorname{tr}(A\mu\mu^{\mathsf T})$, and $\operatorname{tr}(A\mu\mu^{\mathsf T})=\mu^{\mathsf T}A\mu$. ∎

Conversely, every symmetric $\Sigma\ge0$ is a covariance matrix: if $Z$ has independent standard components and $AA^{\mathsf T}=\Sigma$ (for instance $A=\Sigma^{1/2}$), then $\operatorname{Cov}(AZ)=AIA^{\mathsf T}=\Sigma$ by part (1). So "is this matrix a covariance matrix?" means exactly "is it symmetric and nonnegative definite?", and for a correlation matrix one also needs unit diagonal.

✏️ Example 1 — Is it a covariance matrix? Variances of combinations
(a) For which $\rho$ is the $p\times p$ equicorrelation matrix $E_\rho=(1-\rho)I+\rho J$ ($J$ the all-ones matrix) a correlation matrix? Is the $3\times3$ matrix with unit diagonal and all off-diagonal entries $-0.6$ one? (b) Let $X$ have covariance matrix $$\Sigma=\begin{pmatrix}4&2&-1\\2&3&0\\-1&0&2\end{pmatrix}.$$ Check that $\Sigma\gt0$, and find $\operatorname{Var}(X_1-X_2+2X_3)$ and $\operatorname{Cov}(X_1+X_2,\ X_2-X_3)$.
(a) Eigenvalues $J\mathbf 1=p\mathbf 1$ and $Jv=0$ for every $v\perp\mathbf 1$, so $E_\rho$ has eigenvalue $1-\rho+p\rho=1+(p-1)\rho$ on $\mathbf 1$ and $1-\rho$ on the $(p-1)$-dimensional space $\mathbf 1^\perp$. It is nonnegative definite iff both are $\ge0$: $$-\frac{1}{p-1}\le\rho\le1.$$
(a) The $-0.6$ matrix For $p=3$ the lower limit is $-\tfrac12$, so $\rho=-0.6$ is not allowed, although every entry lies in $[-1,1]$. Directly: if it were the correlation matrix of standardised $X_1,X_2,X_3$, then $\operatorname{Var}(X_1+X_2+X_3)=3+6(-0.6)=-0.6\lt0$. Three variables cannot all be strongly negatively correlated with one another.
(b) Positive definite Leading principal minors: $4$, $\ 4\cdot3-2^2=8$, $\ |\Sigma|=4(6-0)-2(4-0)+(-1)(0+3)=13$. All are positive, so $\Sigma\gt0$ (Sylvester's criterion).
(b) Variance With $a=(1,-1,2)^{\mathsf T}$: $\Sigma a=(4-2-2,\ 2-3+0,\ -1-0+4)^{\mathsf T}=(0,-1,3)^{\mathsf T}$, so $\operatorname{Var}(a^{\mathsf T}X)=a^{\mathsf T}\Sigma a=0+1+6=7$.
(b) Covariance With $u=(1,1,0)^{\mathsf T}$ and $v=(0,1,-1)^{\mathsf T}$: $\Sigma v=(2+1,\ 3-0,\ 0-2)^{\mathsf T}=(3,3,-2)^{\mathsf T}$ and $u^{\mathsf T}\Sigma v=6$. Check by bilinearity: $\operatorname{Cov}(X_1+X_2,X_2-X_3)=\sigma_{12}-\sigma_{13}+\sigma_{22}-\sigma_{23}=2+1+3-0=6$. ✓

The Multivariate Normal Distribution

There are several equivalent definitions. The one below needs no density, covers singular covariance matrices, and makes the key properties one-line consequences.

📘 Definition 7.2 — Multivariate normal distribution

A random vector $X$ in $\mathbb R^p$ is multivariate normal (its components are jointly normal) if $a^{\mathsf T}X$ has a univariate normal distribution for every $a\in\mathbb R^p$. Here a constant $c$ counts as the degenerate normal law $N(c,0)$. We shall see that the law of $X$ is then determined by $\mu=E[X]$ and $\Sigma=\operatorname{Cov}(X)$, and we write $X\sim N_p(\mu,\Sigma)$.

📐 Theorem 7.2 — Characteristic function, existence and representation
  1. If $X$ is multivariate normal with mean $\mu$ and covariance $\Sigma$, its characteristic function is $$\varphi_X(t)=E\big[e^{it^{\mathsf T}X}\big]=\exp\left(it^{\mathsf T}\mu-\tfrac12t^{\mathsf T}\Sigma t\right),\qquad t\in\mathbb R^p,$$ and its moment generating function is $M_X(t)=\exp\left(t^{\mathsf T}\mu+\tfrac12t^{\mathsf T}\Sigma t\right)$. Hence the law of $X$ depends only on $(\mu,\Sigma)$.
  2. Conversely, for every $\mu\in\mathbb R^p$ and every symmetric $\Sigma\ge0$ there is such a vector: if $Z=(Z_1,\dots,Z_k)^{\mathsf T}$ has independent $N(0,1)$ components and $A$ is $p\times k$ with $AA^{\mathsf T}=\Sigma$, then $X=\mu+AZ\sim N_p(\mu,\Sigma)$.
  3. Every $X\sim N_p(\mu,\Sigma)$ has the same law as $\mu+\Sigma^{1/2}Z$ with $Z\sim N_p(0,I_p)$.

Proof. (1) Fix $t$. The variable $Y=t^{\mathsf T}X$ is normal by definition, with mean $t^{\mathsf T}\mu$ and variance $t^{\mathsf T}\Sigma t$ (Theorem 7.1). The characteristic function of $N(m,s^2)$ at the point $1$ is $e^{im-s^2/2}$ (§2.2), so $\varphi_X(t)=E[e^{iY}]=\exp\left(it^{\mathsf T}\mu-\tfrac12t^{\mathsf T}\Sigma t\right)$. The mgf is computed in the same way from $E[e^{Y}]=e^{m+s^2/2}$. A probability law on $\mathbb R^p$ is determined by its characteristic function (the uniqueness theorem of §2.2, which holds in $\mathbb R^p$ with the same proof), so the law of $X$ is determined by $(\mu,\Sigma)$.

(2) For $a\in\mathbb R^p$, $a^{\mathsf T}X=a^{\mathsf T}\mu+(A^{\mathsf T}a)^{\mathsf T}Z=a^{\mathsf T}\mu+\sum_jc_jZ_j$ with $c=A^{\mathsf T}a$. A linear combination of independent normal variables is normal (the reproductive property, §2.1), so $a^{\mathsf T}X$ is normal and $X$ is multivariate normal. By Theorem 7.1, $E[X]=\mu$ and $\operatorname{Cov}(X)=AA^{\mathsf T}=\Sigma$. A suitable $A$ always exists: by the spectral theorem $\Sigma=Q\Lambda Q^{\mathsf T}$ with $Q$ orthogonal and $\Lambda=\operatorname{diag}(\lambda_i)$, $\lambda_i\ge0$, and $A=\Sigma^{1/2}=Q\Lambda^{1/2}Q^{\mathsf T}$ works; so does the Cholesky factor when $\Sigma\gt0$.

(3) $\mu+\Sigma^{1/2}Z$ is multivariate normal with mean $\mu$ and covariance $\Sigma$ by (2), so by (1) it has the same law as $X$. ∎

Part (1) contains the Cramér–Wold device: the law of a random vector is determined by the laws of all its linear combinations $a^{\mathsf T}X$, since these give $\varphi_X(t)$ for every $t$. Part (2) shows that many different matrices $A$ give the same law — Interactive 1 lets you watch this happen.

📐 Theorem 7.3 — The density of $N_p(\mu,\Sigma)$

If $\Sigma\gt0$, then $X\sim N_p(\mu,\Sigma)$ has the density $$f(x)=\frac{1}{(2\pi)^{p/2}|\Sigma|^{1/2}}\exp\left(-\tfrac12(x-\mu)^{\mathsf T}\Sigma^{-1}(x-\mu)\right),\qquad x\in\mathbb R^p.$$ If $\Sigma$ is singular of rank $r\lt p$, $X$ has no density on $\mathbb R^p$: with probability $1$ it lies in the $r$-dimensional affine subspace $\mu+\mathcal C(\Sigma)$.

Proof. By Theorem 7.2(3) we may take $X=\mu+\Sigma^{1/2}Z$ with $Z\sim N_p(0,I)$. The components of $Z$ are independent, so $Z$ has density $\prod_i\phi(z_i)=(2\pi)^{-p/2}e^{-z^{\mathsf T}z/2}$. The map $z\mapsto x=\mu+\Sigma^{1/2}z$ is a bijection of $\mathbb R^p$ with inverse $z=\Sigma^{-1/2}(x-\mu)$, whose Jacobian determinant is $|\Sigma^{-1/2}|=|\Sigma|^{-1/2}$. By the change-of-variables formula (§1.4), $$f(x)=(2\pi)^{-p/2}|\Sigma|^{-1/2}\exp\left(-\tfrac12\,z^{\mathsf T}z\right),\qquad z^{\mathsf T}z=(x-\mu)^{\mathsf T}\Sigma^{-1/2}\Sigma^{-1/2}(x-\mu)=(x-\mu)^{\mathsf T}\Sigma^{-1}(x-\mu).$$ If $\Sigma$ is singular, $X-\mu\in\mathcal C(\Sigma)$ almost surely (Theorem 7.1(3)), and a proper subspace has Lebesgue measure $0$; a law that gives probability $1$ to a null set has no density. ∎

For $p=2$, $|\Sigma|=\sigma_1^2\sigma_2^2(1-\rho^2)$ and $\Sigma^{-1}=\frac{1}{1-\rho^2}\begin{pmatrix}1/\sigma_1^2&-\rho/(\sigma_1\sigma_2)\\-\rho/(\sigma_1\sigma_2)&1/\sigma_2^2\end{pmatrix}$, and the formula becomes the bivariate normal density of §1.4.

📐 Theorem 7.4 — Linear transformations and marginals

Let $X\sim N_p(\mu,\Sigma)$, $A$ a $q\times p$ matrix and $b\in\mathbb R^q$. Then $$AX+b\sim N_q\left(A\mu+b,\ A\Sigma A^{\mathsf T}\right).$$ In particular, partition $$X=\begin{pmatrix}X_1\\X_2\end{pmatrix},\quad\mu=\begin{pmatrix}\mu_1\\\mu_2\end{pmatrix},\quad\Sigma=\begin{pmatrix}\Sigma_{11}&\Sigma_{12}\\\Sigma_{21}&\Sigma_{22}\end{pmatrix}$$ with $X_1$ of length $q$. Then $X_1\sim N_q(\mu_1,\Sigma_{11})$: every subvector of a normal vector is normal. If $\Sigma\gt0$, $\ \Sigma^{-1/2}(X-\mu)\sim N_p(0,I_p)$, and for every orthogonal $Q$, $Q(X-\mu)$ has the same law as $X-\mu$ when $\Sigma=\sigma^2I$.

Proof. For $c\in\mathbb R^q$, $c^{\mathsf T}(AX+b)=(A^{\mathsf T}c)^{\mathsf T}X+c^{\mathsf T}b$ is a linear combination of $X$ plus a constant, hence normal. So $AX+b$ is multivariate normal, with the mean and covariance of Theorem 7.1. The marginal is the case $A=(I_q\ \ 0)$; standardisation is $A=\Sigma^{-1/2}$, giving covariance $\Sigma^{-1/2}\Sigma\Sigma^{-1/2}=I$; and $Q\sigma^2IQ^{\mathsf T}=\sigma^2I$. ∎

📐 Theorem 7.5 — For jointly normal vectors, uncorrelated means independent

Let $X=(X_1^{\mathsf T},X_2^{\mathsf T})^{\mathsf T}\sim N_p(\mu,\Sigma)$ be partitioned as in Theorem 7.4. Then $X_1$ and $X_2$ are independent if and only if $\Sigma_{12}=0$. More generally, blocks $X_1,\dots,X_k$ of a jointly normal vector are mutually independent if and only if all cross-covariance blocks vanish; in particular the components of $X\sim N_p(\mu,\Sigma)$ are mutually independent if and only if $\Sigma$ is diagonal.

Proof. Independence always implies zero covariance (when second moments exist). Conversely, suppose $\Sigma_{12}=0$ (so $\Sigma_{21}=\Sigma_{12}^{\mathsf T}=0$). For $t=(t_1^{\mathsf T},t_2^{\mathsf T})^{\mathsf T}$, $$t^{\mathsf T}\Sigma t=t_1^{\mathsf T}\Sigma_{11}t_1+2t_1^{\mathsf T}\Sigma_{12}t_2+t_2^{\mathsf T}\Sigma_{22}t_2=t_1^{\mathsf T}\Sigma_{11}t_1+t_2^{\mathsf T}\Sigma_{22}t_2,$$ so $\varphi_X(t)=\varphi_{X_1}(t_1)\varphi_{X_2}(t_2)$, using Theorem 7.2(1) for $X$ and, by Theorem 7.4, for $X_1$ and $X_2$. The right side is the characteristic function of the law under which $X_1$ and $X_2$ are independent with their given marginals; by uniqueness that is the law of $X$. The case of $k$ blocks is identical. ∎

The theorem is about joint normality. Example 2 shows that it fails for variables that are only marginally normal. A useful corollary: for jointly normal variables, pairwise independence of the components already implies mutual independence, because it makes $\Sigma$ diagonal.

✏️ Example 2 — Normal, uncorrelated, and still dependent
Let $X\sim N(0,1)$, let $S$ be independent of $X$ with $P(S=1)=P(S=-1)=\tfrac12$, and put $Y=SX$. Prove that (a) $Y\sim N(0,1)$; (b) $\operatorname{Cov}(X,Y)=0$; (c) $X$ and $Y$ are not independent; (d) $(X,Y)$ is not bivariate normal. Why does this not contradict Theorem 7.5?
(a) By independence and the symmetry of $N(0,1)$, $P(Y\le y)=\tfrac12P(X\le y)+\tfrac12P(-X\le y)=\tfrac12\Phi(y)+\tfrac12\Phi(y)=\Phi(y)$.
(b) $\operatorname{Cov}(X,Y)=E[XY]=E[S]\,E[X^2]=0\cdot1=0$.
(c) $|Y|=|X|$. So $P(|X|\le1,\ |Y|\gt1)=0$, whereas $P(|X|\le1)P(|Y|\gt1)=(0.6827)(0.3173)\gt0$.
(d) If $(X,Y)$ were bivariate normal, $X+Y$ would be normal. But $X+Y=(1+S)X$ equals $0$ with probability $\tfrac12$ (when $S=-1$) and is $N(0,4)$ otherwise: a mixture with an atom at $0$ that is not $0$ almost surely, so not normal.
Lesson Theorem 7.5 needs every linear combination of $(X,Y)$ to be normal, not just $X$ and $Y$. Normal marginals and zero correlation together say nothing about independence.
✏️ Example 3 — Linear combinations of a trivariate normal
Let $X\sim N_3(\mu,\Sigma)$ with $\mu=(1,-1,2)^{\mathsf T}$ and $\Sigma$ as in Example 1(b). (a) Find the law of $Y=X_1-X_2+2X_3$ and $P(Y\gt10)$. (b) Find the joint law of $U=X_1+X_2$ and $V=X_2-X_3$, and their correlation. (c) Find constants $b_2,b_3$ such that $X_1-b_2X_2-b_3X_3$ is independent of $(X_2,X_3)$.
(a) By Theorem 7.4, $Y$ is normal with mean $1+1+4=6$ and variance $7$ (Example 1). So $P(Y\gt10)=1-\Phi\big(4/\sqrt7\big)=1-\Phi(1.512)=0.0653$.
(b) $(U,V)^{\mathsf T}=AX$ with $A=\begin{pmatrix}1&1&0\\0&1&-1\end{pmatrix}$. Then $A\mu=(0,-3)^{\mathsf T}$, and $A\Sigma A^{\mathsf T}$ has entries $\operatorname{Var}U=4+3+2\cdot2=11$, $\operatorname{Var}V=3+2-0=5$ and $\operatorname{Cov}(U,V)=6$ (Example 1). So $$(U,V)^{\mathsf T}\sim N_2\left(\begin{pmatrix}0\\-3\end{pmatrix},\begin{pmatrix}11&6\\6&5\end{pmatrix}\right),\qquad\rho_{UV}=\frac{6}{\sqrt{55}}=0.809.$$
(c) $W=X_1-b_2X_2-b_3X_3$ and $(X_2,X_3)$ are jointly normal (one linear map of $X$), so by Theorem 7.5 independence is equivalent to $\operatorname{Cov}(W,X_2)=\operatorname{Cov}(W,X_3)=0$: $$\sigma_{12}-b_2\sigma_{22}-b_3\sigma_{32}=2-3b_2=0,\qquad\sigma_{13}-b_2\sigma_{23}-b_3\sigma_{33}=-1-2b_3=0.$$ So $b_2=\tfrac23$ and $b_3=-\tfrac12$. In matrix form $b^{\mathsf T}=\Sigma_{12}\Sigma_{22}^{-1}$ — the regression coefficients of the next subsection.
Interactive 1 — Building $N_2(0,\Sigma)$ from Independent Normals: $X=AZ$

Four hundred points $z$ are drawn from $N_2(0,I)$ (seeded). Press Play to slide each point from $z$ to $Az$, where $AA^{\mathsf T}=\Sigma$. The dashed circle $z^{\mathsf T}z=5.991$ is carried onto the ellipse $x^{\mathsf T}\Sigma^{-1}x=5.991$, whose axes are the eigenvectors of $\Sigma$ (purple) with half-lengths $\sqrt{5.991\lambda_i}$. Change the factor $A$: the red tracked point lands in a different place, but the cloud has the same law — only $AA^{\mathsf T}$ matters.

—
Σ = AAᵀ—
eigenvalues λ₁, λ₂—
sample covariance of the 400 points Az—
share inside the 95% ellipse—

Conditional Distributions and the Mahalanobis Distance

Example 3(c) already contains the idea of this subsection: subtract from $X_1$ the right multiple of $X_2$ and what is left is independent of $X_2$. Conditioning on $X_2$ then only shifts the mean.

📐 Theorem 7.6 — Conditional distributions

Let $X\sim N_p(\mu,\Sigma)$ be partitioned as in Theorem 7.4, with $\Sigma_{22}\gt0$. Put $$\Sigma_{11\cdot2}=\Sigma_{11}-\Sigma_{12}\Sigma_{22}^{-1}\Sigma_{21}.$$ Then the conditional law of $X_1$ given $X_2=x_2$ is $$X_1\mid X_2=x_2\ \sim\ N_q\left(\mu_1+\Sigma_{12}\Sigma_{22}^{-1}(x_2-\mu_2),\ \Sigma_{11\cdot2}\right).$$ Thus (i) the regression $E[X_1\mid X_2=x_2]$ is linear in $x_2$; (ii) the conditional covariance $\Sigma_{11\cdot2}$ does not depend on $x_2$; (iii) $\Sigma_{11\cdot2}\le\Sigma_{11}$, i.e. $\Sigma_{11}-\Sigma_{11\cdot2}\ge0$: conditioning never increases variances.

Proof. Let $B=\Sigma_{12}\Sigma_{22}^{-1}$ ($q\times(p-q)$) and $U=X_1-BX_2$. The vector $(U^{\mathsf T},X_2^{\mathsf T})^{\mathsf T}$ is a linear function of $X$, hence jointly normal (Theorem 7.4), and $$\operatorname{Cov}(U,X_2)=\Sigma_{12}-B\Sigma_{22}=0.$$ By Theorem 7.5, $U$ is independent of $X_2$. Its mean is $\mu_1-B\mu_2$ and its covariance is $$\operatorname{Cov}(U)=\Sigma_{11}-B\Sigma_{21}-\Sigma_{12}B^{\mathsf T}+B\Sigma_{22}B^{\mathsf T}=\Sigma_{11}-\Sigma_{12}\Sigma_{22}^{-1}\Sigma_{21}=\Sigma_{11\cdot2},$$ since $B\Sigma_{21}=\Sigma_{12}B^{\mathsf T}=B\Sigma_{22}B^{\mathsf T}=\Sigma_{12}\Sigma_{22}^{-1}\Sigma_{21}$. Now $X_1=U+BX_2$. Given $X_2=x_2$, the independent part $U$ keeps its law $N_q(\mu_1-B\mu_2,\Sigma_{11\cdot2})$, and $BX_2=Bx_2$ is a constant; so $X_1\mid X_2=x_2\sim N_q(\mu_1+B(x_2-\mu_2),\Sigma_{11\cdot2})$. Finally $\Sigma_{11}=\operatorname{Cov}(U+BX_2)=\Sigma_{11\cdot2}+B\Sigma_{22}B^{\mathsf T}$ and $B\Sigma_{22}B^{\mathsf T}\ge0$. ∎

📐 Theorem 7.7 — Partitioned determinant and inverse (Schur complement)

Let $\Sigma\gt0$ be partitioned as above and $\Omega=\Sigma^{-1}$ (the precision matrix), partitioned in the same way. Then $\Sigma_{11\cdot2}\gt0$ and

  1. $|\Sigma|=|\Sigma_{22}|\,|\Sigma_{11\cdot2}|$;
  2. $\Omega_{11}=\Sigma_{11\cdot2}^{-1}$ and $\Omega_{12}=-\Sigma_{11\cdot2}^{-1}\Sigma_{12}\Sigma_{22}^{-1}$;
  3. consequently the conditional covariance is $\Sigma_{11\cdot2}=\Omega_{11}^{-1}$ and the regression matrix is $\Sigma_{12}\Sigma_{22}^{-1}=-\Omega_{11}^{-1}\Omega_{12}$.

Proof. With $B=\Sigma_{12}\Sigma_{22}^{-1}$ and $T=\begin{pmatrix}I&-B\\0&I\end{pmatrix}$, block multiplication gives $$T\,\Sigma\,T^{\mathsf T}=\begin{pmatrix}\Sigma_{11\cdot2}&0\\0&\Sigma_{22}\end{pmatrix}.$$ (This is the covariance matrix of $(U,X_2)$ from the last proof.) Since $|T|=1$, taking determinants gives (1), and $\Sigma_{11\cdot2}\gt0$ because $T\Sigma T^{\mathsf T}\gt0$. Inverting, $T^{-\mathsf T}\Sigma^{-1}T^{-1}=\operatorname{diag}(\Sigma_{11\cdot2}^{-1},\Sigma_{22}^{-1})$, so $$\Sigma^{-1}=T^{\mathsf T}\begin{pmatrix}\Sigma_{11\cdot2}^{-1}&0\\0&\Sigma_{22}^{-1}\end{pmatrix}T=\begin{pmatrix}\Sigma_{11\cdot2}^{-1}&-\Sigma_{11\cdot2}^{-1}B\\-B^{\mathsf T}\Sigma_{11\cdot2}^{-1}&\Sigma_{22}^{-1}+B^{\mathsf T}\Sigma_{11\cdot2}^{-1}B\end{pmatrix},$$ which gives (2); (3) follows by solving (2) for $\Sigma_{11\cdot2}$ and $B$. ∎

Part (3) is the bridge to partial correlation in §7.2: the entries of the precision matrix describe each variable given all the others. It also shows that the density factorises as $f(x)=f_{X_2}(x_2)\,f_{X_1\mid X_2}(x_1\mid x_2)$ with the two normal densities of Theorems 7.4 and 7.6, as it must.

✏️ Example 4 — Conditioning in three dimensions
With $\mu$ and $\Sigma$ as in Example 3, find (a) the conditional law of $X_1$ given $(X_2,X_3)=(0,3)$; (b) the conditional law of $(X_1,X_2)$ given $X_3=1$. (c) Check (a) against the precision matrix.
(a) Regression $\Sigma_{12}=(2,-1)$ and $\Sigma_{22}=\operatorname{diag}(3,2)$, so $B=\Sigma_{12}\Sigma_{22}^{-1}=\left(\tfrac23,-\tfrac12\right)$, the constants of Example 3(c). The conditional mean is $$1+\tfrac23\,(0-(-1))-\tfrac12\,(3-2)=1+\tfrac23-\tfrac12=\tfrac76.$$
(a) Variance $\Sigma_{11\cdot2}=4-B\Sigma_{21}=4-\left(\tfrac23\cdot2+\tfrac12\cdot1\right)=4-\tfrac{11}{6}=\tfrac{13}{6}$. So $X_1\mid(X_2,X_3)=(0,3)\sim N\left(\tfrac76,\tfrac{13}6\right)$; knowing $X_2$ and $X_3$ cuts the variance of $X_1$ from $4$ to $2.17$.
(b) Now $X_1=(X_1,X_2)$, $X_2=X_3$: $\Sigma_{12}=(-1,0)^{\mathsf T}$, $\Sigma_{22}=2$, $B=(-\tfrac12,0)^{\mathsf T}$. Mean $\begin{pmatrix}1\\-1\end{pmatrix}+\begin{pmatrix}-1/2\\0\end{pmatrix}(1-2)=\begin{pmatrix}1.5\\-1\end{pmatrix}$, covariance $$\begin{pmatrix}4&2\\2&3\end{pmatrix}-\frac12\begin{pmatrix}1&0\\0&0\end{pmatrix}=\begin{pmatrix}3.5&2\\2&3\end{pmatrix}.$$ $X_2$ is uncorrelated with $X_3$, so its conditional law is its marginal law.
(c) Check $|\Sigma|=13$ and the cofactors give $$\Omega=\Sigma^{-1}=\frac{1}{13}\begin{pmatrix}6&-4&3\\-4&7&-2\\3&-2&8\end{pmatrix}.$$ Theorem 7.7: $\Sigma_{11\cdot2}=1/\omega_{11}=\tfrac{13}{6}$ ✓, and $B=-\omega_{11}^{-1}(\omega_{12},\omega_{13})=-\tfrac{13}{6}\left(-\tfrac4{13},\tfrac3{13}\right)=\left(\tfrac23,-\tfrac12\right)$ ✓. Also $|\Sigma_{22}|\,\Sigma_{11\cdot2}=6\cdot\tfrac{13}6=13=|\Sigma|$ ✓.
📘 Definition 7.3 — Mahalanobis distance

Let $\Sigma\gt0$. The Mahalanobis distance of a point $x$ from $\mu$ (relative to $\Sigma$) is $$\Delta(x)=\sqrt{(x-\mu)^{\mathsf T}\Sigma^{-1}(x-\mu)},$$ and the Mahalanobis distance between two populations with means $\mu_1,\mu_2$ and common covariance $\Sigma$ is $\Delta=\sqrt{(\mu_1-\mu_2)^{\mathsf T}\Sigma^{-1}(\mu_1-\mu_2)}$. For $p=1$ it is $|x-\mu|/\sigma$, the number of standard deviations.

📐 Theorem 7.8 — The Mahalanobis distance and the ellipsoids of $N_p(\mu,\Sigma)$

Let $X\sim N_p(\mu,\Sigma)$ with $\Sigma\gt0$, and let $\Sigma=\sum_{i=1}^p\lambda_ie_ie_i^{\mathsf T}$ be its spectral decomposition ($\lambda_i\gt0$, $e_i$ orthonormal).

  1. $D^2=(X-\mu)^{\mathsf T}\Sigma^{-1}(X-\mu)\sim\chi^2_p$.
  2. The density is constant on each ellipsoid $\{x:(x-\mu)^{\mathsf T}\Sigma^{-1}(x-\mu)=c\}$, whose axes point along $e_1,\dots,e_p$ with half-lengths $\sqrt{c\lambda_i}$; its volume is $|\Sigma|^{1/2}c^{p/2}$ times the volume of the unit ball.
  3. The solid ellipsoid $\mathcal E_\alpha=\{x:(x-\mu)^{\mathsf T}\Sigma^{-1}(x-\mu)\le\chi^2_{p,\alpha}\}$ has probability $1-\alpha$, and no other region with probability $1-\alpha$ has smaller volume.
  4. $\Delta$ is invariant under nonsingular affine changes of coordinates: if $y=Cx+d$ with $C$ nonsingular, then the distance of $y$ from $C\mu+d$ relative to $C\Sigma C^{\mathsf T}$ equals the distance of $x$ from $\mu$ relative to $\Sigma$.

Proof. (1) $Z=\Sigma^{-1/2}(X-\mu)\sim N_p(0,I)$ (Theorem 7.4), and $D^2=Z^{\mathsf T}Z=\sum Z_i^2$ is a sum of $p$ squares of independent $N(0,1)$ variables.

(2) The density depends on $x$ only through $(x-\mu)^{\mathsf T}\Sigma^{-1}(x-\mu)$ (Theorem 7.3). With $y_i=e_i^{\mathsf T}(x-\mu)$, the coordinates of $x-\mu$ in the orthonormal basis $e_i$, and $\Sigma^{-1}=\sum\lambda_i^{-1}e_ie_i^{\mathsf T}$, the equation becomes $\sum_iy_i^2/\lambda_i=c$: an ellipsoid with half-axes $\sqrt{c\lambda_i}$ along the $e_i$. Its volume is $\prod_i\sqrt{c\lambda_i}$ times the volume of the unit ball, and $\prod\lambda_i=|\Sigma|$.

(3) $P(X\in\mathcal E_\alpha)=P(D^2\le\chi^2_{p,\alpha})=1-\alpha$ by (1). Let $R$ be any region with $P(X\in R)=1-\alpha$, and let $k$ be the value of the density on the boundary of $\mathcal E_\alpha$, so $f\ge k$ on $\mathcal E_\alpha$ and $f\le k$ outside. Since $P(\mathcal E_\alpha)=P(R)$, also $P(\mathcal E_\alpha\setminus R)=P(R\setminus\mathcal E_\alpha)$. Then $$k\operatorname{vol}(\mathcal E_\alpha\setminus R)\le P(\mathcal E_\alpha\setminus R)=P(R\setminus\mathcal E_\alpha)\le k\operatorname{vol}(R\setminus\mathcal E_\alpha),$$ so $\operatorname{vol}(\mathcal E_\alpha)\le\operatorname{vol}(R)$ — the same argument as the Neyman–Pearson lemma (§5.1).

(4) With $y-C\mu-d=C(x-\mu)$, $$\begin{aligned}&(y-C\mu-d)^{\mathsf T}(C\Sigma C^{\mathsf T})^{-1}(y-C\mu-d)\\&\quad=(x-\mu)^{\mathsf T}C^{\mathsf T}C^{-\mathsf T}\Sigma^{-1}C^{-1}C(x-\mu)\\&\quad=(x-\mu)^{\mathsf T}\Sigma^{-1}(x-\mu).\ ∎\end{aligned}$$

Part (4) is why the Mahalanobis distance, and every statistic built from it (Hotelling's $T^2$, Fisher's discriminant), does not care about units: measuring height in centimetres instead of metres, or adding a constant, changes nothing. The Euclidean distance has neither property.

✏️ Example 5 — Which observation is more unusual?
Standardised measurements $(x_1,x_2)$ of two strongly related quantities follow $N_2(0,\Sigma)$ with $\Sigma=\begin{pmatrix}1&0.8\\0.8&1\end{pmatrix}$. Compare the points $a=(1.5,1.5)$ and $b=(1,-1)$ by Euclidean and by Mahalanobis distance, find $P(D^2\gt D^2(x))$ for each, and describe the $95\%$ ellipse.
Euclidean $\|a\|=1.5\sqrt2=2.12$ and $\|b\|=\sqrt2=1.41$: by Euclidean distance $a$ is the more extreme point.
Mahalanobis $\Sigma^{-1}=\frac{1}{0.36}\begin{pmatrix}1&-0.8\\-0.8&1\end{pmatrix}$, so $D^2(x)=\frac{x_1^2-1.6x_1x_2+x_2^2}{0.36}$. Then $$D^2(a)=\frac{2.25-3.6+2.25}{0.36}=2.5,\qquad D^2(b)=\frac{1+1.6+1}{0.36}=10.$$
Probabilities $\chi^2_2$ is exponential with mean $2$, so $P(\chi^2_2\gt c)=e^{-c/2}$: $e^{-1.25}=0.287$ for $a$ and $e^{-5}=0.0067$ for $b$. The point $b$, with its two coordinates of opposite sign, is far more unusual — the variables are supposed to move together.
Ellipse $\chi^2_{2,0.05}=5.991$. The eigenvalues of $\Sigma$ are $1.8$ on $e_1=(1,1)/\sqrt2$ and $0.2$ on $e_2=(1,-1)/\sqrt2$, so the half-axes are $\sqrt{5.991\times1.8}=3.28$ along the diagonal and $\sqrt{5.991\times0.2}=1.09$ across it. The area is $\pi\cdot5.991\cdot|\Sigma|^{1/2}=\pi(5.991)(0.6)=11.29$. The point $b$ lies outside it, $a$ well inside.

Quadratic Forms and Cochran's Theorem

Sums of squares are quadratic forms. In a normal sample, $\sum(X_i-\bar X)^2=X^{\mathsf T}(I-\tfrac1nJ)X$, and in the linear model of §6.1, $\operatorname{RSS}=Y^{\mathsf T}(I-H)Y$. When is such a form chi-square, and when are two of them independent?

📘 Definition 7.4 — Noncentral chi-square distribution

If $Y\sim N_r(m,I_r)$, the law of $Y^{\mathsf T}Y=\sum_{i=1}^rY_i^2$ is the noncentral chi-square distribution with $r$ degrees of freedom and noncentrality $\delta=m^{\mathsf T}m$, written $\chi^2_r(\delta)$; $\chi^2_r(0)=\chi^2_r$. It depends on $m$ only through $\delta$, and $$E\big[\chi^2_r(\delta)\big]=r+\delta,\qquad\operatorname{Var}\big[\chi^2_r(\delta)\big]=2r+4\delta.$$ (Some books use $\delta/2$ as the noncentrality parameter; we follow §6.1.)

If $m$ and $m'$ have the same length, there is an orthogonal $Q$ with $Qm=m'$, and $QY\sim N_r(m',I)$ has the same squared length as $Y$ (Theorem 7.4); so the law depends on $m$ only through $\|m\|^2=\delta$. Taking $m=(\sqrt\delta,0,\dots,0)$: $E[Y_1^2]=1+\delta$, and for $Y_1=Z+a$ with $Z\sim N(0,1)$, $E[Y_1^4]=a^4+6a^2+3$, so $\operatorname{Var}(Y_1^2)=a^4+6a^2+3-(1+a^2)^2=2+4a^2$. Adding the $r-1$ central terms, each of variance $2$, gives $2r+4\delta$.

📐 Theorem 7.9 — Moments of a quadratic form

Let $Z\sim N_n(\mu,I_n)$ and let $A$ be symmetric with eigenvalues $\lambda_1,\dots,\lambda_n$. Then

  1. $E[Z^{\mathsf T}AZ]=\operatorname{tr}A+\mu^{\mathsf T}A\mu$ and $\operatorname{Var}(Z^{\mathsf T}AZ)=2\operatorname{tr}(A^2)+4\mu^{\mathsf T}A^2\mu$;
  2. if $\mu=0$, then $Z^{\mathsf T}AZ$ has the law of $\sum_i\lambda_iW_i$ with $W_1,\dots,W_n$ iid $\chi^2_1$, and its mgf is $\prod_i(1-2t\lambda_i)^{-1/2}$ for $|t|$ small.

Proof. Write $A=Q\Lambda Q^{\mathsf T}$ with $Q$ orthogonal, and put $V=Q^{\mathsf T}Z\sim N_n(\nu,I)$, $\nu=Q^{\mathsf T}\mu$ (Theorem 7.4). Then $Z^{\mathsf T}AZ=V^{\mathsf T}\Lambda V=\sum_i\lambda_iV_i^2$ with independent $V_i^2\sim\chi^2_1(\nu_i^2)$. By Definition 7.4, $E=\sum\lambda_i(1+\nu_i^2)=\operatorname{tr}A+\nu^{\mathsf T}\Lambda\nu$ and $\operatorname{Var}=\sum\lambda_i^2(2+4\nu_i^2)=2\operatorname{tr}A^2+4\nu^{\mathsf T}\Lambda^2\nu$; finally $\nu^{\mathsf T}\Lambda\nu=\mu^{\mathsf T}A\mu$ and $\nu^{\mathsf T}\Lambda^2\nu=\mu^{\mathsf T}A^2\mu$. If $\mu=0$, $V_i^2\sim\chi^2_1$ with mgf $(1-2t)^{-1/2}$, and the mgf of $\lambda_iV_i^2$ is $(1-2t\lambda_i)^{-1/2}$; independence multiplies them. ∎

📐 Theorem 7.10 — When is a quadratic form chi-square?
  1. Let $Z\sim N_n(\mu,I_n)$ and $A$ symmetric. Then $Z^{\mathsf T}AZ$ has a (noncentral) chi-square law if and only if $A$ is idempotent ($A^2=A$). In that case $$Z^{\mathsf T}AZ\sim\chi^2_r(\mu^{\mathsf T}A\mu),\qquad r=\operatorname{rank}A=\operatorname{tr}A.$$
  2. Let $X\sim N_n(\mu,\Sigma)$ with $\Sigma\gt0$ and $A$ symmetric. Then $X^{\mathsf T}AX\sim\chi^2_r(\mu^{\mathsf T}A\mu)$ for some $r$ if and only if $A\Sigma A=A$ (equivalently, $A\Sigma$ is idempotent), and then $r=\operatorname{rank}A=\operatorname{tr}(A\Sigma)$.

Proof. (1) If. A symmetric idempotent matrix has eigenvalues $\lambda$ with $\lambda^2=\lambda$, so $\lambda\in\{0,1\}$; with $r$ unit eigenvalues and orthonormal eigenvectors $q_1,\dots,q_r$, $A=\sum_{i\le r}q_iq_i^{\mathsf T}$, $\operatorname{rank}A=r=\operatorname{tr}A$, and $Z^{\mathsf T}AZ=\sum_{i\le r}(q_i^{\mathsf T}Z)^2$. The variables $q_i^{\mathsf T}Z$ are jointly normal with covariance $q_i^{\mathsf T}q_j=\delta_{ij}$, hence independent $N(q_i^{\mathsf T}\mu,1)$ (Theorem 7.5), and the sum is $\chi^2_r(\delta)$ with $\delta=\sum(q_i^{\mathsf T}\mu)^2=\mu^{\mathsf T}A\mu$.

Only if (central case $\mu=0$; the noncentral case uses the noncentral mgf in the same way). Suppose $Z^{\mathsf T}AZ\sim\chi^2_r$. Comparing mgfs (Theorem 7.9), $\prod_i(1-2t\lambda_i)^{-1/2}=(1-2t)^{-r/2}$ for all $t$ near $0$. Squaring and inverting, $\prod_{\lambda_i\ne0}(1-2t\lambda_i)=(1-2t)^r$ for all $t$ in an interval. Both sides are polynomials in $t$, so they are identical, and comparing their roots, every nonzero $\lambda_i$ satisfies $1/(2\lambda_i)=1/2$, i.e. $\lambda_i=1$. So every eigenvalue is $0$ or $1$, and a symmetric matrix with such eigenvalues is idempotent.

(2) Put $Y=\Sigma^{-1/2}X\sim N_n(\Sigma^{-1/2}\mu,I)$. Then $X^{\mathsf T}AX=Y^{\mathsf T}MY$ with $M=\Sigma^{1/2}A\Sigma^{1/2}$ symmetric. By (1) it is chi-square iff $M^2=M$, i.e. $\Sigma^{1/2}A\Sigma A\Sigma^{1/2}=\Sigma^{1/2}A\Sigma^{1/2}$, i.e. (cancelling the nonsingular $\Sigma^{1/2}$ on both sides) $A\Sigma A=A$. The degrees of freedom are $\operatorname{tr}M=\operatorname{tr}(A\Sigma)$ and $\operatorname{rank}M=\operatorname{rank}A$, and the noncentrality is $(\Sigma^{-1/2}\mu)^{\mathsf T}M(\Sigma^{-1/2}\mu)=\mu^{\mathsf T}A\mu$. ∎

Two familiar cases: $A=\Sigma^{-1}$ satisfies $A\Sigma A=A$, which is Theorem 7.8(1) again; and for $X\sim N_n(\theta\mathbf 1,\sigma^2I)$, $C=I-\frac1nJ$ is idempotent of trace $n-1$ with $C\mathbf 1=0$, so $\sum(X_i-\bar X)^2/\sigma^2\sim\chi^2_{n-1}$ (the result of §4.1).

📐 Theorem 7.11 — Independence of linear and quadratic forms (Craig's theorem)

Let $X\sim N_n(\mu,\sigma^2I_n)$, let $A$, $B$ be symmetric $n\times n$ and $L$ a $k\times n$ matrix.

  1. If $LA=0$, then $LX$ and $X^{\mathsf T}AX$ are independent.
  2. $X^{\mathsf T}AX$ and $X^{\mathsf T}BX$ are independent if and only if $AB=0$.

For $X\sim N_n(\mu,\Sigma)$ the conditions become $L\Sigma A=0$ and $A\Sigma B=0$.

Proof. (1) Let $A=Q_1\Lambda_1Q_1^{\mathsf T}$, where the columns of $Q_1$ are orthonormal eigenvectors for the nonzero eigenvalues and $\Lambda_1$ is nonsingular. Then $X^{\mathsf T}AX=(Q_1^{\mathsf T}X)^{\mathsf T}\Lambda_1(Q_1^{\mathsf T}X)$ is a function of $Q_1^{\mathsf T}X$. From $AQ_1=Q_1\Lambda_1$ we get $LQ_1=LAQ_1\Lambda_1^{-1}=0$. The vectors $LX$ and $Q_1^{\mathsf T}X$ are jointly normal with $\operatorname{Cov}(LX,Q_1^{\mathsf T}X)=\sigma^2LQ_1=0$, so they are independent (Theorem 7.5), and so are functions of them.

(2) If. Write $B=Q_2\Lambda_2Q_2^{\mathsf T}$ similarly. From $AB=0$, multiplying by $Q_1^{\mathsf T}$ on the left and by $Q_2$ on the right, $\Lambda_1Q_1^{\mathsf T}Q_2\Lambda_2=0$, so $Q_1^{\mathsf T}Q_2=0$. Then $Q_1^{\mathsf T}X$ and $Q_2^{\mathsf T}X$ are jointly normal and uncorrelated, hence independent, and the two forms are functions of them. Only if (sketch). For $\mu=0$, independence makes the joint mgf factor near $(0,0)$: $$|I-2sA-2tB|^{-1/2}=|I-2sA|^{-1/2}\,|I-2tB|^{-1/2}.$$ Comparing the coefficients of $st$ after taking logarithms gives $\operatorname{tr}(AB)=0$, and an argument on the full polynomial identity (Craig's theorem, completed by later authors) gives $AB=0$. For idempotent $A$, $B$ the converse is easy: Exercise III.7 shows $\operatorname{Cov}(Z^{\mathsf T}AZ,Z^{\mathsf T}BZ)=2\operatorname{tr}(AB)=2\|AB\|_F^2$. The general-$\Sigma$ form follows by $X=\Sigma^{1/2}Y$. ∎

With $L=\frac1n\mathbf 1^{\mathsf T}$ and $A=I-\frac1nJ$ we have $LA=0$: so $\bar X$ and $\sum(X_i-\bar X)^2$ are independent — the fundamental theorem for normal samples (§4.1) in two lines. In the linear model, $H(I-H)=0$ gives the independence of the fitted values and $\operatorname{RSS}$ used in §6.1.

📐 Theorem 7.12 — Cochran's theorem

Let $A_1,\dots,A_k$ be symmetric $n\times n$ matrices with $A_1+\cdots+A_k=I_n$, and let $r_i=\operatorname{rank}A_i$. The following are equivalent:

  1. $r_1+\cdots+r_k=n$;
  2. every $A_i$ is idempotent;
  3. $A_iA_j=0$ for all $i\ne j$.

If they hold and $X\sim N_n(\mu,\sigma^2I_n)$, then the forms $Q_i=X^{\mathsf T}A_iX$ (which add up to $X^{\mathsf T}X$) are independent, and $Q_i/\sigma^2\sim\chi^2_{r_i}(\mu^{\mathsf T}A_i\mu/\sigma^2)$.

Proof. (1) ⇒ (2). Fix $i$. Since $I=A_i+(I-A_i)$ and $I-A_i=\sum_{j\ne i}A_j$, rank subadditivity gives $$n=\operatorname{rank}I\le\operatorname{rank}A_i+\operatorname{rank}(I-A_i)\le r_i+\sum_{j\ne i}r_j=n.$$ Hence $\operatorname{rank}(I-A_i)=n-r_i$, so the null space of $I-A_i$ — the eigenspace of $A_i$ for the eigenvalue $1$ — has dimension $r_i$. But $A_i$ has exactly $r_i$ nonzero eigenvalues counted with multiplicity, so all of them equal $1$. A symmetric matrix with eigenvalues in $\{0,1\}$ is idempotent.

(2) ⇒ (1). For idempotent $A_i$, $\operatorname{rank}A_i=\operatorname{tr}A_i$, so $\sum r_i=\operatorname{tr}\sum A_i=\operatorname{tr}I=n$.

(1) and (2) ⇒ (3). Fix $i\ne j$ and let $C=A_i+A_j$. Then $I=C+\sum_{l\ne i,j}A_l$, and the argument of the first step, applied to $C$ (whose rank is at most $r_i+r_j$), shows that all nonzero eigenvalues of $C$ equal $1$: $C^2=C$. Expanding, $A_i^2+A_iA_j+A_jA_i+A_j^2=A_i+A_j$, so $A_iA_j+A_jA_i=0$. Multiplying this on the left by $A_i$ gives $A_iA_j+A_iA_jA_i=0$; multiplying it on the right by $A_i$ gives $A_iA_jA_i+A_jA_i=0$. Subtracting, $A_iA_j=A_jA_i$, and with $A_iA_j+A_jA_i=0$ this gives $A_iA_j=0$.

(3) ⇒ (2). $A_i=A_iI=A_i\sum_jA_j=A_i^2$.

Distribution. Write each idempotent $A_i=P_iP_i^{\mathsf T}$ with $P_i$ an $n\times r_i$ matrix of orthonormal columns. From $A_iA_j=0$, $P_i^{\mathsf T}P_j=P_i^{\mathsf T}(P_iP_i^{\mathsf T})(P_jP_j^{\mathsf T})P_j$ $=P_i^{\mathsf T}A_iA_jP_j=0$. So the $n$ columns of $P=(P_1\ \cdots\ P_k)$ are orthonormal and $P$ is orthogonal. Then $V=P^{\mathsf T}X\sim N_n(P^{\mathsf T}\mu,\sigma^2I)$ has independent coordinates, and $Q_i=\|P_i^{\mathsf T}X\|^2$ is the sum of squares of the $i$-th block of $V$. The blocks are disjoint, so the $Q_i$ are independent, and $Q_i/\sigma^2\sim\chi^2_{r_i}(\|P_i^{\mathsf T}\mu\|^2/\sigma^2)$ with $\|P_i^{\mathsf T}\mu\|^2=\mu^{\mathsf T}A_i\mu$. ∎

This is the theorem quoted in §6.1, now proved. In practice condition (1) is the one checked: if the degrees of freedom of the pieces of a sum of squares add up to the total, the pieces are independent chi-square variables, and ratios of them divided by their degrees of freedom are $F$ variables.

✏️ Example 6 — Splitting $X_1^2+X_2^2+X_3^2$, and a form that is not chi-square
(a) Let $X\sim N_3(\theta\mathbf 1,\sigma^2I)$. Show that $$Q_1=\tfrac12(X_1-X_2)^2,\quad Q_2=\tfrac16(X_1+X_2-2X_3)^2,\quad Q_3=\tfrac13(X_1+X_2+X_3)^2$$ add up to $X^{\mathsf T}X$, and find their joint law. (b) Let $Z\sim N_2(0,I)$ and $Q=Z_1^2+Z_1Z_2+Z_2^2$. Is $Q$, or any multiple $cQ$, chi-square? Compare the mean and variance of $Q$ with those of $\chi^2_2$.
(a) Matrices $Q_i=(u_i^{\mathsf T}X)^2$ with $u_1=\tfrac1{\sqrt2}(1,-1,0)$, $u_2=\tfrac1{\sqrt6}(1,1,-2)$, $u_3=\tfrac1{\sqrt3}(1,1,1)$. These are orthonormal (check: $u_1^{\mathsf T}u_2=\tfrac{1-1}{\sqrt{12}}=0$, $u_2^{\mathsf T}u_3=\tfrac{1+1-2}{\sqrt{18}}=0$, $u_1^{\mathsf T}u_3=0$, unit lengths), so $A_i=u_iu_i^{\mathsf T}$ has rank $1$ and $A_1+A_2+A_3=UU^{\mathsf T}=I$ for the orthogonal $U=(u_1\ u_2\ u_3)$. Hence $\sum Q_i=X^{\mathsf T}X$.
(a) Cochran The ranks add to $1+1+1=3=n$, so $Q_1,Q_2,Q_3$ are independent and $Q_i/\sigma^2\sim\chi^2_1(\delta_i)$ with $\delta_i=(u_i^{\mathsf T}\theta\mathbf 1)^2/\sigma^2$: $\delta_1=\delta_2=0$ and $\delta_3=3\theta^2/\sigma^2$. Note $Q_3=3\bar X^2$ and $Q_1+Q_2=\sum(X_i-\bar X)^2$: we have recovered the independence of $\bar X$ and $S^2$ and $2S^2/\sigma^2\sim\chi^2_2$ — this is the Helmert transformation of §4.1.
(b) Not idempotent $Q=Z^{\mathsf T}AZ$ with $A=\begin{pmatrix}1&1/2\\1/2&1\end{pmatrix}$, eigenvalues $\tfrac32$ and $\tfrac12$. By Theorem 7.10, $cQ$ is chi-square iff the eigenvalues $\tfrac32c,\tfrac12c$ of $cA$ lie in $\{0,1\}$, impossible for $c\ne0$. So no multiple of $Q$ is chi-square: $Q\overset{d}{=}\tfrac32W_1+\tfrac12W_2$ with $W_i$ iid $\chi^2_1$.
(b) Moments $E[Q]=\operatorname{tr}A=2=E[\chi^2_2]$, but $\operatorname{Var}Q=2\operatorname{tr}A^2=2\left(\tfrac94+\tfrac14\right)=5\ne4=\operatorname{Var}\chi^2_2$. Matching the mean (or the trace) is not enough; Interactive 2 shows the mismatch in the histogram.
Interactive 2 — When Is a Quadratic Form Chi-Square?

Each tick simulates many values of a quadratic form $Q=X^{\mathsf T}AX$ in normal variables (seeded) and adds them to the histogram. The dashed purple curve is the $\chi^2_k$ density with $k=\operatorname{tr}(A\Sigma)$, the law you would guess from the mean; the green curve is the true density. They coincide exactly when $A\Sigma A=A$ (Theorem 7.10). In the noncentral case the purple curve is the central $\chi^2_3$, for comparison.

draws—
eigenvalues of AΣ—
mean: sample (theory)—
variance: sample (theory)—
AΣA = A ?—

The Wishart Distribution and the Sample Mean and Covariance

For a univariate normal sample, $\sum(X_i-\bar X)^2\sim\sigma^2\chi^2_{n-1}$, and $\sigma^2\chi^2_m$ is the law of $\sum_{i=1}^mZ_i^2$ with $Z_i$ iid $N(0,\sigma^2)$. Replacing each square $Z_i^2$ by the outer product $Z_iZ_i^{\mathsf T}$ gives the matrix version.

📘 Definition 7.5 — Wishart distribution

Let $Z_1,\dots,Z_m$ be iid $N_p(0,\Sigma)$ and let $\mathbf Z$ be the $m\times p$ matrix with rows $Z_i^{\mathsf T}$. The law of the $p\times p$ random matrix $$W=\sum_{i=1}^mZ_iZ_i^{\mathsf T}=\mathbf Z^{\mathsf T}\mathbf Z$$ is the Wishart distribution with $m$ degrees of freedom and scale matrix $\Sigma$, written $W_p(m,\Sigma)$. For $p=1$, $W_1(m,\sigma^2)$ is the law of $\sigma^2\chi^2_m$. Its entries are $W_{jk}=\sum_iZ_{ij}Z_{ik}$, the sums of squares and products of the columns of $\mathbf Z$.

📐 Theorem 7.13 — Properties of the Wishart distribution

Let $W\sim W_p(m,\Sigma)$.

  1. (Moments) $E[W]=m\Sigma$ and $\operatorname{Var}(W_{jk})=m(\sigma_{jk}^2+\sigma_{jj}\sigma_{kk})$.
  2. (Additivity) If $W_1\sim W_p(m_1,\Sigma)$ and $W_2\sim W_p(m_2,\Sigma)$ are independent, then $W_1+W_2\sim W_p(m_1+m_2,\Sigma)$.
  3. (Linear maps) For a $q\times p$ matrix $C$, $CWC^{\mathsf T}\sim W_q(m,C\Sigma C^{\mathsf T})$. In particular every diagonal block $W_{11}\sim W_q(m,\Sigma_{11})$, $W_{jj}/\sigma_{jj}\sim\chi^2_m$, and $a^{\mathsf T}Wa/a^{\mathsf T}\Sigma a\sim\chi^2_m$ for every fixed $a$ with $a^{\mathsf T}\Sigma a\gt0$; also $\operatorname{tr}(\Sigma^{-1}W)\sim\chi^2_{mp}$ when $\Sigma\gt0$.
  4. (Random directions) If $a$ is a random vector independent of $W$ with $a^{\mathsf T}\Sigma a\gt0$ almost surely, then $a^{\mathsf T}Wa/a^{\mathsf T}\Sigma a\sim\chi^2_m$ and is independent of $a$.
  5. (Inverse) Let $\Sigma\gt0$. If $m\lt p$, $W$ is singular; if $m\ge p$, $W\gt0$ with probability $1$, and for every fixed $a\ne0$, $$\frac{a^{\mathsf T}\Sigma^{-1}a}{a^{\mathsf T}W^{-1}a}\sim\chi^2_{m-p+1};$$ as in (4), the same holds for a random $a$ independent of $W$, and the ratio is then independent of $a$.
  6. (Generalised variance) For $m\ge p$ and $\Sigma\gt0$, $|W|/|\Sigma|$ has the law of a product $\prod_{j=1}^pU_j$ of independent $U_j\sim\chi^2_{m-j+1}$; so $E|W|=m(m-1)\cdots(m-p+1)|\Sigma|$.
  7. (Density, statement) For $m\ge p$ and $\Sigma\gt0$, $W$ has the density $$f(w)=\frac{|w|^{(m-p-1)/2}\exp\left(-\tfrac12\operatorname{tr}(\Sigma^{-1}w)\right)}{2^{mp/2}|\Sigma|^{m/2}\Gamma_p(m/2)}$$ on positive definite $w$, where $\Gamma_p(a)=\pi^{p(p-1)/4}\prod_{j=1}^p\Gamma\left(a-\tfrac{j-1}2\right)$.

Proof. (1) $E[Z_iZ_i^{\mathsf T}]=\Sigma$ for each of the $m$ terms. For the variance, $W_{jk}$ is a sum of $m$ iid copies of $Z_jZ_k$ (dropping the index $i$), and for jointly normal centred variables $E[Z_j^2Z_k^2]=\sigma_{jj}\sigma_{kk}+2\sigma_{jk}^2$ (expand $Z_k=\beta Z_j+U$ with $U$ independent of $Z_j$, or use Isserlis' formula); so $\operatorname{Var}(Z_jZ_k)=\sigma_{jj}\sigma_{kk}+\sigma_{jk}^2$.

(2) Stack the two independent samples: $W_1+W_2$ is the sum of $m_1+m_2$ iid outer products. (3) $CWC^{\mathsf T}=\sum_i(CZ_i)(CZ_i)^{\mathsf T}$ and the $CZ_i$ are iid $N_q(0,C\Sigma C^{\mathsf T})$. With $C=a^{\mathsf T}$, $a^{\mathsf T}Wa=\sum_i(a^{\mathsf T}Z_i)^2$ with $a^{\mathsf T}Z_i$ iid $N(0,a^{\mathsf T}\Sigma a)$, which is $a^{\mathsf T}\Sigma a$ times a $\chi^2_m$ variable. For the trace, $\operatorname{tr}(\Sigma^{-1}W)=\sum_iZ_i^{\mathsf T}\Sigma^{-1}Z_i$, a sum of $m$ independent $\chi^2_p$ variables (Theorem 7.8). (4) Conditionally on $a$, by independence and (3), the ratio is $\chi^2_m$. This conditional law does not depend on $a$, so it is also the unconditional law, and the ratio is independent of $a$.

(5) $\operatorname{rank}W=\operatorname{rank}\mathbf Z\le m$, so $W$ is singular when $m\lt p$. Reduce to $\Sigma=I$ and $a=e_1$: choose $C$ nonsingular with $C\Sigma C^{\mathsf T}=I$ (so $\Sigma^{-1}=C^{\mathsf T}C$), let $b=Ca$, choose an orthogonal $Q$ with $Qb=\|b\|e_1$, and put $W^*=QCWC^{\mathsf T}Q^{\mathsf T}\sim W_p(m,I)$ by (3). Since $W^{-1}=C^{\mathsf T}Q^{\mathsf T}W^{*-1}QC$, we get $$a^{\mathsf T}W^{-1}a=(QCa)^{\mathsf T}W^{*-1}(QCa)=\|b\|^2(W^{*-1})_{11},$$ while $a^{\mathsf T}\Sigma^{-1}a=\|Ca\|^2=\|b\|^2$; so the ratio is $1/(W^{*-1})_{11}$. By Theorem 7.7(2) applied to $W^*$, $1/(W^{*-1})_{11}=W^*_{11}-W^*_{12}(W^*_{22})^{-1}W^*_{21}$. Write $\mathbf Z=(z_1\ \ \mathbf Z_2)$ for the $m\times p$ data matrix of $W^*$ (columns iid $N_m(0,I)$ because $\Sigma=I$). Then this Schur complement equals $z_1^{\mathsf T}(I-H_2)z_1$, where $H_2$ is the orthogonal projection onto the column space of $\mathbf Z_2$ — it is the residual sum of squares of the least-squares regression of $z_1$ on the other $p-1$ columns (§6.1). For $m\ge p$, $\mathbf Z_2$ has rank $p-1$ almost surely, so $I-H_2$ is idempotent of rank $m-p+1$; conditionally on $\mathbf Z_2$, $z_1\sim N_m(0,I)$ is independent of it, so $z_1^{\mathsf T}(I-H_2)z_1\sim\chi^2_{m-p+1}$ (Theorem 7.10). The conditional law is free of $\mathbf Z_2$, so it is the unconditional law. Finally $W^*=\mathbf Z^{\mathsf T}\mathbf Z\gt0$ as soon as $\mathbf Z$ has rank $p$, and this happens with probability $1$: the rows have a density, so each of the first $p$ rows falls into the span of the earlier ones (a proper subspace, of Lebesgue measure $0$) with probability $0$. The random-$a$ statement follows as in (4).

(6) (Sketch, with $\Sigma=I$ after the reduction in (5), which multiplies $|W|$ by $|C|^2=1/|\Sigma|$.) Gram–Schmidt on the columns $z_1,\dots,z_p$ of $\mathbf Z$ gives $|\mathbf Z^{\mathsf T}\mathbf Z|=\prod_{j=1}^pR_j$, where $R_j$ is the residual sum of squares of $z_j$ regressed on $z_1,\dots,z_{j-1}$. By the argument of (5), conditionally on $z_1,\dots,z_{j-1}$, $R_j\sim\chi^2_{m-j+1}$; since these conditional laws do not depend on the conditioning columns, $R_1,\dots,R_p$ are independent. The expectation is $\prod_j(m-j+1)$. (This product representation is Bartlett's decomposition.) (7) is obtained from the same decomposition by a change of variables; we omit the calculation. ∎

📐 Theorem 7.14 — The sample mean vector and covariance matrix of a normal sample

Let $X_1,\dots,X_n$ be iid $N_p(\mu,\Sigma)$ with $n\ge2$, and let $$\bar X=\frac1n\sum_{i=1}^nX_i,\qquad A=\sum_{i=1}^n(X_i-\bar X)(X_i-\bar X)^{\mathsf T},\qquad S=\frac{A}{n-1}.$$

  1. $\bar X\sim N_p(\mu,\Sigma/n)$;
  2. $A\sim W_p(n-1,\Sigma)$; in particular $E[S]=\Sigma$;
  3. $\bar X$ and $A$ (hence $\bar X$ and $S$) are independent;
  4. if $\Sigma\gt0$ and $n\gt p$, then $S\gt0$ almost surely, and the maximum likelihood estimators of $\mu$ and $\Sigma$ are $\hat\mu=\bar X$ and $\hat\Sigma=A/n$.

Proof. Let $\mathbf X$ be the $n\times p$ data matrix with rows $X_i^{\mathsf T}$, and let $H$ be an $n\times n$ orthogonal matrix whose first row is $\mathbf 1^{\mathsf T}/\sqrt n$ (for instance the Helmert matrix of §4.1). Put $\mathbf Y=H\mathbf X$, with rows $Y_j^{\mathsf T}$, so $Y_j=\sum_ih_{ji}X_i$.

The rows of $\mathbf Y$ are independent normal vectors. All entries of $\mathbf Y$ are linear combinations of the entries of the independent normal vectors $X_i$, so they are jointly normal. Their cross-covariances are $$\operatorname{Cov}(Y_j,Y_k)=\sum_ih_{ji}h_{ki}\Sigma=(HH^{\mathsf T})_{jk}\Sigma=\delta_{jk}\Sigma,$$ so by Theorem 7.5 the rows $Y_1,\dots,Y_n$ are independent, each with covariance $\Sigma$. Their means are $E[Y_j]=\big(\sum_ih_{ji}\big)\mu$; for $j=1$ this is $\sqrt n\mu$, and for $j\ge2$ it is $0$, because row $j$ of $H$ is orthogonal to row $1$, i.e. its entries add up to $0$.

Identification. $Y_1=\frac1{\sqrt n}\sum X_i=\sqrt n\bar X$, which gives (1). Since $H^{\mathsf T}H=I$, $\sum_jY_jY_j^{\mathsf T}=\mathbf Y^{\mathsf T}\mathbf Y=\mathbf X^{\mathsf T}\mathbf X=\sum_iX_iX_i^{\mathsf T}$. Expanding $A=\sum X_iX_i^{\mathsf T}-n\bar X\bar X^{\mathsf T}$ and using $n\bar X\bar X^{\mathsf T}=Y_1Y_1^{\mathsf T}$, $$A=\sum_{j=2}^nY_jY_j^{\mathsf T},$$ a sum of $n-1$ outer products of iid $N_p(0,\Sigma)$ vectors: $A\sim W_p(n-1,\Sigma)$, which is (2). $A$ is a function of $Y_2,\dots,Y_n$ and $\bar X$ of $Y_1$, so they are independent: (3).

(4) Positive definiteness is Theorem 7.13(5) with $m=n-1\ge p$. For the likelihood, using the identity $$\textstyle\sum_i(x_i-\mu)^{\mathsf T}\Sigma^{-1}(x_i-\mu)=\operatorname{tr}(\Sigma^{-1}A)+n(\bar x-\mu)^{\mathsf T}\Sigma^{-1}(\bar x-\mu)$$ (expand about $\bar x$; the cross terms vanish), $$\log L(\mu,\Sigma)=\text{const}-\frac n2\log|\Sigma|-\frac12\operatorname{tr}(\Sigma^{-1}A)-\frac n2(\bar x-\mu)^{\mathsf T}\Sigma^{-1}(\bar x-\mu).$$ For each $\Sigma$ the last term is maximised (made $0$) by $\mu=\bar x$. Next let $B=A^{1/2}\Sigma^{-1}A^{1/2}$, with eigenvalues $\eta_1,\dots,\eta_p\gt0$. Then $\operatorname{tr}(\Sigma^{-1}A)=\operatorname{tr}B=\sum\eta_i$ and $\log|\Sigma|=\log|A|-\log|B|$ $=\log|A|-\sum\log\eta_i$, so $$-\frac n2\log|\Sigma|-\frac12\operatorname{tr}(\Sigma^{-1}A)=-\frac n2\log|A|+\sum_{i=1}^p\left(\frac n2\log\eta_i-\frac{\eta_i}2\right).$$ Each term is maximised at $\eta_i=n$, so the maximum is attained when $B=nI$, i.e. $\Sigma=A/n$. ∎

✏️ Example 7 — Sample statistics, and four statements about Wishart matrices
(a) Five bivariate observations are $(2,5),(4,7),(3,4),(6,9),(5,10)$. Compute $\bar x$, $A$, $S$, the sample correlation and the MLE of $\Sigma$. (b) Assuming the observations are iid $N_2(\mu,\Sigma)$ with $\Sigma=\begin{pmatrix}2&2\\2&5\end{pmatrix}$, give the laws of $A_{11}$ and of $A_{11}-2A_{12}+A_{22}$, and find $\operatorname{Var}(S_{12})$. (c) Prove or disprove: (i) if $W\sim W_2(m,I)$ then $W_{12}$ is a multiple of a chi-square variable; (ii) if $W\sim W_2(m,I)$ then $W_{11}$ and $W_{22}$ are independent; (iii) if $W_1\sim W_2(3,I)$ and $W_2\sim W_2(4,2I)$ are independent, then $W_1+W_2$ is Wishart; (iv) if $W\sim W_3(2,\Sigma)$ then $|W|=0$.
(a) $\bar x=(4,7)$; the deviations are $(-2,-2),(0,0),(-1,-3),(2,2),(1,3)$. So $A_{11}=4+0+1+4+1=10$, $A_{22}=4+0+9+4+9=26$, $A_{12}=4+0+3+4+3=14$, and $$S=\frac A4=\begin{pmatrix}2.5&3.5\\3.5&6.5\end{pmatrix},\qquad r=\frac{14}{\sqrt{10\cdot26}}=0.868,\qquad\hat\Sigma=\frac A5=\begin{pmatrix}2&2.8\\2.8&5.2\end{pmatrix}.$$ Also $|S|=16.25-12.25=4$, the sample generalised variance.
(b) $A\sim W_2(4,\Sigma)$ (Theorem 7.14). By Theorem 7.13(3): $A_{11}/2\sim\chi^2_4$; with $a=(1,-1)$, $a^{\mathsf T}\Sigma a=2-4+5=3$, so $(A_{11}-2A_{12}+A_{22})/3\sim\chi^2_4$. And $\operatorname{Var}(A_{12})=4(\sigma_{12}^2+\sigma_{11}\sigma_{22})=4(4+10)=56$, so $\operatorname{Var}(S_{12})=56/16=3.5$.
(c)(i) False $W_{12}=\sum_iZ_{i1}Z_{i2}$ has mean $m\sigma_{12}=0$ and is symmetric about $0$ (change the sign of the second column), so it takes both signs with positive probability; no multiple $c\chi^2_k$ with $c\ne0$ does that.
(c)(ii) True With $\Sigma=I$ the two columns of $\mathbf Z$ are independent ($\operatorname{Cov}(Z_{i1},Z_{i2})=0$ for jointly normal entries), and $W_{11}=\|z_1\|^2$, $W_{22}=\|z_2\|^2$ are functions of different columns. (With $\sigma_{12}\ne0$ they are dependent: $\operatorname{Cov}(W_{11},W_{22})=2m\sigma_{12}^2$.)
(c)(iii) False If $W_1+W_2\sim W_2(7,\Sigma^*)$, then matching means, $7\Sigma^*=3I+8I$, so $\Sigma^*=\tfrac{11}7I$ and Theorem 7.13(1) would give $\operatorname{Var}(W_{11})=2\cdot7\cdot\left(\tfrac{11}7\right)^2=34.57$. But by independence $\operatorname{Var}(W_{11})=2\cdot3\cdot1^2+2\cdot4\cdot2^2=38$. Additivity needs a common scale matrix.
(c)(iv) True $W=\mathbf Z^{\mathsf T}\mathbf Z$ with $\mathbf Z$ of size $2\times3$, so $\operatorname{rank}W\le2\lt3$ (Theorem 7.13(5)).
⚠️ Counterexamples and pitfalls
  1. Normal marginals do not make a normal vector. $X\sim N(0,1)$ and $Y=SX$ with an independent random sign $S$ (Example 2); or the density $2\phi(x)\phi(y)$ on $\{xy\gt0\}$ (Exercise I.7). Both have $N(0,1)$ marginals and are not bivariate normal.
  2. Uncorrelated is not independent unless the variables are jointly normal. In Example 2, $\operatorname{Cov}(X,Y)=0$ but $|X|=|Y|$.
  3. Pairwise is enough — but only for jointly normal vectors. For $X\sim N_p$, pairwise independent components are mutually independent; for general random vectors this fails (three events from two fair coins, §1.2).
  4. A singular $\Sigma$ is allowed. $(X,2X+1)$ with $X\sim N(0,1)$ is bivariate normal (every $aX+b(2X+1)$ is normal) but has no density: it lives on a line.
  5. Entries in $[-1,1]$ do not make a correlation matrix. The $3\times3$ matrix with off-diagonal entries $-0.6$ has $\operatorname{Var}(X_1+X_2+X_3)=-0.6$; equicorrelation needs $\rho\ge-1/(p-1)$.
  6. The conditional variance is constant only for the normal law. For non-normal vectors $\operatorname{Var}(X_1\mid X_2=x_2)$ usually depends on $x_2$, and $E[X_1\mid X_2]$ need not be linear.
  7. Marginal independence is not conditional independence. In Exercise II.3, $X_1$ and $X_3$ are independent, but given $X_2$ they are negatively correlated; conversely variables can be dependent but conditionally independent (§7.2).
  8. Right mean, wrong law. $Z_1^2+Z_1Z_2+Z_2^2$ has mean $2$ but is not $\chi^2_2$ (variance $5$, Example 6). Chi-square needs an idempotent matrix; the trace only gives the degrees of freedom after idempotency is checked.
  9. With correlated variables, check $A\Sigma A=A$, not $A^2=A$. If $X\sim N_2(0,\Sigma)$ with $\rho\ne0$, then $X_1^2+X_2^2$ is not chi-square although $A=I$ is idempotent; $X^{\mathsf T}\Sigma^{-1}X$ is.
  10. Independence of forms needs $AB=0$ (or $A\Sigma B=0$), not $AB=BA$ and not merely zero correlation of the variables involved. $Z_1^2$ and $(Z_1+Z_2)^2$ are dependent.
  11. The sum of two forms is not automatically chi-square. If $P$ is idempotent, $Z^{\mathsf T}PZ+Z^{\mathsf T}PZ=2Z^{\mathsf T}PZ$ is $2\chi^2_r$, not $\chi^2_{2r}$.
  12. Wishart pitfalls. The degrees of freedom of $A=\sum(X_i-\bar X)(X_i-\bar X)^{\mathsf T}$ are $n-1$, not $n$; $W_p(m,\Sigma)$ is singular when $m\lt p$ (so $S$ is singular when $n\le p$); off-diagonal entries are not chi-square; and sums of independent Wisharts are Wishart only with a common $\Sigma$.
  13. Independence of $\bar X$ and $S$ characterises normality. For non-normal samples it fails: with $n=2$ Bernoulli$(\tfrac12)$ observations, $S^2=\tfrac12$ when $\bar X=\tfrac12$ and $S^2=0$ otherwise.

Exercises

I
Mean vectors, covariance matrices and the definition.
  • 1 $X$ has mean $\mu=(2,-1,0)^{\mathsf T}$ and covariance $\Sigma=\begin{pmatrix}3&1&0\\1&2&-1\\0&-1&4\end{pmatrix}$. Check that $\Sigma\gt0$ and find $E$ and $\operatorname{Var}$ of $2X_1-X_2+3X_3$, and $\operatorname{Cov}(X_1+X_3,\ X_2-X_3)$.
  • 2 Prove that the equicorrelation matrix $(1-\rho)I_p+\rho J$ has eigenvalues $1-\rho$ (multiplicity $p-1$) and $1+(p-1)\rho$, and find its determinant and its inverse.
  • 3 $X$ has covariance $\Sigma=\begin{pmatrix}1&1&2\\1&2&3\\2&3&5\end{pmatrix}$. Show that $\Sigma$ is singular and find a linear combination of $X_1,X_2,X_3$ that is constant with probability $1$.
  • 4 Show that $E\|X-a\|^2$ is minimised over $a\in\mathbb R^p$ at $a=\mu$, with minimum $\operatorname{tr}\Sigma$.
  • 5 For $X\sim N_2(\mu,\Sigma)$ with $\mu=(0,1)^{\mathsf T}$ and $\Sigma=\begin{pmatrix}1&0.5\\0.5&2\end{pmatrix}$, compute $E\big[e^{X_1+X_2}\big]$.
  • 6 Let $Z\sim N_p(0,I)$ and $Q$ be orthogonal. Show $QZ\sim N_p(0,I)$; deduce that $(Z_1+Z_2)/\sqrt2$ and $(Z_1-Z_2)/\sqrt2$ are iid $N(0,1)$.
  • 7 $(X,Y)$ has density $2\phi(x)\phi(y)$ on $\{xy\gt0\}$ and $0$ elsewhere. Show that $X$ and $Y$ are $N(0,1)$, compute $\operatorname{Corr}(X,Y)$, and show that $(X,Y)$ is not bivariate normal.
1 Minors $3$, $5$, $|\Sigma|=3(8-1)-1(4-0)+0=17$: all positive. $a=(2,-1,3)$: $E=4+1+0=5$; $\Sigma a=(5,-3,13)$, $\operatorname{Var}=10+3+39=52$. $u=(1,0,1)$, $v=(0,1,-1)$: $\Sigma v=(1,3,-5)$, $\operatorname{Cov}=u^{\mathsf T}\Sigma v=1-5=-4$.
2 As in Example 1: $J\mathbf 1=p\mathbf 1$ and $Jv=0$ on $\mathbf 1^\perp$. The determinant is the product $(1-\rho)^{p-1}(1+(p-1)\rho)$. Try $E_\rho^{-1}=aI+bJ$: $(cI+\rho J)(aI+bJ)=caI+(cb+\rho a+\rho pb)J$ with $c=1-\rho$; setting $ca=1$ and $cb+\rho a+p\rho b=0$ gives $a=\frac1{1-\rho}$, $b=-\frac{\rho}{(1-\rho)(1+(p-1)\rho)}$ (valid when $\rho\ne1,\ -1/(p-1)$).
3 Column $3$ = column $1$ + column $2$, so $\Sigma(1,1,-1)^{\mathsf T}=0$ and $|\Sigma|=0$. By Theorem 7.1(3), $\operatorname{Var}(X_1+X_2-X_3)=0$: $X_1+X_2-X_3=\mu_1+\mu_2-\mu_3$ almost surely. (The eigenvalues are $0$, $0.394$, $7.606$, so $X$ lives on a plane.)
4 $E\|X-a\|^2=E\|X-\mu\|^2+2E[(X-\mu)^{\mathsf T}](\mu-a)+\|\mu-a\|^2=\operatorname{tr}\Sigma+\|\mu-a\|^2$ (Theorem 7.1(4) with $A=I$; the middle term is $0$). Minimised at $a=\mu$.
5 $M(t)=\exp(t^{\mathsf T}\mu+\tfrac12t^{\mathsf T}\Sigma t)$ at $t=(1,1)$: $t^{\mathsf T}\mu=1$, $t^{\mathsf T}\Sigma t=1+2\cdot0.5+2=4$. So $E[e^{X_1+X_2}]=e^{1+2}=e^3=20.09$.
6 Theorem 7.4: $QZ\sim N_p(0,QQ^{\mathsf T})=N_p(0,I)$. The matrix $\frac1{\sqrt2}\begin{pmatrix}1&1\\1&-1\end{pmatrix}$ is orthogonal; the two new coordinates are jointly normal with identity covariance, hence iid $N(0,1)$ (Theorem 7.5).
7 For $x\gt0$: $f_X(x)=\int_0^\infty2\phi(x)\phi(y)\,dy=\phi(x)$, and similarly for $x\lt0$. $E[XY]=2\cdot2\int_0^\infty\!\!\int_0^\infty xy\,\phi(x)\phi(y)\,dx\,dy=4\left(\tfrac1{\sqrt{2\pi}}\right)^2=\tfrac2\pi$, using $\int_0^\infty x\phi(x)dx=\tfrac1{\sqrt{2\pi}}$. So $\rho=2/\pi=0.637$. A nondegenerate bivariate normal law ($|\rho|\lt1$) has a positive density everywhere, but this one vanishes on two quadrants.
II
Marginals, conditionals and the Mahalanobis distance. In parts 1–3, $X\sim N_3(\mu,\Sigma)$ with $\mu=(0,1,2)^{\mathsf T}$ and $\Sigma=\begin{pmatrix}2&1&0\\1&2&1\\0&1&2\end{pmatrix}$.
  • 1 Find the conditional law of $X_1$ given $X_2=2$.
  • 2 Find the conditional law of $X_1$ given $(X_2,X_3)=(2,2)$. Why does the value of $X_3$ matter, although $X_1$ and $X_3$ are uncorrelated?
  • 3 Are $X_1$ and $X_3$ independent? Are they conditionally independent given $X_2$?
  • 4 Let $X\sim N_2(\mu,\Sigma)$. Show that $X_1+X_2$ and $X_1-X_2$ are independent if and only if $\sigma_{11}=\sigma_{22}$.
  • 5 For $\Sigma=\begin{pmatrix}4&-2\\-2&2\end{pmatrix}$ and $\mu=0$, find the Mahalanobis distance of $x=(2,0)$ and $P(D^2\gt D^2(x))$.
  • 6 Find the directions and half-lengths of the axes of the $90\%$ probability ellipse of $N_2\left(0,\begin{pmatrix}5&2\\2&2\end{pmatrix}\right)$.
  • 7 For $X\sim N_p(\mu,\Sigma)$, $\Sigma\gt0$, show $E[D^2]=p$ and $\operatorname{Var}(D^2)=2p$, and that $P(D^2\le p)\to\tfrac12$ as $p\to\infty$. What does this say about where high-dimensional normal points lie?
  • 8 Derive the conditional law of $Y$ given $X=x$ for the bivariate normal law of §1.4 from Theorem 7.6.
1 Mean $0+\frac{\sigma_{12}}{\sigma_{22}}(2-1)=\tfrac12$, variance $2-\tfrac{1}{2}=\tfrac32$: $N\left(\tfrac12,\tfrac32\right)$.
2 $\Sigma_{12}=(1,0)$, $\Sigma_{22}^{-1}=\tfrac13\begin{pmatrix}2&-1\\-1&2\end{pmatrix}$, $B=\left(\tfrac23,-\tfrac13\right)$. Mean $0+\tfrac23(2-1)-\tfrac13(2-2)=\tfrac23$; variance $2-B\Sigma_{21}=2-\tfrac23=\tfrac43$. $X_3$ matters because $X_2$ is related to both: once $X_2$ is known, a large $X_3$ "explains away" part of the value of $X_2$, leaving less of it to be attributed to $X_1$ — hence the negative coefficient $-\tfrac13$ of $X_3$.
3 Yes: $\sigma_{13}=0$ and they are jointly normal (Theorem 7.5). No: the conditional covariance of $(X_1,X_3)$ given $X_2$ is $\sigma_{13}-\sigma_{12}\sigma_{23}/\sigma_{22}=0-\tfrac12=-\tfrac12\ne0$ (conditional variances $\tfrac32$ each, conditional correlation $-\tfrac13$).
4 They are jointly normal with $\operatorname{Cov}(X_1+X_2,X_1-X_2)=\sigma_{11}-\sigma_{12}+\sigma_{21}-\sigma_{22}=\sigma_{11}-\sigma_{22}$. Apply Theorem 7.5.
5 $|\Sigma|=8-4=4$, $\Sigma^{-1}=\tfrac14\begin{pmatrix}2&2\\2&4\end{pmatrix}$, $D^2=\tfrac14(2\cdot4)=2$, $D=\sqrt2$. $P(\chi^2_2\gt2)=e^{-1}=0.368$.
6 Trace $7$, determinant $6$: eigenvalues $6$ and $1$, with eigenvectors $(2,1)/\sqrt5$ and $(1,-2)/\sqrt5$. With $\chi^2_{2,0.10}=4.605$ the half-lengths are $\sqrt{4.605\cdot6}=5.26$ and $\sqrt{4.605}=2.15$.
7 $D^2\sim\chi^2_p$, with mean $p$ and variance $2p$. By the CLT, $(D^2-p)/\sqrt{2p}\Rightarrow N(0,1)$, so $P(D^2\le p)\to\Phi(0)=\tfrac12$ (numerically $0.632$, $0.560$, $0.519$ for $p=2,10,100$). Since $D/\sqrt p\to1$ in probability, most of the mass of $N_p(0,I)$ lies near the sphere of radius $\sqrt p$ — not near the mode at $0$.
8 With $X_1=Y$, $X_2=X$: $\Sigma_{12}\Sigma_{22}^{-1}=\rho\sigma_1\sigma_2/\sigma_1^2=\rho\sigma_2/\sigma_1$ and $\Sigma_{11\cdot2}=\sigma_2^2-\rho^2\sigma_1^2\sigma_2^2/\sigma_1^2=\sigma_2^2(1-\rho^2)$, so $Y\mid X=x\sim N\left(\mu_2+\rho\tfrac{\sigma_2}{\sigma_1}(x-\mu_1),\ \sigma_2^2(1-\rho^2)\right)$, as in §1.4.
III
Quadratic forms and Cochran's theorem. Here $Z\sim N_n(0,I)$ unless stated otherwise.
  • 1 For $n=3$, decide which forms are chi-square and give the degrees of freedom: (a) $\tfrac13(Z_1+Z_2+Z_3)^2$; (b) $\tfrac23(Z_1^2+Z_2^2+Z_3^2-Z_1Z_2-Z_1Z_3-Z_2Z_3)$; (c) $Z_1^2+2Z_1Z_2+Z_2^2+Z_3^2$; (d) $\tfrac12(Z_1^2+Z_2^2)+Z_1Z_2$.
  • 2 $X\sim N_2(0,\Sigma)$ with $\Sigma=\begin{pmatrix}2&1\\1&3\end{pmatrix}$. Find $c$ such that $c(X_1-X_2)^2\sim\chi^2_1$.
  • 3 $Z\sim N_2(\mu,I)$ with $\mu=(1,1)^{\mathsf T}$ and $Q=Z_1^2+Z_1Z_2+Z_2^2$. Find $E[Q]$ and $\operatorname{Var}(Q)$.
  • 4 $X\sim N_n(\theta\mathbf 1,\sigma^2I)$. Use Theorems 7.10 and 7.11 to show that $\sum(X_i-\bar X)^2$ and $\bar X$ are independent, and find the law of $n\bar X^2/\sigma^2$.
  • 5 For $n=4$ let $Q_1=\tfrac14(Z_1+Z_2+Z_3+Z_4)^2$, $Q_2=\tfrac12(Z_1-Z_2)^2$, $Q_3=\tfrac12(Z_3-Z_4)^2$, $Q_4=\tfrac14(Z_1+Z_2-Z_3-Z_4)^2$. Show that Cochran's theorem applies, and find the law of $\dfrac{2Q_4}{Q_2+Q_3}$.
  • 6 Prove that a symmetric idempotent $A$ satisfies $\operatorname{tr}A=\operatorname{rank}A$, and that $I-A$ is symmetric idempotent of rank $n-\operatorname{rank}A$.
  • 7 Show that $\operatorname{Cov}(Z^{\mathsf T}AZ,Z^{\mathsf T}BZ)=2\operatorname{tr}(AB)$ for symmetric $A,B$. Deduce that for symmetric idempotent $P,Q$ the forms $Z^{\mathsf T}PZ$ and $Z^{\mathsf T}QZ$ are independent as soon as they are uncorrelated.
  • 8 $Z\sim N_3(\mu,I)$ with $\mu=(1,2,2)^{\mathsf T}$. Give the law of $Z^{\mathsf T}Z$ and its mean and variance.
1 (a) $A=\tfrac13J$, idempotent, trace $1$: $\chi^2_1$. (b) It equals $\sum(Z_i-\bar Z)^2$ (expand $\sum Z_i^2-\tfrac13(\sum Z_i)^2$), $A=I-\tfrac13J$, idempotent of trace $2$: $\chi^2_2$. (c) $A=\begin{pmatrix}1&1&0\\1&1&0\\0&0&1\end{pmatrix}$ has eigenvalues $2,1,0$: not chi-square (it is $2\chi^2_1+\chi^2_1$ with independent parts). (d) $=\tfrac12(Z_1+Z_2)^2$, $A=\tfrac12\begin{pmatrix}1&1\\1&1\end{pmatrix}$ (padded with zeros), idempotent of trace $1$: $\chi^2_1$.
2 $X_1-X_2\sim N(0,\ 2+3-2)=N(0,3)$, so $c=\tfrac13$. (In Theorem 7.10(2): $A=c\,aa^{\mathsf T}$ with $a=(1,-1)$, and $A\Sigma A=c^2(a^{\mathsf T}\Sigma a)aa^{\mathsf T}=A$ iff $c=1/a^{\mathsf T}\Sigma a$.)
3 $A=\begin{pmatrix}1&1/2\\1/2&1\end{pmatrix}$: $\operatorname{tr}A=2$, $\mu^{\mathsf T}A\mu=3$, so $E[Q]=5$. $A^2=\begin{pmatrix}5/4&1\\1&5/4\end{pmatrix}$: $\operatorname{tr}A^2=\tfrac52$, $\mu^{\mathsf T}A^2\mu=\tfrac92$, so $\operatorname{Var}(Q)=5+18=23$ (a simulation gives $4.99$ and $22.8$).
4 $C=I-\tfrac1nJ$ is symmetric idempotent of trace $n-1$, and $L=\tfrac1n\mathbf 1^{\mathsf T}$ has $LC=\tfrac1n(\mathbf 1^{\mathsf T}-\mathbf 1^{\mathsf T})=0$: independence by Theorem 7.11(1). $\sqrt n\bar X/\sigma\sim N(\sqrt n\theta/\sigma,1)$, so $n\bar X^2/\sigma^2\sim\chi^2_1(n\theta^2/\sigma^2)$.
5 $Q_i=(u_i^{\mathsf T}Z)^2$ with $u_1=\tfrac12(1,1,1,1)$, $u_2=\tfrac1{\sqrt2}(1,-1,0,0)$, $u_3=\tfrac1{\sqrt2}(0,0,1,-1)$, $u_4=\tfrac12(1,1,-1,-1)$, which are orthonormal. So $\sum A_i=I$ and the ranks add to $4$: the $Q_i$ are independent $\chi^2_1$. Then $Q_2+Q_3\sim\chi^2_2$ is independent of $Q_4$, and $\dfrac{2Q_4}{Q_2+Q_3}=\dfrac{Q_4/1}{(Q_2+Q_3)/2}\sim F_{1,2}$.
6 The eigenvalues are $0$ or $1$ (Theorem 7.10's proof), and trace and rank both count the unit eigenvalues. $(I-A)^2=I-2A+A^2=I-A$; its eigenvalues are $1-\lambda_i$, so its rank is the number of zero eigenvalues of $A$, $n-\operatorname{rank}A$.
7 $\operatorname{Cov}(Z^{\mathsf T}AZ,Z^{\mathsf T}BZ)=\tfrac14\big[\operatorname{Var}Z^{\mathsf T}(A+B)Z-\operatorname{Var}Z^{\mathsf T}(A-B)Z\big]=\tfrac14\cdot2\big[\operatorname{tr}(A+B)^2-\operatorname{tr}(A-B)^2\big]=2\operatorname{tr}(AB)$ by Theorem 7.9. For idempotent $P,Q$: $\operatorname{tr}(PQ)=\operatorname{tr}(P^2Q^2)=\operatorname{tr}(PQQP)=\|QP\|_F^2$. So zero covariance forces $QP=0$, hence $PQ=(QP)^{\mathsf T}=0$, and Theorem 7.11(2) gives independence.
8 $\chi^2_3(\delta)$ with $\delta=1+4+4=9$; mean $3+9=12$, variance $6+36=42$.
IV
The Wishart law and the sample covariance matrix.
  • 1 $W\sim W_3(10,\Sigma)$ with $\Sigma=\begin{pmatrix}2&1&0\\1&2&1\\0&1&2\end{pmatrix}$. Find $E[W]$, $\operatorname{Var}(W_{12})$, the law of $W_{11}+2W_{12}+W_{22}$, and the number $c$ with $P(W_{11}\gt c)=0.05$.
  • 2 Prove that $\operatorname{tr}(\Sigma^{-1}W)\sim\chi^2_{mp}$ for $W\sim W_p(m,\Sigma)$, $\Sigma\gt0$.
  • 3 For the four observations $(1,2),(3,3),(2,5),(6,6)$ compute $\bar x$, $S$, $|S|$ and the sample correlation.
  • 4 Prove that the sample covariance matrix of $n\le p$ observations in $\mathbb R^p$ is singular, whatever the population.
  • 5 For $W\sim W_2(m,\Sigma)$ with $m\ge2$ and $\Sigma\gt0$, prove $E|W|=m(m-1)|\Sigma|$ by writing $|W|=W_{22}\,W_{11\cdot2}$.
  • 6 Show that $E[S]=\Sigma$ for iid observations from any distribution with finite second moments, but that the Wishart law of $(n-1)S$ needs normality (consider the variance of $S_{11}$ when $p=1$).
  • 7 Show that $\bar X$ and $S^2$ are dependent for a sample of size $2$ from the Bernoulli$(\tfrac12)$ law.
1 $E[W]=10\Sigma$. $\operatorname{Var}(W_{12})=10(\sigma_{12}^2+\sigma_{11}\sigma_{22})=10(1+4)=50$. With $a=(1,1,0)$, $a^{\mathsf T}\Sigma a=2+2+2=6$: $W_{11}+2W_{12}+W_{22}\sim6\chi^2_{10}$. $W_{11}/2\sim\chi^2_{10}$ and $\chi^2_{10,0.05}=18.307$, so $c=36.61$.
2 $\operatorname{tr}(\Sigma^{-1}W)=\sum_{i=1}^mZ_i^{\mathsf T}\Sigma^{-1}Z_i$, a sum of $m$ independent $\chi^2_p$ variables (Theorem 7.8(1)), hence $\chi^2_{mp}$. Alternatively $\Sigma^{-1/2}W\Sigma^{-1/2}\sim W_p(m,I)$ and its trace is the sum of squares of all $mp$ entries of an $m\times p$ matrix of iid $N(0,1)$ variables.
3 $\bar x=(3,4)$; deviations $(-2,-2),(0,-1),(-1,1),(3,2)$; $A_{11}=14$, $A_{22}=10$, $A_{12}=4+0-1+6=9$. $S=\begin{pmatrix}14/3&3\\3&10/3\end{pmatrix}$, $|S|=\tfrac{140}9-9=\tfrac{59}9=6.56$, $r=9/\sqrt{140}=0.761$.
4 $A=\sum(x_i-\bar x)(x_i-\bar x)^{\mathsf T}$ is a sum of $n$ rank-one matrices whose vectors satisfy $\sum(x_i-\bar x)=0$, so they span a space of dimension at most $n-1$; hence $\operatorname{rank}A\le n-1\lt p$.
5 Theorem 7.7(1): $|W|=W_{22}W_{11\cdot2}$. By Theorem 7.13(3), $W_{22}/\sigma_{22}\sim\chi^2_m$. Writing $W$ through its data matrix, $W_{11\cdot2}$ is the residual sum of squares of column $1$ regressed (without intercept) on column $2$; given column $2$, column $1$ follows a normal linear model with error variance $\sigma_{11\cdot2}$, so $W_{11\cdot2}/\sigma_{11\cdot2}\sim\chi^2_{m-1}$ independently of column $2$. Hence $E|W|=m\sigma_{22}\cdot(m-1)\sigma_{11\cdot2}=m(m-1)|\Sigma|$. (Simulation with $m=6$, $|\Sigma|=5$: $149.5$ against $150$.)
6 $E[A]=\sum E[(X_i-\mu)(X_i-\mu)^{\mathsf T}]-nE[(\bar X-\mu)(\bar X-\mu)^{\mathsf T}]=n\Sigma-\Sigma$, using only second moments. For $p=1$, $\operatorname{Var}(S^2)=\frac{\sigma^4}{n}\left(\kappa-\frac{n-3}{n-1}\right)$ with $\kappa=\mu_4/\sigma^4$ (§4.1); it equals the Wishart value $2\sigma^4/(n-1)$ only when $\kappa=3$.
7 $S^2=\tfrac12(X_1-X_2)^2$. If $\bar X\in\{0,1\}$ then $S^2=0$; if $\bar X=\tfrac12$ then $S^2=\tfrac12$. So $P(S^2=0\mid\bar X=\tfrac12)=0\ne P(S^2=0)=\tfrac12$.
🎯 Exam corner CSIR Part BCSIR Part CGATE ST

E1. CSIR Part B GATE ST For which real $\rho$ is $\Sigma=\begin{pmatrix}1&\rho&0\\\rho&1&\rho\\0&\rho&1\end{pmatrix}$ the covariance matrix of a random vector with a density on $\mathbb R^3$? (A) $|\rho|\lt1$; (B) $|\rho|\lt1/\sqrt2$; (C) $|\rho|\lt1/2$; (D) $0\le\rho\lt1$.

Idea A density exists only if the covariance matrix is positive definite (Theorem 7.1(3)); use the leading principal minors.
Minors $1$, $\ 1-\rho^2$, $\ |\Sigma|=1(1-\rho^2)-\rho(\rho-0)+0=1-2\rho^2$. All positive iff $\rho^2\lt\tfrac12$.
Answer (B). (A) includes $\rho=0.8$, where $|\Sigma|=-0.28$; (C) is sufficient but not necessary; (D) excludes valid negative values and includes invalid ones.

E2. CSIR Part B $(X,Y,Z)$ is trivariate normal with zero means, unit variances and all three correlations equal to $\tfrac12$. Then $\operatorname{Var}(X\mid Y,Z)$ equals (A) $\tfrac12$; (B) $\tfrac23$; (C) $\tfrac34$; (D) $\tfrac13$.

Idea Theorem 7.6: $\Sigma_{11\cdot2}=1-\Sigma_{12}\Sigma_{22}^{-1}\Sigma_{21}$.
Compute $\Sigma_{22}=\begin{pmatrix}1&1/2\\1/2&1\end{pmatrix}$, $\Sigma_{22}^{-1}=\tfrac43\begin{pmatrix}1&-1/2\\-1/2&1\end{pmatrix}$, $\Sigma_{22}^{-1}(\tfrac12,\tfrac12)^{\mathsf T}=(\tfrac13,\tfrac13)^{\mathsf T}$, so $\Sigma_{12}\Sigma_{22}^{-1}\Sigma_{21}=\tfrac16+\tfrac16=\tfrac13$.
Answer (B) $\tfrac23$; also $E[X\mid Y,Z]=\tfrac13(Y+Z)$. (C) is $\operatorname{Var}(X\mid Y)=1-\tfrac14$, conditioning on one variable only.

E3. CSIR Part B Let $Z\sim N_4(0,I_4)$. Which of the following has the $\chi^2_2$ distribution? (A) $\tfrac12(Z_1+Z_2)^2+\tfrac12(Z_3-Z_4)^2$; (B) $Z_1^2+2Z_2^2$; (C) $\tfrac12(Z_1+Z_2+Z_3+Z_4)^2$; (D) $Z_1Z_2+Z_3Z_4$.

(A) Yes $(Z_1+Z_2)/\sqrt2$ and $(Z_3-Z_4)/\sqrt2$ are independent $N(0,1)$ (orthonormal coefficient vectors), so the sum of their squares is $\chi^2_2$.
Others (B) eigenvalues $1,2$: not idempotent, variance $2+8=10\ne4$. (C) $(Z_1+\dots+Z_4)/2\sim N(0,1)$, so (C) is $2\chi^2_1$. (D) has mean $0$.
Answer (A).

E4. CSIR Part C Let $(X_1,X_2)\sim N_2\left(0,\begin{pmatrix}1&\rho\\\rho&1\end{pmatrix}\right)$ with $0\lt\rho\lt1$. Which of the following are true? (One or more options may be correct.) (A) $X_1+X_2$ and $X_1-X_2$ are independent. (B) $X_1^2+X_2^2\sim\chi^2_2$. (C) $\dfrac{X_1^2-2\rho X_1X_2+X_2^2}{1-\rho^2}\sim\chi^2_2$. (D) $P(X_1\gt0,\ X_2\gt0)=\tfrac14$.

(A) True Jointly normal with covariance $\operatorname{Var}X_1-\operatorname{Var}X_2=0$ (Exercise II.4).
(B) False $A=I$ but $A\Sigma A=\Sigma\ne A$; the law is $(1+\rho)W_1+(1-\rho)W_2$ with $W_i$ iid $\chi^2_1$, of variance $2(1+\rho)^2+2(1-\rho)^2=4+4\rho^2\ne4$.
(C) True It is $X^{\mathsf T}\Sigma^{-1}X\sim\chi^2_2$ (Theorem 7.8).
(D) False Write $X_2=\rho X_1+\sqrt{1-\rho^2}\,W$ with $W\sim N(0,1)$ independent of $X_1$, and $(X_1,W)=(R\cos\theta,R\sin\theta)$ with $\theta$ uniform on $(-\pi,\pi]$. With $\varphi=\arcsin\rho$, $X_2=R\sin(\theta+\varphi)$, so the event is $\theta\in(-\varphi,\pi/2)$, of probability $\frac{\pi/2+\varphi}{2\pi}=\tfrac14+\frac{\arcsin\rho}{2\pi}\gt\tfrac14$.
Answer (A) and (C).

E5. CSIR Part C GATE ST Let $X_1,\dots,X_n$ be iid $N_p(\mu,\Sigma)$ with $\Sigma\gt0$ and $n\gt p$, and let $A=\sum(X_i-\bar X)(X_i-\bar X)^{\mathsf T}$. Which of the following are true? (A) $A\sim W_p(n,\Sigma)$. (B) $A_{11}/\sigma_{11}\sim\chi^2_{n-1}$. (C) $\bar X$ and $A$ are independent. (D) $\operatorname{tr}(\Sigma^{-1}A)\sim\chi^2_{p(n-1)}$.

(A) False One degree of freedom is used by $\bar X$: $A\sim W_p(n-1,\Sigma)$ (Theorem 7.14). $W_p(n,\Sigma)$ is the law of $\sum(X_i-\mu)(X_i-\mu)^{\mathsf T}$.
(B) True Diagonal entry of $W_p(n-1,\Sigma)$ (Theorem 7.13(3)).
(C) True Theorem 7.14(3).
(D) True Exercise IV.2 with $m=n-1$.
Answer (B), (C) and (D).

E6. CSIR Part C Let $Z\sim N_n(0,I_n)$, $n\ge3$, and let $P$ and $Q$ be symmetric idempotent $n\times n$ matrices of ranks $r\ge1$ and $s\ge1$. Which of the following are true? (A) If $PQ=0$, then $\dfrac{Z^{\mathsf T}PZ/r}{Z^{\mathsf T}QZ/s}\sim F_{r,s}$. (B) $Z^{\mathsf T}PZ+Z^{\mathsf T}QZ\sim\chi^2_{r+s}$ for all such $P,Q$. (C) If $P+Q=I_n$, then $Z^{\mathsf T}PZ$ and $Z^{\mathsf T}QZ$ are independent. (D) If $\operatorname{tr}(PQ)=0$, then $Z^{\mathsf T}PZ$ and $Z^{\mathsf T}QZ$ are independent.

(A) True Independent (Theorem 7.11(2)) $\chi^2_r$ and $\chi^2_s$ variables (Theorem 7.10), divided by their degrees of freedom.
(B) False Take $Q=P$: the sum is $2Z^{\mathsf T}PZ\sim2\chi^2_r$, whose variance $8r$ differs from $2\cdot2r=4r$. It is $\chi^2_{r+s}$ when $PQ=0$.
(C) True Cochran's theorem: $P$ and $Q$ are idempotent with $P+Q=I$, so $PQ=0$ (Theorem 7.12, (2) ⇒ (3)).
(D) True $\operatorname{tr}(PQ)=\operatorname{tr}(PPQQ)=\operatorname{tr}(PQQP)=\|QP\|_F^2$, so $\operatorname{tr}(PQ)=0$ forces $QP=0$ and $PQ=(QP)^{\mathsf T}=0$ (Exercise III.7).
Answer (A), (C) and (D).
🧑‍🏫 For the teacher

Suggested plan — 3 lectures × 60 min.

  1. Random vectors and the multivariate normal. Definition 7.1 and Theorem 7.1 with Example 1 on the board (ask the class whether the $-0.6$ matrix is valid before computing). Definition 7.2 and Theorem 7.2 — stress that the definition through linear combinations makes everything else short. Project Interactive 1: change the factor $A$ and ask what stays the same. Theorems 7.3–7.5, Examples 2 and 3.
  2. Conditioning and distances. Theorem 7.6 by the "subtract the regression" trick, Theorem 7.7 by block elimination, Example 4 (including the precision-matrix check). Definition 7.3, Theorem 7.8 and Example 5; return to the chapter-intro teaser and explain why red points exist. Assign Exercise II.2–3 as a discussion problem.
  3. Quadratic forms and Wishart. Definition 7.4, Theorems 7.9–7.11, Example 6; project Interactive 2 in the order centring → $Z_1^2+Z_1Z_2+Z_2^2$ → $X_1^2+X_2^2$ → $X^{\mathsf T}\Sigma^{-1}X$. Prove Cochran's theorem (Theorem 7.12) in full. Definition 7.5, Theorems 7.13 (parts 1–4 in detail, 5–6 sketched) and 7.14 via the Helmert matrix, Example 7. Finish with E4 and E6 live.

Discussion questions. (1) Why is it better to define the multivariate normal by linear combinations than by its density? What would be lost for a singular $\Sigma$? (2) Two variables are uncorrelated and each is normal. What extra information would you need before concluding independence, and how could you check it from data? (3) A quadratic form has the right mean for $\chi^2_k$. What simple numerical check could reveal that it is not chi-square?

Misconceptions to watch for. "Normal marginals ⇒ jointly normal"; "uncorrelated ⇒ independent" without joint normality; using $A^2=A$ instead of $A\Sigma A=A$ for correlated variables; taking the degrees of freedom of a non-idempotent form to be its trace; forgetting that $A=\sum(X_i-\bar X)(X_i-\bar X)^{\mathsf T}$ has $n-1$ degrees of freedom; believing that a marginally independent pair stays independent after conditioning; writing $\Sigma_{11}-\Sigma_{21}\Sigma_{22}^{-1}\Sigma_{12}$ with the blocks in the wrong order (the dimensions do not even match unless $q=p-q$).

Suggested homework. I(1, 2, 3, 7), II(2, 3, 5, 6), III(1, 2, 5, 7), IV(1, 3, 5, 7); Exam corner E2, E5, E6.

🔬 Research–Teaching NexusResearch-tutored · optional, about 60 minutes

Research question. Are real multivariate measurements close enough to normal for the methods of this chapter, and can the Mahalanobis distance pick out the observations that are not?

Why it matters. Hotelling's $T^2$, discriminant analysis and multivariate control charts all assume approximate multivariate normality, and all of them flag "unusual" observations through a Mahalanobis distance. Hospital laboratories screen blood panels this way, aircraft-engine monitoring systems flag sensor vectors whose combination is unusual even when each reading is within limits, and credit-card fraud filters work on the same principle. If the data are far from normal, the $\chi^2_p$ cut-offs are wrong and the false-alarm rate is not what the user believes.

What to do.

  1. Take a public multivariate data set with a few continuous variables — for example the four flower measurements of one species in the classical iris data, or a few physical measurements of athletes from a sports-science data set.
  2. Compute $\bar x$, $S$ and the squared distances $d_i^2=(x_i-\bar x)^{\mathsf T}S^{-1}(x_i-\bar x)$. Check numerically that $\sum d_i^2=(n-1)p$, and explain why from $\operatorname{tr}(S^{-1}A)$.
  3. Draw a chi-square Q–Q plot: the ordered $d_{(i)}^2$ against the $\chi^2_p$ quantiles at levels $(i-\tfrac12)/n$. A straight line through the origin with slope $1$ supports normality; an upward bend at the top points to heavy tails or outliers.
  4. Repeat the plot for simulated $N_p(\bar x,S)$ data of the same size, five times, to see how much a Q–Q plot wobbles when the model is exactly right.
  5. Compute Mardia's kurtosis $b_{2,p}=\frac1n\sum d_i^4$ (with $S$ replaced by $A/n$) and compare it with its normal-theory value $p(p+2)$.

Tools: Python (numpy, scipy, matplotlib) or R; a spreadsheet is enough for $p=2$.

Hand in: The two Q–Q plots (real and one simulated), the observations you would flag at $\chi^2_{p,0.01}$, the value of $b_{2,p}$ and a paragraph on whether the normal model is adequate.

Going further. With $\bar x$ and $S$ estimated from the same data, $\frac{n}{(n-1)^2}d_i^2$ has exactly a $\operatorname{Beta}\left(\tfrac p2,\tfrac{n-p-1}2\right)$ law under normality; compare these beta quantiles with the $\chi^2_p$ quantiles for small $n$. Read about robust alternatives (minimum covariance determinant estimates) that prevent outliers from masking themselves by inflating $S$.

Instructor note: the identity $\sum_id_i^2=\operatorname{tr}\big(S^{-1}\sum(x_i-\bar x)(x_i-\bar x)^{\mathsf T}\big)=\operatorname{tr}((n-1)I_p)=(n-1)p$ holds for every data set, so the average $d_i^2$ is $p(n-1)/n$, slightly below $p$. For small $n$ the $\chi^2_p$ approximation overstates the largest distances — the beta law is bounded by $(n-1)^2/n$ — which is a nice point to make when students' plots bend "the wrong way" at the top. Masking is easy to demonstrate: add two identical gross outliers and see how little their $d_i^2$ grows.

🧪 Self-check quiz

0/5 correct

Q1. A random vector $X\in\mathbb R^p$ is multivariate normal if and only if

Q2. If $(X_1,X_2)\sim N_2(\mu,\Sigma)$ and $\sigma_{12}=0$, then

Q3. For $X\sim N_p(\mu,\Sigma)$ partitioned into $(X_1,X_2)$ with $\Sigma_{22}\gt0$, the conditional covariance of $X_1$ given $X_2=x_2$ is

Q4. Let $Z\sim N_n(0,I)$ and $A$ symmetric. $Z^{\mathsf T}AZ$ is chi-square if and only if

Q5. For a sample of size $n$ from $N_p(\mu,\Sigma)$, the matrix $A=\sum(X_i-\bar X)(X_i-\bar X)^{\mathsf T}$ has the law