7.2 Partial and Multiple Correlation; Inference on Means
How strongly are two variables related once others are held fixed, how well is one variable predicted by several, and how do we test a claim about a whole mean vector at once? Partial and multiple correlation with their tests, Hotelling's $T^2$, and likelihood ratio tests for mean vectors and covariance structure.
Across a group of primary schools, pupils' reading scores are strongly correlated with their shoe sizes. Nobody believes that big feet help reading: both grow with age. The useful question is whether reading and shoe size are still related among children of the same age. The partial correlation answers it by removing the linear effect of the third variable from both and correlating what is left. Its companion, the multiple correlation coefficient, measures how well one variable can be predicted from several others together; it is the square root of the $R^2$ of a regression (§6.3). Both have simple formulas in terms of ordinary correlations, both have exact sampling distributions for normal data, and both are favourites of CSIR NET Part B.
The second half of the section tests hypotheses about a whole mean vector. A sports drink is meant to contain $50$ mg of sodium and $30$ mg of potassium per $100$ ml; two separate $t$-tests can each find nothing wrong while the pair of averages is jointly far from $(50,30)$, and with $p$ separate tests the overall chance of a false alarm grows with $p$. Hotelling's $T^2$ (1931) replaces the $p$ tests by one, built on the Mahalanobis distance of §7.1; its null distribution is an exact $F$ law, it is the likelihood ratio test, and it is the largest of all the univariate $t^2$ statistics for linear combinations. We finish with likelihood ratio tests for several mean vectors and for the structure of $\Sigma$.
Partial Correlation
Throughout, $X=(X_1^{\mathsf T},X_2^{\mathsf T})^{\mathsf T}$ is partitioned as in §7.1, with $\Sigma_{22}\gt0$; usually $X_1=(X_i,X_j)$ is a pair of variables and $X_2$ holds the variables to be "held fixed". We use the partial covariance matrix $\Sigma_{11\cdot2}=\Sigma_{11}-\Sigma_{12}\Sigma_{22}^{-1}\Sigma_{21}$ from the conditional-distribution theorem of §7.1.
The entries $\sigma_{ij\cdot2}$ of $\Sigma_{11\cdot2}$ are the partial covariances of the components of $X_1$ given (or "eliminating") $X_2$, and $$\rho_{ij\cdot2}=\frac{\sigma_{ij\cdot2}}{\sqrt{\sigma_{ii\cdot2}\,\sigma_{jj\cdot2}}}$$ is the partial correlation of $X_i$ and $X_j$ given $X_2$. For three variables we write $\rho_{12\cdot3}$ for the partial correlation of $X_1,X_2$ given $X_3$; with $k$ variables eliminated it is a partial correlation of order $k$, and $\rho_{12}$ is of order $0$.
- (Residuals, any distribution with finite second moments.) For each component $X_i$ of $X_1$, the linear predictor $a+b^{\mathsf T}X_2$ with the smallest mean squared error $E(X_i-a-b^{\mathsf T}X_2)^2$ has $b=\Sigma_{22}^{-1}\sigma_{2i}$ (where $\sigma_{2i}=\operatorname{Cov}(X_2,X_i)$) and $a=\mu_i-b^{\mathsf T}\mu_2$. The residuals $e_i=X_i-\mu_i-\sigma_{2i}^{\mathsf T}\Sigma_{22}^{-1}(X_2-\mu_2)$ satisfy $\operatorname{Cov}(e_i,e_j)=\sigma_{ij\cdot2}$, so $\rho_{ij\cdot2}=\operatorname{Corr}(e_i,e_j)$: a partial correlation is the correlation of the two residuals after linear regression on the fixed variables.
- (Normal case.) If $X$ is multivariate normal, $\rho_{ij\cdot2}$ is the correlation of $X_i$ and $X_j$ in their conditional law given $X_2=x_2$, the same for every $x_2$; and $\rho_{ij\cdot2}=0$ if and only if $X_i$ and $X_j$ are conditionally independent given $X_2$.
Proof. (1) $E(X_i-a-b^{\mathsf T}X_2)^2$ $=\operatorname{Var}(X_i-b^{\mathsf T}X_2)$ $+(\mu_i-a-b^{\mathsf T}\mu_2)^2$. The second term vanishes for $a=\mu_i-b^{\mathsf T}\mu_2$, and with $b_0=\Sigma_{22}^{-1}\sigma_{2i}$ the first is (complete the square) $$\begin{aligned}&\sigma_{ii}-2b^{\mathsf T}\sigma_{2i}+b^{\mathsf T}\Sigma_{22}b\\&\quad=\sigma_{ii}-\sigma_{2i}^{\mathsf T}\Sigma_{22}^{-1}\sigma_{2i}+(b-b_0)^{\mathsf T}\Sigma_{22}(b-b_0),\end{aligned}$$ which is smallest at $b=b_0$ because $\Sigma_{22}\gt0$. The vector of residuals is $e=X_1-\mu_1-B(X_2-\mu_2)$ with $B=\Sigma_{12}\Sigma_{22}^{-1}$, and the computation of $\operatorname{Cov}(U)$ in the proof of the conditional-distribution theorem (§7.1) — which uses only second moments — gives $\operatorname{Cov}(e)=\Sigma_{11\cdot2}$.
(2) By the conditional-distribution theorem (§7.1), $X_1\mid X_2=x_2\sim N_q(\cdot,\Sigma_{11\cdot2})$ with a covariance matrix that does not depend on $x_2$; its correlations are the $\rho_{ij\cdot2}$. In this conditional normal law, zero correlation of $X_i$ and $X_j$ is equivalent to their independence (§7.1, uncorrelated jointly normal variables are independent). ∎
Let $\rho_{12},\rho_{13},\rho_{23}$ be the correlations of $X_1,X_2,X_3$, with $|\rho_{13}|\lt1$ and $|\rho_{23}|\lt1$. Then $$\rho_{12\cdot3}=\frac{\rho_{12}-\rho_{13}\rho_{23}}{\sqrt{\left(1-\rho_{13}^2\right)\left(1-\rho_{23}^2\right)}}.$$ More generally, for a set $S$ of variables not containing $1,2,k$, $$\rho_{12\cdot S\cup\{k\}}=\frac{\rho_{12\cdot S}-\rho_{1k\cdot S}\,\rho_{2k\cdot S}}{\sqrt{\left(1-\rho_{1k\cdot S}^2\right)\left(1-\rho_{2k\cdot S}^2\right)}}:$$ partial correlations of order $k$ are computed from those of order $k-1$ by the same formula. Moreover the $3\times3$ correlation matrix $\mathcal P$ satisfies $$|\mathcal P|=\left(1-\rho_{13}^2\right)\left(1-\rho_{23}^2\right)\left(1-\rho_{12\cdot3}^2\right),$$ so a triple of numbers in $(-1,1)$ is a valid (positive definite) correlation matrix if and only if the formula gives $|\rho_{12\cdot3}|\lt1$.
Proof. Correlations do not change if each variable is standardised, so assume unit variances. With $X_2$ of Definition 7.6 equal to the single variable $X_3$, $\Sigma_{12}\Sigma_{22}^{-1}\Sigma_{21}$ is the matrix $\begin{pmatrix}\rho_{13}^2&\rho_{13}\rho_{23}\\\rho_{13}\rho_{23}&\rho_{23}^2\end{pmatrix}$, so $$\sigma_{11\cdot3}=1-\rho_{13}^2,\qquad\sigma_{22\cdot3}=1-\rho_{23}^2,\qquad\sigma_{12\cdot3}=\rho_{12}-\rho_{13}\rho_{23},$$ and the formula follows from Definition 7.6.
For the recursion, work in the space of mean-zero random variables with finite variance and the inner product $\langle U,V\rangle=\operatorname{Cov}(U,V)$. By Theorem 7.15(1), the residual $e_i^S$ of $X_i$ after regression on $X_S$ is the orthogonal projection of $X_i-\mu_i$ onto the orthogonal complement of $\operatorname{span}(X_S)$. Now $\operatorname{span}(X_S,X_k)=\operatorname{span}(X_S)\oplus\operatorname{span}(e_k^S)$, an orthogonal sum. So the residual of $X_1$ after regression on $S\cup\{k\}$ is the residual of $e_1^S$ after regression on $e_k^S$, and the same for $X_2$. Hence $\rho_{12\cdot S\cup\{k\}}$ is the first-order partial correlation of $e_1^S,e_2^S$ given $e_k^S$, whose ordinary correlations are $\rho_{12\cdot S},\rho_{1k\cdot S},\rho_{2k\cdot S}$; apply the first formula.
Finally, by the partitioned-determinant formula of §7.1 (with the single variable $X_3$ as the second block), $|\mathcal P|=1\cdot|\Sigma_{11\cdot3}|$ $=(1-\rho_{13}^2)(1-\rho_{23}^2)$ $-(\rho_{12}-\rho_{13}\rho_{23})^2$, which factorises as stated. Given $|\rho_{13}|,|\rho_{23}|\lt1$, the leading $2\times2$ minor on $\{1,3\}$ (order the variables as $1,3,2$) is positive, so $\mathcal P\gt0$ iff $|\mathcal P|\gt0$ iff $\rho_{12\cdot3}^2\lt1$. ∎
Let $\Sigma\gt0$ and $\Omega=\Sigma^{-1}=(\omega_{ij})$. The partial correlation of $X_i$ and $X_j$ given all the other $p-2$ variables is $$\rho_{ij\cdot\text{rest}}=-\frac{\omega_{ij}}{\sqrt{\omega_{ii}\,\omega_{jj}}}.$$ Hence, for a multivariate normal vector, $X_i$ and $X_j$ are conditionally independent given the rest if and only if $\omega_{ij}=0$.
Proof. Reorder so that $X_1=(X_i,X_j)$ and $X_2$ is the rest. By the partitioned-inverse formula of §7.1, $\Sigma_{11\cdot2}=\Omega_{11}^{-1}$ with $\Omega_{11}=\begin{pmatrix}\omega_{ii}&\omega_{ij}\\\omega_{ij}&\omega_{jj}\end{pmatrix}$, so $$\Sigma_{11\cdot2}=\frac{1}{\omega_{ii}\omega_{jj}-\omega_{ij}^2}\begin{pmatrix}\omega_{jj}&-\omega_{ij}\\-\omega_{ij}&\omega_{ii}\end{pmatrix},$$ whose correlation is $-\omega_{ij}/\sqrt{\omega_{ii}\omega_{jj}}$. The last statement is Theorem 7.15(2). ∎
Theorem 7.17 is the basis of Gaussian graphical models: draw an edge between $i$ and $j$ exactly when $\omega_{ij}\ne0$. The zeros of $\Sigma$ describe marginal independence; the zeros of $\Sigma^{-1}$ describe conditional independence, and the two patterns can be completely different (Exercise I.3).
The Multiple Correlation Coefficient
Now let $X_1$ be a single variable and $X_2=(X_2,\dots,X_p)$ the rest; write $\sigma_{12}=\operatorname{Cov}(X_1,X_2)$ (a row of length $p-1$) and $\sigma_{21}=\sigma_{12}^{\mathsf T}$.
The multiple correlation coefficient of $X_1$ on $X_2,\dots,X_p$ is $$R_{1\cdot2\cdots p}=\max_{b}\operatorname{Corr}\left(X_1,\ b^{\mathsf T}X_2\right),$$ the maximum over all $b$ with $\operatorname{Var}(b^{\mathsf T}X_2)\gt0$: the largest correlation between $X_1$ and any linear combination of the others.
Let $\sigma_{11}\gt0$ and $\Sigma_{22}\gt0$.
- The maximum in Definition 7.7 is attained at $b=\Sigma_{22}^{-1}\sigma_{21}$ — the best linear predictor of Theorem 7.15 — and $$R^2_{1\cdot2\cdots p}=\frac{\sigma_{12}\Sigma_{22}^{-1}\sigma_{21}}{\sigma_{11}},\qquad R\ge0.$$
- $1-R^2_{1\cdot2\cdots p}=\dfrac{\sigma_{11\cdot2}}{\sigma_{11}}=\dfrac{1}{\sigma_{11}\,\omega_{11}}=\dfrac{|\Sigma|}{\sigma_{11}|\Sigma_{22}|}=\dfrac{|\mathcal P|}{|\mathcal P_{22}|}$, where $\sigma_{11\cdot2}$ is the residual variance of $X_1$ after linear regression on the rest and $\mathcal P_{22}$ the correlation matrix of $X_2,\dots,X_p$.
- $0\le R\le1$; $R\ge|\rho_{1j}|$ for every $j$; $R$ never decreases when a variable is added to the predictors; $R=0$ iff $\sigma_{12}=0$; $R=1$ iff $X_1$ is (almost surely) a linear function of $X_2,\dots,X_p$.
- (Three variables.) $$R^2_{1\cdot23}=\frac{\rho_{12}^2+\rho_{13}^2-2\rho_{12}\rho_{13}\rho_{23}}{1-\rho_{23}^2},\qquad1-R^2_{1\cdot23}=\left(1-\rho_{12}^2\right)\left(1-\rho_{13\cdot2}^2\right),$$ and in general $1-R^2_{1\cdot2\cdots p}=(1-\rho_{12}^2)(1-\rho_{13\cdot2}^2)(1-\rho_{14\cdot23}^2)\cdots(1-\rho_{1p\cdot23\cdots(p-1)}^2)$.
Proof. (1) $\operatorname{Cov}(X_1,b^{\mathsf T}X_2)=\sigma_{12}b$ and $\operatorname{Var}(b^{\mathsf T}X_2)=b^{\mathsf T}\Sigma_{22}b$. In the inner product $\langle u,v\rangle=u^{\mathsf T}\Sigma_{22}v$ we have $\sigma_{12}b=\langle\Sigma_{22}^{-1}\sigma_{21},b\rangle$, so by the Cauchy–Schwarz inequality $$\operatorname{Corr}^2(X_1,b^{\mathsf T}X_2)=\frac{\langle\Sigma_{22}^{-1}\sigma_{21},b\rangle^2}{\sigma_{11}\langle b,b\rangle}\le\frac{\langle\Sigma_{22}^{-1}\sigma_{21},\Sigma_{22}^{-1}\sigma_{21}\rangle}{\sigma_{11}}=\frac{\sigma_{12}\Sigma_{22}^{-1}\sigma_{21}}{\sigma_{11}},$$ with equality iff $b$ is a multiple of $\Sigma_{22}^{-1}\sigma_{21}$. For $b=\Sigma_{22}^{-1}\sigma_{21}$ itself the covariance $\sigma_{12}\Sigma_{22}^{-1}\sigma_{21}$ is $\ge0$, so the maximum of the correlation (not only of its square) is the nonnegative root. (If $\sigma_{12}=0$ every $b$ gives correlation $0$.)
(2) $\sigma_{11\cdot2}=\sigma_{11}-\sigma_{12}\Sigma_{22}^{-1}\sigma_{21}=\sigma_{11}(1-R^2)$ by (1), and it is the residual variance by Theorem 7.15. By the partitioned determinant and inverse formulas of §7.1, $|\Sigma|=|\Sigma_{22}|\sigma_{11\cdot2}$ and $\omega_{11}=1/\sigma_{11\cdot2}$. Rescaling to unit variances does not change $R$ and turns $\Sigma$ into $\mathcal P$ (with $\sigma_{11}=1$).
(3) $R$ is a correlation (or $0$) so $0\le R\le1$; choosing $b=\pm e_j$ gives $R\ge|\rho_{1j}|$; adding a predictor enlarges the set of combinations over which the maximum is taken. $R=0$ iff $\sigma_{12}\Sigma_{22}^{-1}\sigma_{21}=0$ iff $\sigma_{21}=0$ (as $\Sigma_{22}^{-1}\gt0$). $R=1$ iff $\sigma_{11\cdot2}=\operatorname{Var}(e_1)=0$ iff $X_1=\mu_1+b^{\mathsf T}(X_2-\mu_2)$ almost surely.
(4) With unit variances, $\mathcal P_{22}^{-1}=\frac1{1-\rho_{23}^2}\begin{pmatrix}1&-\rho_{23}\\-\rho_{23}&1\end{pmatrix}$, and (1) gives the first formula. For the product: by (2) and Theorem 7.16, $1-R^2_{1\cdot23}=|\mathcal P|/(1-\rho_{23}^2)$ and $|\mathcal P|=(1-\rho_{12}^2)(1-\rho_{23}^2)(1-\rho_{13\cdot2}^2)$ (Theorem 7.16 with the roles of $2$ and $3$ exchanged). In general, adding $X_{k}$ to the predictors multiplies the residual variance of $X_1$ by $1-\rho_{1k\cdot2\cdots(k-1)}^2$, because the new residual is the residual of $e_1$ after regression on $e_k$ (the orthogonal decomposition in the proof of Theorem 7.16). ∎
Sample Partial and Multiple Correlations and Their Tests
For a sample with covariance matrix $S$ (or correlation matrix $R=(r_{ij})$), the sample partial correlation $r_{ij\cdot2}$ and the sample multiple correlation $R_{1\cdot2\cdots p}$ are given by the formulas of Definitions 7.6–7.7 and Theorems 7.16–7.18 with $\Sigma$ replaced by $S$ (or $\rho_{ij}$ by $r_{ij}$). Equivalently (Theorem 7.15 applied to the empirical distribution), $r_{ij\cdot2}$ is the ordinary correlation of the least-squares residuals of $x_i$ and $x_j$ after regression on the variables in $X_2$ with an intercept, and $R^2_{1\cdot2\cdots p}$ is the coefficient of determination $R^2$ of the least-squares regression of $x_1$ on $x_2,\dots,x_p$ with an intercept (§6.3).
Let $X_1,\dots,X_n$ be iid $N_p(\mu,\Sigma)$.
- (Simple correlation.) If $\rho_{12}=0$ and $n\ge3$: $\ T=\dfrac{r_{12}\sqrt{n-2}}{\sqrt{1-r_{12}^2}}\sim t_{n-2}$.
- (Partial correlation.) If $\rho_{12\cdot S}=0$, where $S$ is a set of $k$ other variables and $n\ge k+3$: $\ T=\dfrac{r_{12\cdot S}\sqrt{n-2-k}}{\sqrt{1-r_{12\cdot S}^2}}\sim t_{n-2-k}$. A sample partial correlation of order $k$ behaves like a simple correlation computed from $n-k$ observations.
- (Multiple correlation.) If $R_{1\cdot2\cdots p}=0$ and $n\gt p$: $\ F=\dfrac{R^2/(p-1)}{(1-R^2)/(n-p)}\sim F_{p-1,n-p}$; equivalently $R^2\sim\operatorname{Beta}\left(\tfrac{p-1}2,\tfrac{n-p}2\right)$, so $E[R^2]=\frac{p-1}{n-1}$.
- (Fisher's $z$, large $n$, any $\rho$.) $z=\operatorname{artanh}r=\tfrac12\log\frac{1+r}{1-r}$ is approximately $N\left(\operatorname{artanh}\rho,\ \frac1{n-3}\right)$; for a partial correlation of order $k$ use variance $\frac1{n-3-k}$.
Proof. (3) Condition on the values $x_{2i},\dots,x_{pi}$ of the predictors for all $n$ observations. By the conditional-distribution theorem of §7.1, the $X_{1i}$ are then independent with $X_{1i}\sim N\big(\alpha+\beta^{\mathsf T}x_{(2)i},\ \sigma_{11\cdot2}\big)$, where $\beta=\Sigma_{22}^{-1}\sigma_{21}$: a normal linear model $Y=X\beta+\varepsilon$ as in §6.1. The hypothesis $R=0$ means $\sigma_{21}=0$, i.e. $\beta=0$. The $F$-test of the linear hypothesis $\beta=0$ (§6.1) uses $F=\frac{(\operatorname{SST}-\operatorname{SSE})/(p-1)}{\operatorname{SSE}/(n-p)}$, and since $R^2=1-\operatorname{SSE}/\operatorname{SST}$ this is the statistic in (3). Conditionally on the predictors it has the $F_{p-1,n-p}$ law under $H_0$; this law does not depend on the predictor values, so it is also the unconditional law. The beta form follows from the relation between $F$ and beta laws (§4.1), and the beta mean is $\frac{(p-1)/2}{(n-1)/2}$.
(1) is (3) with $p=2$: $F=T^2$, and $T$ is symmetric about $0$ under $H_0$, so $T\sim t_{n-2}$. (It is also the $t$ statistic for the slope in the regression of $x_1$ on $x_2$, §6.3.)
(2) Let $e_1,e_2$ be the least-squares residuals of $x_1,x_2$ on an intercept and the $k$ variables of $S$, and let $\operatorname{SSE}_0=\|e_1\|^2$ and $\operatorname{SSE}_1$ be the residual sums of squares of $x_1$ on (intercept, $S$) and on (intercept, $S$, $x_2$). The second model's column space is the first one plus the direction $e_2$, orthogonally, so $\operatorname{SSE}_1=\|e_1\|^2-(e_1^{\mathsf T}e_2)^2/\|e_2\|^2$ $=\operatorname{SSE}_0(1-r_{12\cdot S}^2)$, because the residuals have mean $0$ and $r_{12\cdot S}=e_1^{\mathsf T}e_2/(\|e_1\|\|e_2\|)$ (Definition 7.8). Hence the $t$ statistic for the coefficient of $x_2$ in the larger regression satisfies $$t^2=\frac{\operatorname{SSE}_0-\operatorname{SSE}_1}{\operatorname{SSE}_1/(n-k-2)}=\frac{(n-k-2)\,r_{12\cdot S}^2}{1-r_{12\cdot S}^2}.$$ Given the values of $x_2$ and $X_S$, $X_1$ follows a normal linear model whose coefficient on $x_2$ is $\sigma_{12\cdot S}/\sigma_{22\cdot S}$ (Theorem 7.15 with the pair $(X_1,X_2)$ given $X_S$, then given $X_2$); it vanishes iff $\rho_{12\cdot S}=0$. So under $H_0$, conditionally and hence unconditionally, $t\sim t_{n-k-2}$, and $t$ has the sign of $r_{12\cdot S}$.
(4) (Sketch.) The delta method (§2.3) applied to the asymptotic law $\sqrt n(r-\rho)\Rightarrow N(0,(1-\rho^2)^2)$ for bivariate normal data gives $\sqrt n(z-\operatorname{artanh}\rho)\Rightarrow N(0,1)$, because $\frac{d}{d\rho}\operatorname{artanh}\rho=\frac1{1-\rho^2}$: the transformation stabilises the variance. Replacing $n$ by $n-3$ is a small-sample correction that makes the variance very accurate. ∎
- Count what is eliminated. Simple $r$: $t$ with $n-2$ df. Partial correlation with $k$ variables held fixed: $t$ with $n-2-k$ df. Multiple $R$ of one variable on $q=p-1$ others: $F$ with $(q,\ n-q-1)$ df.
- Testing $\rho=\rho_0\ne0$, intervals, or comparing two correlations: use Fisher's $z$ with variance $1/(n-3-k)$; for two independent samples, $\frac{z_1-z_2}{\sqrt{1/(n_1-3)+1/(n_2-3)}}\approx N(0,1)$.
- Several variables, "are they all uncorrelated?": Bartlett's test based on $|R|$ (Theorem 7.25 below), not many separate $t$-tests.
Hotelling's $T^2$ for One and Two Mean Vectors
(a) One sample. For $X_1,\dots,X_n$ in $\mathbb R^p$ with $n\gt p$, mean $\bar X$ and covariance matrix $S$, and a hypothesised mean $\mu_0$, $$T^2=n\,(\bar X-\mu_0)^{\mathsf T}S^{-1}(\bar X-\mu_0).$$
(b) Two samples. For independent samples of sizes $n_1,n_2$ with means $\bar X_1,\bar X_2$, covariance matrices $S_1,S_2$ and pooled covariance $S_{\text{pl}}=\frac{(n_1-1)S_1+(n_2-1)S_2}{n_1+n_2-2}$, $$T^2=\frac{n_1n_2}{n_1+n_2}\,(\bar X_1-\bar X_2)^{\mathsf T}S_{\text{pl}}^{-1}(\bar X_1-\bar X_2).$$ For $p=1$ these are the squares of the one- and two-sample $t$ statistics.
Let $X_1,\dots,X_n$ be iid $N_p(\mu,\Sigma)$ with $\Sigma\gt0$ and $n\gt p$. Then $$\frac{n-p}{p(n-1)}\,T^2\sim F_{p,n-p}(\delta),\qquad\delta=n(\mu-\mu_0)^{\mathsf T}\Sigma^{-1}(\mu-\mu_0),$$ a noncentral $F$ law that is central under $H_0:\mu=\mu_0$. The size-$\alpha$ test rejects $H_0$ when $$T^2\gt\frac{p(n-1)}{n-p}\,F_{p,n-p,\alpha}.$$ Its power depends on $\mu$ and $\Sigma$ only through $\delta$, and increases with $\delta$.
Proof. Let $d=\sqrt n(\bar X-\mu_0)$ and $A=(n-1)S$. By the sampling theorem of §7.1, $d\sim N_p(\sqrt n(\mu-\mu_0),\Sigma)$, $A\sim W_p(n-1,\Sigma)$, and $d$ and $A$ are independent. Now $T^2=d^{\mathsf T}S^{-1}d=(n-1)\,d^{\mathsf T}A^{-1}d$, and we write $$\frac{T^2}{n-1}=\frac{U}{V},\qquad U=d^{\mathsf T}\Sigma^{-1}d,\qquad V=\frac{d^{\mathsf T}\Sigma^{-1}d}{d^{\mathsf T}A^{-1}d}.$$ $U$ is the squared Mahalanobis length of a normal vector: $U=\|\Sigma^{-1/2}d\|^2$ with $\Sigma^{-1/2}d\sim N_p(\sqrt n\Sigma^{-1/2}(\mu-\mu_0),I)$, so $U\sim\chi^2_p(\delta)$. For $V$ use the inverse-Wishart property of §7.1 (Theorem on the properties of the Wishart law, part on $a^{\mathsf T}\Sigma^{-1}a/a^{\mathsf T}W^{-1}a$) with $m=n-1$ and the random direction $a=d$, which is independent of $A$ and nonzero almost surely: $V\sim\chi^2_{(n-1)-p+1}=\chi^2_{n-p}$, and $V$ is independent of $d$, hence of $U$. Therefore $$\frac{n-p}{p}\cdot\frac{T^2}{n-1}=\frac{U/p}{V/(n-p)}\sim F_{p,n-p}(\delta).$$ The power statement holds because the noncentral $F$ law is stochastically increasing in $\delta$ (§6.1). ∎
- (Likelihood ratio.) For iid $N_p(\mu,\Sigma)$ data with both parameters unknown and $n\gt p$, the likelihood ratio statistic for $H_0:\mu=\mu_0$ is $\Lambda=\left(|\hat\Sigma|/|\hat\Sigma_0|\right)^{n/2}$, and $$\Lambda^{2/n}=\left(1+\frac{T^2}{n-1}\right)^{-1}.$$ So the LRT rejects for large $T^2$ and is the $T^2$-test.
- (Union–intersection.) For $a\ne0$ let $t_a=\dfrac{\sqrt n\,a^{\mathsf T}(\bar X-\mu_0)}{\sqrt{a^{\mathsf T}Sa}}$, the univariate $t$ statistic for $H_0:a^{\mathsf T}\mu=a^{\mathsf T}\mu_0$. Then $T^2=\max_{a\ne0}t_a^2$, attained at $a\propto S^{-1}(\bar X-\mu_0)$.
- (Invariance.) $T^2$ does not change if every observation is transformed by $x\mapsto Cx+b$ with $C$ nonsingular and $\mu_0$ is replaced by $C\mu_0+b$.
Proof. (1) By the sampling theorem of §7.1, the unrestricted MLE is $\hat\Sigma=A/n$, and the maximised likelihood is $(2\pi)^{-np/2}|\hat\Sigma|^{-n/2}e^{-np/2}$ (at the MLE, $\operatorname{tr}(\hat\Sigma^{-1}A)=np$). Under $H_0$ the same argument with $\mu=\mu_0$ gives $\hat\Sigma_0=\frac1n\sum(X_i-\mu_0)(X_i-\mu_0)^{\mathsf T}$ $=\frac1n\left(A+n\,vv^{\mathsf T}\right)$ with $v=\bar X-\mu_0$. So $\Lambda^{2/n}=|A|/|A+nvv^{\mathsf T}|$. The matrix determinant lemma $|A+uu^{\mathsf T}|=|A|(1+u^{\mathsf T}A^{-1}u)$ (proof: $|A+uu^{\mathsf T}|=|A|\,|I+A^{-1}uu^{\mathsf T}|$, and $I+wu^{\mathsf T}$ has eigenvalue $1+u^{\mathsf T}w$ on $w$ and $1$ on the $(p-1)$-dimensional space $u^\perp$) with $u=\sqrt nv$ gives $\Lambda^{2/n}=(1+nv^{\mathsf T}A^{-1}v)^{-1}$ $=(1+T^2/(n-1))^{-1}$.
(2) Cauchy–Schwarz in the inner product defined by $S\gt0$: $(a^{\mathsf T}v)^2=\langle a,S^{-1}v\rangle_S^2\le(a^{\mathsf T}Sa)(v^{\mathsf T}S^{-1}v)$, with equality iff $a\propto S^{-1}v$. Hence $t_a^2=n(a^{\mathsf T}v)^2/(a^{\mathsf T}Sa)\le nv^{\mathsf T}S^{-1}v=T^2$, with equality at $a=S^{-1}v$.
(3) The new mean is $C\bar X+b$, the new covariance matrix $CSC^{\mathsf T}$, and $(C\bar X-C\mu_0)^{\mathsf T}(CSC^{\mathsf T})^{-1}(C\bar X-C\mu_0)$ $=v^{\mathsf T}S^{-1}v$, exactly as for the Mahalanobis distance in §7.1. ∎
Part (2) explains the relation between $T^2$ and univariate tests. The $T^2$-test rejects iff some linear combination $a^{\mathsf T}\mu$ looks significant by a $t$-test with the inflated critical value $\frac{p(n-1)}{n-p}F_{p,n-p,\alpha}$ instead of $t_{n-1,\alpha/2}^2$. In particular it can reject when every coordinate test accepts (Example 6), and a coordinate test at level $\alpha$ can reject when $T^2$ accepts (Exercise IV.3).
In the setting of Theorem 7.20, let $c^2=\frac{p(n-1)}{n-p}F_{p,n-p,\alpha}$.
- $\left\{\mu:\ n(\bar X-\mu)^{\mathsf T}S^{-1}(\bar X-\mu)\le c^2\right\}$ is a $100(1-\alpha)\%$ confidence ellipsoid for $\mu$.
- With probability $1-\alpha$, simultaneously for all $a\in\mathbb R^p$, $$a^{\mathsf T}\mu\in a^{\mathsf T}\bar X\pm c\sqrt{a^{\mathsf T}Sa/n}.$$
- (Bonferroni.) For $m$ combinations chosen in advance, the intervals $a_j^{\mathsf T}\bar X\pm t_{n-1,\alpha/(2m)}\sqrt{a_j^{\mathsf T}Sa_j/n}$ have simultaneous coverage at least $1-\alpha$; they are shorter than the $T^2$ intervals when $m$ is small.
Proof. (1) Under the true $\mu$, $T^2(\mu)\le c^2$ has probability $1-\alpha$ by Theorem 7.20. (2) By Theorem 7.21(2), $T^2(\mu)\le c^2$ iff $t_a^2(\mu)\le c^2$ for every $a$, i.e. iff every one of the stated intervals contains $a^{\mathsf T}\mu$. (3) Each interval misses with probability $\alpha/m$, and $P(\bigcup B_j)\le\sum P(B_j)=\alpha$. ∎
Let $X_{11},\dots,X_{1n_1}$ be iid $N_p(\mu_1,\Sigma)$ and $X_{21},\dots,X_{2n_2}$ iid $N_p(\mu_2,\Sigma)$, independent, with the same $\Sigma\gt0$ and $n_1+n_2-2\ge p$. Under $H_0:\mu_1=\mu_2$, $$\frac{n_1+n_2-p-1}{p(n_1+n_2-2)}\,T^2\sim F_{p,\ n_1+n_2-p-1},$$ and in general the law is noncentral with $\delta=\frac{n_1n_2}{n_1+n_2}(\mu_1-\mu_2)^{\mathsf T}\Sigma^{-1}(\mu_1-\mu_2)$. For paired observations (two measurements on each of $n$ units), apply the one-sample $T^2$ to the differences.
Proof. Let $d=\sqrt{\frac{n_1n_2}{n_1+n_2}}(\bar X_1-\bar X_2)$, so $d\sim N_p\left(\sqrt{\tfrac{n_1n_2}{n_1+n_2}}(\mu_1-\mu_2),\Sigma\right)$ because $\operatorname{Cov}(\bar X_1-\bar X_2)=\left(\frac1{n_1}+\frac1{n_2}\right)\Sigma$. By the sampling theorem of §7.1, $(n_1-1)S_1\sim W_p(n_1-1,\Sigma)$ and $(n_2-1)S_2\sim W_p(n_2-1,\Sigma)$, independent of each other and of $\bar X_1,\bar X_2$; by additivity, $A=(n_1+n_2-2)S_{\text{pl}}\sim W_p(m,\Sigma)$ with $m=n_1+n_2-2$, independent of $d$. Then $T^2=m\,d^{\mathsf T}A^{-1}d$, and the proof of Theorem 7.20 with $n-1$ replaced by $m$ gives $\frac{m-p+1}{p}\cdot\frac{T^2}{m}\sim F_{p,m-p+1}(\delta)$, which is the claim since $m-p+1=n_1+n_2-p-1$. ∎
Likelihood Ratio Tests for Mean Vectors and Covariance Structure
- ($\Sigma$ known.) For iid $N_p(\mu,\Sigma)$ data with known $\Sigma\gt0$, the LRT of $H_0:\mu=\mu_0$ rejects when $n(\bar X-\mu_0)^{\mathsf T}\Sigma^{-1}(\bar X-\mu_0)\gt\chi^2_{p,\alpha}$, and this test has size exactly $\alpha$.
- (Large samples.) Under $H_0$, as $n\to\infty$ with $p$ fixed, $T^2\Rightarrow\chi^2_p$, and $-2\log\Lambda=n\log\left(1+\frac{T^2}{n-1}\right)\Rightarrow\chi^2_p$, in agreement with Wilks' theorem (§5.1). The convergence $T^2\Rightarrow\chi^2_p$ holds for any population with finite $\Sigma\gt0$ (the multivariate CLT).
- (Several mean vectors, one-way MANOVA.) For $k$ independent samples of sizes $n_1,\dots,n_k$ ($N=\sum n_j$) from $N_p(\mu_j,\Sigma)$, let $W=\sum_j\sum_i(X_{ji}-\bar X_j)(X_{ji}-\bar X_j)^{\mathsf T}$ (within groups) and $B=\sum_jn_j(\bar X_j-\bar X)(\bar X_j-\bar X)^{\mathsf T}$ (between groups). The LRT of $H_0:\mu_1=\cdots=\mu_k$ rejects for small Wilks' lambda $$\Lambda_W=\frac{|W|}{|W+B|},$$ with Bartlett's approximation $-\left(N-1-\frac{p+k}2\right)\log\Lambda_W\approx\chi^2_{p(k-1)}$. For $k=2$, $\Lambda_W=\left(1+\frac{T^2}{N-2}\right)^{-1}$ with the two-sample $T^2$, and the test is exact.
Proof. (1) With $\Sigma$ known, the identity used for the MLE in §7.1 shows that the log-likelihood depends on $\mu$ only through $-\frac n2(\bar x-\mu)^{\mathsf T}\Sigma^{-1}(\bar x-\mu)$, so $-2\log\Lambda=n(\bar X-\mu_0)^{\mathsf T}\Sigma^{-1}(\bar X-\mu_0)$, which is $\chi^2_p$ under $H_0$ because $\sqrt n(\bar X-\mu_0)\sim N_p(0,\Sigma)$ (§7.1, Mahalanobis distance). (2) $S\to\Sigma$ almost surely (strong law, entry by entry), $\sqrt n(\bar X-\mu_0)\Rightarrow N_p(0,\Sigma)$ by the multivariate CLT, so $T^2\Rightarrow\chi^2_p$ by Slutsky's theorem and continuous mapping (§2.3); and $n\log(1+T^2/(n-1))-T^2\to0$ in probability. (3) (Sketch.) Under the full model the MLE of $\Sigma$ is $W/N$; under $H_0$ it is $(W+B)/N$, the total sum of squares and products divided by $N$; the ratio of maximised likelihoods is $(|W|/|W+B|)^{N/2}$. For $k=2$, $B=\frac{n_1n_2}{N}(\bar X_1-\bar X_2)(\bar X_1-\bar X_2)^{\mathsf T}$ has rank one, and the matrix determinant lemma (proof of Theorem 7.21) gives $|W+B|=|W|(1+T^2/(N-2))$. ∎
Let $X_1,\dots,X_n$ be iid $N_p(\mu,\Sigma)$, $\hat\Sigma=A/n$, and $R$ the sample correlation matrix. Each statistic below is $-2\log$ of a likelihood ratio, with the large-sample $\chi^2$ law given by Wilks' theorem (§5.1); the degrees of freedom are the number of free covariance parameters removed by $H_0$.
- $H_0:\Sigma=\Sigma_0$ ($\mu$ unknown): $\ n\left[\operatorname{tr}(\Sigma_0^{-1}\hat\Sigma)-\log|\Sigma_0^{-1}\hat\Sigma|-p\right]\approx\chi^2_{p(p+1)/2}$.
- $H_0$: the components are independent ($\Sigma$ diagonal): $\ -n\log|R|\approx\chi^2_{p(p-1)/2}$; Bartlett's corrected version $-\left(n-1-\frac{2p+5}{6}\right)\log|R|$ is more accurate.
- $H_0$: two sets of variables are independent ($\Sigma_{12}=0$, sizes $p_1,p_2$): $\ -n\log\dfrac{|S|}{|S_{11}|\,|S_{22}|}\approx\chi^2_{p_1p_2}$. For $p_1=1$ this is $-n\log(1-R^2_{1\cdot2\cdots p})$, a monotone function of the multiple correlation.
- $H_0:\Sigma_1=\cdots=\Sigma_k$ for $k$ groups: Box's $M$ statistic, based on $\sum(n_j-1)\log|S_{\text{pl}}|-\sum(n_j-1)\log|S_j|$, approximately $\chi^2_{(k-1)p(p+1)/2}$ after a scale correction.
Justification of (2). Under $H_0$, the MLE of the diagonal $\Sigma$ is $\operatorname{diag}(\hat\sigma_{11},\dots,\hat\sigma_{pp})$, and the maximised likelihoods are proportional to $|\hat\Sigma|^{-n/2}$ and $\prod_j\hat\sigma_{jj}^{-n/2}$ (with the same factor $e^{-np/2}$). So $\lambda^{2/n}=|\hat\Sigma|/\prod_j\hat\sigma_{jj}=|R|$ and $-2\log\lambda=-n\log|R|$. The full model has $p(p+1)/2$ covariance parameters and $H_0$ has $p$, a difference of $p(p-1)/2$. Statement (3) is proved in the same way with the block-diagonal MLE, and $|S|/(|S_{11}||S_{22}|)=1-R^2$ for $p_1=1$ by Theorem 7.18(2).
- Zero correlation and zero partial correlation are unrelated. $\rho_{12}=0$ with $\rho_{12\cdot3}=-0.5625$ (Example 2(a)); $\rho_{12}=0.48$ with $\rho_{12\cdot3}=0$ (Example 1).
- Partial correlations can reverse sign: $\rho_{12}=0.4$ but $\rho_{12\cdot3}=-0.667$ when $\rho_{13}=\rho_{23}=0.8$.
- Not every triple of correlations is possible. $(0.9,0.9,0.5)$ gives $\rho_{12\cdot3}\gt1$ and $|\mathcal P|\lt0$; always check $|\rho_{12\cdot3}|\lt1$ or $|\mathcal P|\gt0$ before interpreting.
- Partial correlation is conditional correlation only for normal data. In general it is the correlation of linear-regression residuals; the conditional correlation given $X_3=x_3$ may vary with $x_3$ and differ from $\rho_{12\cdot3}$.
- Marginally dependent, conditionally independent. With $\Omega$ tridiagonal (Exercise I.3), $X_1$ and $X_3$ are correlated but $\rho_{13\cdot2}=0$. Zeros of $\Sigma$ and zeros of $\Sigma^{-1}$ mean different things.
- $R^2$ is not $\sum\rho_{1j}^2$ unless the predictors are uncorrelated; it can be larger (a suppressor, Example 4(a)) or smaller (redundant predictors). And $R=1$ is possible with small simple correlations (Example 4(d)).
- Sample $R^2$ is biased upwards. Under $R=0$, $E[R^2]=(p-1)/(n-1)$: ten useless predictors and $n=21$ give $R^2\approx0.5$ on average.
- Degrees of freedom. Partial correlation of order $k$: $t_{n-2-k}$, not $t_{n-2}$; multiple correlation on $p-1$ predictors: $F_{p-1,n-p}$; one-sample $T^2$: $\frac{n-p}{p(n-1)}T^2\sim F_{p,n-p}$, not $\frac{n-p}{p}T^2$; two-sample: $F_{p,n_1+n_2-p-1}$, not $n_1+n_2-2$.
- $T^2$ and coordinate tests can disagree in both directions. $T^2$ rejects while no $t$-test does (Example 6), and a $t$-test at level $\alpha$ rejects while $T^2$ accepts (Exercise IV.3).
- $T^2$ needs $n\gt p$ (two samples: $n_1+n_2-2\ge p$). Otherwise $S$ is singular (§7.1) and $T^2$ is undefined.
- The two-sample $T^2$ assumes a common $\Sigma$. With unequal covariance matrices and unequal sizes the exact $F$ law fails (the multivariate Behrens–Fisher problem).
- $E[T^2]$ is not $p$. Under $H_0$, $E[T^2]=\frac{p(n-1)}{n-p-2}$ for $n\gt p+2$; it tends to $p$ only as $n\to\infty$.
Exercises
- 1 Given $r_{12}=0.7$, $r_{13}=0.5$, $r_{23}=0.6$, compute $r_{12\cdot3}$, $r_{13\cdot2}$ and $r_{23\cdot1}$.
- 2 Show that $\rho_{12\cdot3}=0$ if and only if $\rho_{12}=\rho_{13}\rho_{23}$, and interpret this when $X_1=aX_3+U$, $X_2=bX_3+V$ with $U,V,X_3$ uncorrelated.
- 3 $X\sim N_3(0,\Sigma)$ with precision matrix $\Omega=\Sigma^{-1}=\begin{pmatrix}2&-1&0\\-1&2&-1\\0&-1&2\end{pmatrix}$. Find all partial correlations given the third variable, and $\Sigma$. Are $X_1$ and $X_3$ independent? Conditionally independent given $X_2$?
- 4 Give numerical examples of three correlations with (a) $\rho_{12}\ne0$ but $\rho_{12\cdot3}=0$; (b) $\rho_{12}=0$ but $\rho_{12\cdot3}\ne0$.
- 5 Prove that $|\rho_{12\cdot3}|\le1$ for every valid correlation matrix, with equality only if $|\mathcal P|=0$.
- 6 $X_1,X_2,X_3$ have $\operatorname{Corr}(X_i,X_j)=\phi^{|i-j|}$ with $|\phi|\lt1$ (as in a stationary autoregressive series). Show that $\rho_{13\cdot2}=0$ and find $\rho_{12\cdot3}$.
- 7 For three equicorrelated variables with correlation $\rho$, show $\rho_{12\cdot3}=\rho/(1+\rho)$. For $p$ equicorrelated variables, show that the partial correlation of two of them given the other $p-2$ is $\rho/(1+(p-2)\rho)$.
- 8 Show that $\rho_{12\cdot3}$ is unchanged by $X_i\mapsto a_iX_i+b_i$ with all $a_i\gt0$, and changes sign if exactly one of $a_1,a_2$ is negative. What happens if $a_3\lt0$?
- 1 With $r_{12}=0.7$, $r_{13}=0.5$, $r_{23}=0.6$, compute $R_{1\cdot23}$ and verify $1-R^2_{1\cdot23}=(1-r_{12}^2)(1-r_{13\cdot2}^2)$.
- 2 Show that $R_{1\cdot23}=0$ if and only if $\rho_{12}=\rho_{13}=0$.
- 3 Show that $R^2_{1\cdot23}=\rho_{12}^2+\rho_{13}^2$ when $\rho_{23}=0$.
- 4 Show that $R_{1\cdot23}$ is at least the larger of $|\rho_{12}|$ and $|\rho_{13}|$, directly from the formula of Theorem 7.18(4).
- 5 For $p$ equicorrelated variables with correlation $\rho\gt0$, show that $R^2_{1\cdot2\cdots p}=\dfrac{(p-1)\rho^2}{1+(p-2)\rho}$, and find its limit as $p\to\infty$.
- 6 Prove $R^2_{1\cdot2\cdots p}=1-\dfrac{1}{\sigma_{11}\omega_{11}}$ and evaluate it for the $\Sigma$ of Exercise I.3.
- 7 Show that the sample multiple correlation $R_{1\cdot2\cdots p}$ equals the sample correlation between $x_1$ and its least-squares fitted values $\hat x_1$.
- 1 $n=27$ pairs give $r=0.40$. Test $\rho=0$.
- 2 $n=30$ observations of $(X_1,X_2,X_3)$ give $r_{12\cdot3}=0.35$. Test $\rho_{12\cdot3}=0$.
- 3 The regression of $X_1$ on three other variables, from $n=25$ observations, has $R^2=0.30$. Test $R_{1\cdot234}=0$.
- 4 Find an approximate $95\%$ confidence interval for $\rho$ when $n=52$ and $r=0.6$.
- 5 Two independent samples give $r_1=0.5$ ($n_1=43$) and $r_2=0.3$ ($n_2=53$). Test $\rho_1=\rho_2$.
- 6 Under $R=0$, find $E[R^2]$ when $p-1=10$ predictors are used with $n=21$ observations, and relate this to the adjusted $\bar R^2$ of §6.3.
- 7 Show that $r\sqrt{n-2}/\sqrt{1-r^2}$ equals the $t$ statistic $\hat\beta_1/\operatorname{SE}(\hat\beta_1)$ of the simple regression of $y$ on $x$.
- 1 A sample of $n=15$ trivariate observations gives $T^2=12.0$ for $H_0:\mu=\mu_0$. Carry out the test.
- 2 Show that for $p=1$, $T^2=t^2$ and Theorem 7.20 reduces to $t^2\sim F_{1,n-1}$.
- 3 $n=25$, $\mu_0=(0,0)$, $\bar x=(0.4,0.5)$, $S=\begin{pmatrix}1&0.5\\0.5&1\end{pmatrix}$. Compute $T^2$ and test $H_0$; compare with the two $t$-tests.
- 4 Twelve patients have blood pressure and heart rate measured before and after a treatment. The differences have $\bar d=(1.2,\ 0.9)$ and $S_d=\begin{pmatrix}2&1\\1&1.5\end{pmatrix}$. Test for a treatment effect.
- 5 Prove the matrix determinant lemma $|A+uu^{\mathsf T}|=|A|(1+u^{\mathsf T}A^{-1}u)$ for $A\gt0$, and deduce $\Lambda^{2/n}=(1+T^2/(n-1))^{-1}$.
- 6 For the data of part 3, find the linear combination $a^{\mathsf T}\mu$ whose $t^2$ equals $T^2$.
- 7 For the data of part 3, compute the simultaneous $95\%$ $T^2$ intervals for $\mu_1$ and $\mu_2$ and the Bonferroni intervals, and comment.
- 8 Two samples of size $15$ in $\mathbb R^3$ give $T^2=10.5$. Test $\mu_1=\mu_2$.
- 9 With $n=30$ and $r_{12}=0.3$, $r_{13}=0.2$, $r_{23}=0.1$, test that the three variables are mutually independent using Bartlett's statistic.
E1. CSIR Part B GATE ST For three variables, $\rho_{12}=0.6$, $\rho_{13}=0.4$ and $\rho_{23}=0.5$. The partial correlation $\rho_{12\cdot3}$ is closest to (A) $0.40$; (B) $0.50$; (C) $0.60$; (D) $0.25$.
E2. CSIR Part B With the correlations of E1, the squared multiple correlation $R^2_{1\cdot23}$ equals (A) $\tfrac{7}{25}$; (B) $\tfrac{28}{75}$; (C) $\tfrac{13}{25}$; (D) $\tfrac35$.
E3. CSIR Part B A sample of size $21$ from $N_4(\mu,\Sigma)$ gives Hotelling's $T^2=20$ for $H_0:\mu=\mu_0$. The $F$ statistic used to test $H_0$ and its degrees of freedom are (A) $4.25$ on $(4,17)$; (B) $5$ on $(4,20)$; (C) $4.25$ on $(4,20)$; (D) $4.75$ on $(4,16)$.
E4. CSIR Part C $(X_1,X_2,X_3)$ is trivariate normal with unit variances, $\operatorname{Corr}(X_1,X_2)=\operatorname{Corr}(X_2,X_3)=\rho$ and $\operatorname{Corr}(X_1,X_3)=\rho^2$, where $0\lt|\rho|\lt1$. Which of the following are true? (A) $X_1$ and $X_3$ are independent. (B) $\rho_{13\cdot2}=0$. (C) $X_1$ and $X_3$ are conditionally independent given $X_2$. (D) $R^2_{1\cdot23}=\rho^2$, so $X_3$ adds nothing to the prediction of $X_1$ once $X_2$ is used.
E5. CSIR Part C Let $T^2$ be Hotelling's one-sample statistic for $H_0:\mu=\mu_0$ from a sample of size $n$ from $N_p(\mu,\Sigma)$, with $n\gt p+2$. Which of the following are true? (A) $T^2$ is unchanged if every observation $x$ is replaced by $Cx+b$ (and $\mu_0$ by $C\mu_0+b$) for any nonsingular $C$. (B) If one of the $p$ univariate $t$-tests at level $\alpha$ rejects, the $T^2$-test at level $\alpha$ also rejects. (C) The $T^2$-test is the likelihood ratio test of $H_0$. (D) Under $H_0$, $E[T^2]=p$.
E6. CSIR Part C GATE ST Let $x_1,\dots,x_n$ be a sample from $N_p(\mu,\Sigma)$, $p\ge3$, $n\gt p$, and let $R$ denote the sample multiple correlation of the first variable on the other $p-1$. Which of the following are true? (A) $R$ equals the sample correlation between the first variable and its least-squares fitted values. (B) If the population multiple correlation is $0$, then $R^2\sim\operatorname{Beta}\left(\frac{p-1}2,\frac{n-p}2\right)$. (C) Adding a further predictor can decrease $R$. (D) To test $\rho_{12\cdot3}=0$ the statistic $r_{12\cdot3}\sqrt{n-2}/\sqrt{1-r_{12\cdot3}^2}$ is referred to $t_{n-2}$.
Suggested plan — 3 lectures × 60 min.
- Partial correlation. Open with the reading–shoe-size story and ask the class to guess $r_{12\cdot3}$. Definition 7.6, Theorem 7.15 (residuals first, then the normal case), Theorem 7.16 and Example 1. Project Interactive 1 with the defaults, then set $\rho_{12}=0$, $\rho_{13}=\rho_{23}=0.6$ (Berkson) and $\rho_{12}=0.4$, $\rho_{13}=\rho_{23}=0.8$ (sign reversal). Theorem 7.17 and Exercise I.3 as a picture of a graphical model; Example 2.
- Multiple correlation and tests. Definition 7.7, Theorem 7.18 (Cauchy–Schwarz proof in full), Examples 3 and 4, then Theorem 7.19 by conditioning on the predictors — emphasise that the regression $F$-test of §6.1 does all the work. Example 5 and the strategy box.
- Hotelling's $T^2$. Motivate with the sports-drink data before any theory: two non-significant $t$-tests. Definition 7.9, Theorem 7.20 (the $U/V$ proof), Theorem 7.21, Example 6. Project Interactive 2: first $\delta=0$ (both procedures hold their size), then $\delta=(0.4,-0.4)$ with $\rho=0.7$, then $\delta=(0.4,0.4)$. Theorems 7.22–7.23, Example 7; finish with Theorems 7.24–7.25 briefly and E4–E5 live.
Discussion questions. (1) A newspaper reports that towns with more doctors have higher death rates. Which third variable would you partial out, and what sign do you expect for the partial correlation? (2) Why does $T^2$ reject in Example 6 when both $t$-tests do not, but fail to reject in Exercise IV.3 when one $t$-test does? Draw both situations as an ellipse and a box. (3) Is a correlation matrix with all off-diagonal entries $0.9$ possible for $p=10$ variables? What is the partial correlation of two of them given the other eight?
Misconceptions to watch for. Thinking that zero correlation implies zero partial correlation (or the reverse); treating $R^2$ as $\sum r_{1j}^2$; using $n-2$ degrees of freedom for a partial correlation; scaling $T^2$ by $(n-p)/p$ instead of $(n-p)/(p(n-1))$; using $n_1+n_2-2$ as the second degrees of freedom of the two-sample test; believing that $T^2$ must reject whenever some coordinate test rejects; applying $T^2$ with $n\le p$.
Suggested homework. I(1, 3, 6, 7), II(1, 4, 5, 7), III(1, 2, 4, 5), IV(1, 3, 4, 6, 8); Exam corner E1, E4, E5.
Research question. Which variables in a small environmental or financial system are directly linked, and which are linked only through others? Can the zeros of the inverse covariance matrix be found reliably from data?
Why it matters. Air-quality agencies study how ozone, temperature, wind speed and solar radiation are related; portfolio managers ask which stock indices move together after the common market factor is removed; systems biologists infer gene networks from expression data. In each case marginal correlations show a tangle in which everything is related to everything, while partial correlations (Theorem 7.17) separate direct links from indirect ones. The resulting graph is a Gaussian graphical model, a research field in its own right.
What to do.
- Choose a public data set with $p=4$ to $6$ continuous variables and $n\ge50$ observations — for example a daily air-quality record (ozone, temperature, wind, radiation) or daily returns of five national stock indices.
- Compute the correlation matrix $R$ and the matrix of partial correlations $-\omega_{ij}/\sqrt{\omega_{ii}\omega_{jj}}$ from $\hat\Omega=S^{-1}$. Make a table of both, side by side.
- Test each partial correlation given all the others with Theorem 7.19(2): $k=p-2$, so the $t$ statistic has $n-p$ degrees of freedom. Use a Bonferroni or Holm correction for the $p(p-1)/2$ tests.
- Draw two graphs on the $p$ variables: one with an edge for every significant marginal correlation, one for every significant partial correlation. Which edges disappear, and is there a plausible mechanism?
- Check one disappearing edge by regression: regress each of the two variables on the rest and correlate the residuals (Interactive 1 in your own data).
Tools: Python (numpy, scipy, pandas) or R; any graph-drawing tool, or paper.
Hand in: The two tables, the two graphs, and one paragraph interpreting a link that is present marginally but absent conditionally.
Going further. When $p$ is comparable to $n$, $S^{-1}$ is unstable or does not exist. Read about the graphical lasso, which estimates a sparse $\Omega$ by penalising $\sum_{i\ne j}|\omega_{ij}|$ in the normal likelihood, and try it on your data with a few penalty values.
🧪 Self-check quiz
Q1. If $\rho_{12}=0.5$, $\rho_{13}=0.5$ and $\rho_{23}=0.5$, then $\rho_{12\cdot3}$ equals
Q2. For a multivariate normal vector with precision matrix $\Omega=\Sigma^{-1}$, $\omega_{13}=0$ means
Q3. Which statement about the multiple correlation $R_{1\cdot2\cdots p}$ is always true?
Q4. For a one-sample Hotelling test with $n=12$ and $p=3$, $H_0$ is rejected at level $\alpha$ when
Q5. A sample partial correlation with $k$ variables eliminated, from $n$ normal observations, is tested with a $t$ statistic having