Chapter 6 · Linear Models and Regression

6.1 Gauss–Markov Models, Estimability and BLUE

The linear model $y=X\beta+\varepsilon$: least squares as orthogonal projection, normal equations and generalised inverses, which parametric functions can be estimated at all, the Gauss–Markov theorem with proof, the unbiased estimate of $\sigma^2$, exact distribution theory under normality, confidence intervals, and the $F$-test of a general linear hypothesis.

Almost every classical statistical method of the twentieth century is a special case of one model. A chemist calibrating an instrument fits a straight line; an agronomist compares four fertilisers on twenty plots; an economist explains wages by education, experience and region; a clinical trial adjusts a treatment effect for the patients' baseline values. In each case the expected response is a linear function of unknown parameters, $E[y]=X\beta$, where the matrix $X$ records the design: the values of the regressors, or zeros and ones saying which treatment each unit received. Gauss and Legendre fitted such models by least squares two hundred years ago. The questions that make the method a theory are newer. Which functions of $\beta$ can the data estimate at all, when $X$ does not have full column rank (as in every analysis of variance)? In what sense is least squares best? How is the error variance estimated, and what are the exact distributions needed for intervals and tests?

This section answers these questions in one framework, which the rest of the chapter uses. We show that least squares is the orthogonal projection of $y$ onto the column space of $X$, characterise the estimable functions $\lambda^{\mathsf T}\beta$ by the row space of $X$, and prove the Gauss–Markov theorem: the least squares estimator of an estimable function has the smallest variance among all linear unbiased estimators. Adding normality, we derive the $t$-intervals and the $F$-test of a general linear hypothesis $\Lambda^{\mathsf T}\beta=d$. Analysis of variance (§6.2), regression (§6.3) and the designs of Chapter 8 are applications. CSIR NET Part B questions ask for a variance, a degrees-of-freedom count or an $F$ value; Part C questions ask which functions are estimable, which statements about residuals and fitted values are true, and what Gauss–Markov does and does not assert. Quantiles are upper points as in §4.3: $t_{\nu,\alpha}$, $\chi^2_{\nu,\alpha}$ and $F_{\nu_1,\nu_2,\alpha}$ are exceeded with probability $\alpha$. Vectors are columns, $^{\mathsf T}$ denotes the transpose, $\mathcal C(A)$ the column space of a matrix $A$, $\mathbf 1$ a vector of ones and $\|v\|^2=v^{\mathsf T}v$.

The Linear Model and Least Squares

📘 Definition 6.1 — The Gauss–Markov linear model

Observations $y=(y_1,\dots,y_n)^{\mathsf T}$ follow the linear model $$y=X\beta+\varepsilon,$$ where $X$ is a known $n\times p$ design matrix of rank $r\le p$, $\beta\in\mathbb R^p$ is an unknown parameter vector and $\varepsilon$ is a random error vector. It is a Gauss–Markov model if $$E[\varepsilon]=0,\qquad\operatorname{Cov}(\varepsilon)=\sigma^2I_n,$$ with $\sigma^2\gt0$ unknown: the errors have mean zero, a common variance, and are uncorrelated. It is the normal linear model if, in addition, $\varepsilon\sim N_n(0,\sigma^2I_n)$, that is, $\varepsilon_1,\dots,\varepsilon_n$ are iid $N(0,\sigma^2)$. The model has full rank if $r=p$.

The word linear refers to the parameters, not to the regressors: $E[y_i]=\beta_0+\beta_1x_i+\beta_2x_i^2$ and $E[y_i]=\beta_0+\beta_1\log x_i$ are linear models. Three examples recur.

📘 Definition 6.2 — Least squares, normal equations, fitted values and residuals

A vector $\hat\beta$ is a least squares estimate of $\beta$ if it minimises $$S(b)=\|y-Xb\|^2=\sum_{i=1}^n\left(y_i-x_i^{\mathsf T}b\right)^2$$ over $b\in\mathbb R^p$, where $x_i^{\mathsf T}$ is the $i$-th row of $X$. The equations $$X^{\mathsf T}Xb=X^{\mathsf T}y$$ are the normal equations. For a least squares estimate, $\hat y=X\hat\beta$ is the vector of fitted values, $e=y-\hat y$ the vector of residuals and $\operatorname{RSS}=\|e\|^2$ the residual sum of squares.

📐 Theorem 6.1 — Least squares and the normal equations
  1. $\mathcal N(X^{\mathsf T}X)=\mathcal N(X)$, so $\operatorname{rank}(X^{\mathsf T}X)=\operatorname{rank}X=r$ and $\mathcal C(X^{\mathsf T}X)=\mathcal C(X^{\mathsf T})$.
  2. The normal equations are consistent for every $y$.
  3. $b$ minimises $S$ if and only if $b$ solves the normal equations. For any solution $\hat\beta$ and any $b$, $$S(b)=S(\hat\beta)+\|X(b-\hat\beta)\|^2.$$
  4. The fitted vector $\hat y=X\hat\beta$ and the residual vector $e$ are the same for every solution: $\hat y$ is the orthogonal projection of $y$ onto $\mathcal C(X)$, and $X^{\mathsf T}e=0$.
  5. If $r=p$, the solution is unique: $\hat\beta=(X^{\mathsf T}X)^{-1}X^{\mathsf T}y$.

Proof. (1) If $Xv=0$ then $X^{\mathsf T}Xv=0$. Conversely, if $X^{\mathsf T}Xv=0$ then $\|Xv\|^2=v^{\mathsf T}X^{\mathsf T}Xv=0$, so $Xv=0$. Equal null spaces give equal ranks by the rank–nullity theorem. Since $\mathcal C(X^{\mathsf T}X)\subseteq\mathcal C(X^{\mathsf T})$ and the two spaces have the same dimension $r$, they are equal.

(2) $X^{\mathsf T}y\in\mathcal C(X^{\mathsf T})=\mathcal C(X^{\mathsf T}X)$ by (1), so $X^{\mathsf T}Xb=X^{\mathsf T}y$ has a solution.

(3) Let $\hat\beta$ solve the normal equations and put $e=y-X\hat\beta$, so $X^{\mathsf T}e=0$. For any $b$, $$y-Xb=e+X(\hat\beta-b),$$ and the cross term $2(\hat\beta-b)^{\mathsf T}X^{\mathsf T}e$ vanishes. This gives the displayed identity, so $\hat\beta$ minimises $S$. If $b$ is another minimiser, then $\|X(b-\hat\beta)\|^2=0$, so $Xb=X\hat\beta$ and $X^{\mathsf T}Xb=X^{\mathsf T}X\hat\beta=X^{\mathsf T}y$: $b$ solves the normal equations.

(4) Two solutions $b,\hat\beta$ satisfy $X^{\mathsf T}X(b-\hat\beta)=0$, hence $X(b-\hat\beta)=0$ by (1), so the fitted vectors coincide. The relation $X^{\mathsf T}e=0$ says that $e$ is orthogonal to every column of $X$, so $y=\hat y+e$ with $\hat y\in\mathcal C(X)$ and $e\perp\mathcal C(X)$: this is the orthogonal decomposition of $y$, and $\hat y$ is the projection.

(5) If $r=p$, $X^{\mathsf T}X$ is a $p\times p$ matrix of rank $p$, hence invertible. ∎

When $r\lt p$ the normal equations have infinitely many solutions: if $\hat\beta$ is one, so is $\hat\beta+v$ for every $v\in\mathcal N(X)$. To write them down we need a substitute for the inverse.

📘 Definition 6.3 — Generalised inverse and the hat matrix

A generalised inverse (g-inverse) of a matrix $A$ is any matrix $A^-$ with $AA^-A=A$. The Moore–Penrose inverse $A^+$ is the unique g-inverse that also satisfies $A^+AA^+=A^+$ and makes $AA^+$ and $A^+A$ symmetric. Write $G=(X^{\mathsf T}X)^-$ for a g-inverse of $X^{\mathsf T}X$. The matrix $$H=X(X^{\mathsf T}X)^-X^{\mathsf T}$$ is the hat matrix (it "puts the hat on $y$": $\hat y=Hy$).

A g-inverse always exists. If $A$ has rank $r$ and is written as $A=P\begin{pmatrix}D&0\\0&0\end{pmatrix}Q$ with $P,Q$ invertible and $D$ an invertible $r\times r$ matrix, then $Q^{-1}\begin{pmatrix}D^{-1}&0\\0&0\end{pmatrix}P^{-1}$ is a g-inverse. In practice, for symmetric $A=X^{\mathsf T}X$: pick an invertible $r\times r$ principal submatrix, invert it, put the inverse in the same position and fill the rest with zeros. For any g-inverse $G$, $b=GX^{\mathsf T}y$ solves the normal equations, because $X^{\mathsf T}XGX^{\mathsf T}y=X^{\mathsf T}XGX^{\mathsf T}Xc=X^{\mathsf T}Xc=X^{\mathsf T}y$, where $c$ is any solution.

📐 Theorem 6.2 — Properties of the hat matrix

For any g-inverse $G$ of $X^{\mathsf T}X$:

  1. $XGX^{\mathsf T}X=X$ and $X^{\mathsf T}XGX^{\mathsf T}=X^{\mathsf T}$;
  2. $H=XGX^{\mathsf T}$ does not depend on the choice of $G$, and $H$ is symmetric and idempotent ($H^2=H$);
  3. $HX=X$, $\mathcal C(H)=\mathcal C(X)$ and $\operatorname{rank}H=\operatorname{tr}H=r$;
  4. $I-H$ is symmetric and idempotent, of rank $n-r$, $(I-H)X=0$, and $\hat y=Hy$, $e=(I-H)y$, $\operatorname{RSS}=y^{\mathsf T}(I-H)y$.

So $H$ is the orthogonal projection onto $\mathcal C(X)$ and $I-H$ the orthogonal projection onto its orthogonal complement.

Proof. (1) Put $B=XGX^{\mathsf T}X-X$. Then $X^{\mathsf T}B=X^{\mathsf T}XGX^{\mathsf T}X-X^{\mathsf T}X=0$. Since $B^{\mathsf T}=X^{\mathsf T}XG^{\mathsf T}X^{\mathsf T}-X^{\mathsf T}$, we get $B^{\mathsf T}B=X^{\mathsf T}XG^{\mathsf T}(X^{\mathsf T}B)-X^{\mathsf T}B=0$, and a matrix with $B^{\mathsf T}B=0$ is zero (its columns have zero length). The second identity is the transpose of the first applied to the g-inverse $G^{\mathsf T}$ (transposing $X^{\mathsf T}XGX^{\mathsf T}X=X^{\mathsf T}X$ shows that $G^{\mathsf T}$ is also a g-inverse of the symmetric matrix $X^{\mathsf T}X$).

(2) Let $G_1,G_2$ be g-inverses. By (1) applied to $G_2^{\mathsf T}$, $X^{\mathsf T}=X^{\mathsf T}XG_2^{\mathsf T}X^{\mathsf T}$. Hence $$XG_1X^{\mathsf T}=XG_1\left(X^{\mathsf T}XG_2^{\mathsf T}X^{\mathsf T}\right)=\left(XG_1X^{\mathsf T}X\right)G_2^{\mathsf T}X^{\mathsf T}=XG_2^{\mathsf T}X^{\mathsf T}.$$ With $G_1=G_2=G$ this says $XGX^{\mathsf T}=XG^{\mathsf T}X^{\mathsf T}=(XGX^{\mathsf T})^{\mathsf T}$, so $H$ is symmetric; then for general $G_1,G_2$, $XG_1X^{\mathsf T}=XG_2^{\mathsf T}X^{\mathsf T}=XG_2X^{\mathsf T}$, so $H$ does not depend on $G$. Idempotence: $H^2=(XGX^{\mathsf T}X)GX^{\mathsf T}=XGX^{\mathsf T}=H$ by (1).

(3) $HX=X$ is (1). Thus $\mathcal C(X)\subseteq\mathcal C(H)$, and $\mathcal C(H)\subseteq\mathcal C(X)$ because $H=X(GX^{\mathsf T})$. For an idempotent matrix the eigenvalues are $0$ or $1$ and, $H$ being symmetric, it is diagonalisable, so $\operatorname{tr}H$ is the number of unit eigenvalues, which is $\operatorname{rank}H=r$.

(4) $(I-H)^2=I-2H+H^2=I-H$, $\operatorname{tr}(I-H)=n-r$ and $(I-H)X=X-X=0$. Since $\hat\beta=GX^{\mathsf T}y$ is a solution, $\hat y=XGX^{\mathsf T}y=Hy$, and $\operatorname{RSS}=y^{\mathsf T}(I-H)^{\mathsf T}(I-H)y=y^{\mathsf T}(I-H)y$. A symmetric idempotent matrix $P$ is the orthogonal projection onto $\mathcal C(P)$, because $v=Pv+(I-P)v$ and $(Pv)^{\mathsf T}(I-P)w=v^{\mathsf T}(P-P^2)w=0$. ∎

For a full-rank model, $H=X(X^{\mathsf T}X)^{-1}X^{\mathsf T}$, and its diagonal entries $h_{ii}$, the leverages, satisfy $0\le h_{ii}\le1$ and $\sum h_{ii}=p$; §6.3 uses them in regression diagnostics. Theorem 6.2(4) gives the Pythagorean decomposition $$\|y\|^2=\|\hat y\|^2+\|e\|^2,$$ and, when $\mathbf 1\in\mathcal C(X)$ (the model has an intercept), $\sum e_i=\mathbf 1^{\mathsf T}e=0$ and $$\sum_i(y_i-\bar y)^2=\sum_i(\hat y_i-\bar y)^2+\sum_ie_i^2.$$

✏️ Example 1 — A straight line by least squares
A sensor gives the readings $y=(3,4,6,7,10)$ at the settings $x=(1,2,3,4,5)$. Fit $y_i=\beta_0+\beta_1x_i+\varepsilon_i$ by least squares; find the fitted values, the residuals, $\operatorname{RSS}$ and the leverages, and check $X^{\mathsf T}e=0$.
Normal equations $X^{\mathsf T}X=\begin{pmatrix}n&\sum x_i\\\sum x_i&\sum x_i^2\end{pmatrix}=\begin{pmatrix}5&15\\15&55\end{pmatrix}$ and $X^{\mathsf T}y=\begin{pmatrix}30\\107\end{pmatrix}$. Eliminating $\beta_0$ gives the familiar formulas $\hat\beta_1=S_{xy}/S_{xx}$, $\hat\beta_0=\bar y-\hat\beta_1\bar x$ with $S_{xx}=\sum(x_i-\bar x)^2=10$ and $S_{xy}=\sum(x_i-\bar x)(y_i-\bar y)=17$.
Estimates $\hat\beta_1=1.7$ and $\hat\beta_0=6-1.7\times3=0.9$.
Fit $\hat y=(2.6,\ 4.3,\ 6.0,\ 7.7,\ 9.4)$, $e=(0.4,\ -0.3,\ 0,\ -0.7,\ 0.6)$, $\operatorname{RSS}=0.16+0.09+0+0.49+0.36=1.10$.
Check $\sum e_i=0$ and $\sum x_ie_i=0.4-0.6+0-2.8+3.0=0$: the residuals are orthogonal to both columns of $X$. Also $\sum(y_i-\bar y)^2=30=28.9+1.1$, where $28.9=\hat\beta_1^2S_{xx}$ is the regression sum of squares.
Leverages For a straight line $h_{ii}=\frac1n+\frac{(x_i-\bar x)^2}{S_{xx}}$, so $h=(0.6,\ 0.3,\ 0.2,\ 0.3,\ 0.6)$, with sum $2=p$. The end points have the most influence on the fit.
✏️ Example 2 — A model that is not of full rank
Two groups are observed: group 1 gives $4,6$ and group 2 gives $7,8,9$. In the model $y_{ij}=\mu+\alpha_i+\varepsilon_{ij}$, write down the normal equations, find all their solutions, and show that the fitted values and the estimate of $\alpha_2-\alpha_1$ do not depend on the solution chosen. Compare the solution given by the g-inverse that inverts the lower $2\times2$ block with the Moore–Penrose solution.
Set-up With $\beta=(\mu,\alpha_1,\alpha_2)^{\mathsf T}$, $$X^{\mathsf T}X=\begin{pmatrix}5&2&3\\2&2&0\\3&0&3\end{pmatrix},\qquad X^{\mathsf T}y=\begin{pmatrix}34\\10\\24\end{pmatrix}.$$ The first row is the sum of the other two, so the rank is $2$.
All solutions The last two equations read $2\mu+2\alpha_1=10$ and $3\mu+3\alpha_2=24$; the first is their sum. So the solutions are $(\mu,\alpha_1,\alpha_2)=(t,\ 5-t,\ 8-t)$, $t\in\mathbb R$.
Two particular solutions The g-inverse $G=\begin{pmatrix}0&0&0\\0&1/2&0\\0&0&1/3\end{pmatrix}$ gives $GX^{\mathsf T}y=(0,5,8)$, the solution with $t=0$. The Moore–Penrose solution is the one of smallest length, orthogonal to $\mathcal N(X)=\operatorname{span}\{(1,-1,-1)\}$: $t-(5-t)-(8-t)=0$ gives $t=13/3$ and $(4.333,\ 0.667,\ 3.667)$.
Invariants Every solution has $\hat\mu+\hat\alpha_1=5=\bar y_1$ and $\hat\mu+\hat\alpha_2=8=\bar y_2$, so the fitted values are the group means and $\hat\alpha_2-\hat\alpha_1=3$. The residuals are $(-1,1,-1,0,1)$, $\operatorname{RSS}=4$. In contrast, "$\hat\alpha_1$" can be any number: by itself it estimates nothing. Theorem 6.3 explains why.
Interactive 1 — Least Squares Is an Orthogonal Projection

Three observations $y=(y_1,y_2,y_3)$ at $x=(0,1,2)$ and the model $E[y_i]=\beta_0+\beta_1x_i$. The shaded plane is $\mathcal C(X)$, spanned by $\mathbf 1=(1,1,1)$ and $x=(0,1,2)$. Move the sliders to change $y$ (black). The fitted vector $\hat y=Hy$ (blue) is the foot of the perpendicular from $y$ to the plane, and the residual $e=y-\hat y$ (red) meets the plane at a right angle, so $X^{\mathsf T}e=0$. Press Play to turn the picture and see the right angle from all sides.

0.53.51.5
β̂ = (β̂₀, β̂₁)—
ŷ = Hy—
e = y − ŷ—
Xᵀe—
‖y‖² = ‖ŷ‖² + ‖e‖²—

Estimability and the Gauss–Markov Theorem

Example 2 showed that in a model of less than full rank some functions of $\beta$, such as $\alpha_2-\alpha_1$, are determined by the data while others, such as $\alpha_1$, are not. The reason is not a shortage of data: even with infinitely many observations per group we would learn $\mu+\alpha_1$ and $\mu+\alpha_2$ and nothing more. The parameters $(\mu,\alpha_1,\alpha_2)$ and $(\mu+c,\alpha_1-c,\alpha_2-c)$ give the same distribution of $y$; they are not identifiable.

📘 Definition 6.4 — Estimable parametric function

A linear parametric function $\lambda^{\mathsf T}\beta$ ($\lambda\in\mathbb R^p$) is estimable if it has a linear unbiased estimator: there is $a\in\mathbb R^n$ with $E[a^{\mathsf T}y]=\lambda^{\mathsf T}\beta$ for every $\beta\in\mathbb R^p$. A vector $\Lambda^{\mathsf T}\beta$ ($\Lambda$ a $p\times q$ matrix) is estimable if each of its components is.

📐 Theorem 6.3 — Criteria for estimability

In the linear model $E[y]=X\beta$, the following are equivalent:

  1. $\lambda^{\mathsf T}\beta$ is estimable;
  2. $\lambda\in\mathcal C(X^{\mathsf T})$, i.e. $\lambda^{\mathsf T}$ is a linear combination of the rows of $X$: $\lambda=X^{\mathsf T}a$ for some $a$;
  3. $\lambda^{\mathsf T}GX^{\mathsf T}X=\lambda^{\mathsf T}$ for some (equivalently, every) g-inverse $G$ of $X^{\mathsf T}X$;
  4. $\lambda\perp\mathcal N(X)$: $\lambda^{\mathsf T}v=0$ whenever $Xv=0$;
  5. $\lambda^{\mathsf T}\beta_1=\lambda^{\mathsf T}\beta_2$ whenever $X\beta_1=X\beta_2$.

Consequently every component of $X\beta$ is estimable, linear combinations of estimable functions are estimable, all of $\beta$ is estimable if and only if $r=p$, and there are exactly $r$ linearly independent estimable functions.

Proof. (1)⇔(2): $E[a^{\mathsf T}y]=a^{\mathsf T}X\beta$, and $a^{\mathsf T}X\beta=\lambda^{\mathsf T}\beta$ for all $\beta$ exactly when $X^{\mathsf T}a=\lambda$.

(2)⇒(3): If $\lambda=X^{\mathsf T}a$, then $\lambda^{\mathsf T}GX^{\mathsf T}X=a^{\mathsf T}XGX^{\mathsf T}X=a^{\mathsf T}X=\lambda^{\mathsf T}$ by Theorem 6.2(1), for every $G$. (3)⇒(2): $\lambda=X^{\mathsf T}(XG^{\mathsf T}\lambda)$.

(2)⇔(4): $\mathcal C(X^{\mathsf T})$ is the orthogonal complement of $\mathcal N(X)$ in $\mathbb R^p$ (the row space is orthogonal to the null space, and their dimensions $r$ and $p-r$ add up to $p$).

(4)⇔(5): put $v=\beta_1-\beta_2$.

The consequences: the $i$-th component of $X\beta$ is $x_i^{\mathsf T}\beta$ and $x_i\in\mathcal C(X^{\mathsf T})$; $\mathcal C(X^{\mathsf T})$ is a subspace; it equals $\mathbb R^p$ iff $r=p$; and its dimension is $r$. ∎

Condition (5) is the identifiability statement: an estimable function is one that is determined by the mean vector $X\beta$. In the one-way model, $\mathcal N(X)$ is spanned by $(1,-1,\dots,-1)$, so by (4), $\lambda^{\mathsf T}\beta=\lambda_0\mu+\sum\lambda_i\alpha_i$ is estimable iff $\lambda_0=\sum_{i=1}^k\lambda_i$. Hence $\mu+\alpha_i$ is estimable, every contrast $\sum c_i\alpha_i$ with $\sum c_i=0$ is estimable, and $\mu$, $\alpha_i$ and $\alpha_1+\alpha_2$ are not. Imposing a side condition such as $\sum n_i\alpha_i=0$ or $\alpha_k=0$ picks one solution of the normal equations and makes every parameter "estimable" in the constrained model, but the meaning of $\alpha_i$ then depends on the constraint; estimable functions have the same estimate under every constraint.

📐 Theorem 6.4 — The Gauss–Markov theorem

In the Gauss–Markov model of Definition 6.1, let $\lambda^{\mathsf T}\beta$ be estimable and let $\hat\beta$ be any solution of the normal equations. Then:

  1. $\lambda^{\mathsf T}\hat\beta$ has the same value for every solution $\hat\beta$, and $\lambda^{\mathsf T}\hat\beta=\lambda^{\mathsf T}GX^{\mathsf T}y$ for every g-inverse $G$;
  2. $\lambda^{\mathsf T}\hat\beta$ is unbiased for $\lambda^{\mathsf T}\beta$, with $\operatorname{Var}(\lambda^{\mathsf T}\hat\beta)=\sigma^2\lambda^{\mathsf T}G\lambda$ (also independent of $G$);
  3. if $a^{\mathsf T}y$ is any linear unbiased estimator of $\lambda^{\mathsf T}\beta$, then $\operatorname{Var}(a^{\mathsf T}y)\ge\operatorname{Var}(\lambda^{\mathsf T}\hat\beta)$, with equality if and only if $a^{\mathsf T}y=\lambda^{\mathsf T}\hat\beta$ (i.e. $a=XG\lambda$).

So $\lambda^{\mathsf T}\hat\beta$ is the unique best linear unbiased estimator (BLUE) of $\lambda^{\mathsf T}\beta$. For estimable $\lambda_1^{\mathsf T}\beta,\lambda_2^{\mathsf T}\beta$, $\operatorname{Cov}(\lambda_1^{\mathsf T}\hat\beta,\lambda_2^{\mathsf T}\hat\beta)=\sigma^2\lambda_1^{\mathsf T}G\lambda_2$.

Proof. Write $\lambda=X^{\mathsf T}a_0$ (Theorem 6.3). (1) $\lambda^{\mathsf T}\hat\beta=a_0^{\mathsf T}X\hat\beta=a_0^{\mathsf T}\hat y$, and $\hat y$ is the same for every solution (Theorem 6.1(4)). Since $GX^{\mathsf T}y$ is a solution, $\lambda^{\mathsf T}\hat\beta=\lambda^{\mathsf T}GX^{\mathsf T}y$.

(3) first. A linear estimator $a^{\mathsf T}y$ is unbiased iff $X^{\mathsf T}a=\lambda$, and then $\operatorname{Var}(a^{\mathsf T}y)=a^{\mathsf T}(\sigma^2I)a=\sigma^2\|a\|^2$. Split $a=Ha+(I-H)a$, an orthogonal decomposition. The first part is the same for every unbiased $a$: $$Ha=XGX^{\mathsf T}a=XG\lambda=:a^*.$$ Therefore $$\|a\|^2=\|a^*\|^2+\|(I-H)a\|^2\ge\|a^*\|^2,$$ with equality iff $(I-H)a=0$, i.e. $a=a^*$. And $a^*$ itself is unbiased ($X^{\mathsf T}a^*=X^{\mathsf T}Ha=X^{\mathsf T}a=\lambda$, using $X^{\mathsf T}H=X^{\mathsf T}$), with $$a^{*\mathsf T}y=\lambda^{\mathsf T}G^{\mathsf T}X^{\mathsf T}y=\lambda^{\mathsf T}\hat\beta,$$ because $G^{\mathsf T}$ is a g-inverse too and (1) holds for every g-inverse. So the minimum variance is attained by $\lambda^{\mathsf T}\hat\beta$ and only by it.

(2) Unbiasedness: $$E[\lambda^{\mathsf T}\hat\beta]=a_0^{\mathsf T}E[\hat y]=a_0^{\mathsf T}HX\beta=a_0^{\mathsf T}X\beta=\lambda^{\mathsf T}\beta.$$ Variance, using $a^*=Ha_0$ (take $a=a_0$ above): $$\begin{aligned}\sigma^2\|a^*\|^2&=\sigma^2a_0^{\mathsf T}H^{\mathsf T}Ha_0=\sigma^2a_0^{\mathsf T}Ha_0\\&=\sigma^2a_0^{\mathsf T}XGX^{\mathsf T}a_0=\sigma^2\lambda^{\mathsf T}G\lambda.\end{aligned}$$ The covariance formula follows in the same way from $\operatorname{Cov}(a_1^{*\mathsf T}y,a_2^{*\mathsf T}y)=\sigma^2a_1^{*\mathsf T}a_2^*$. ∎

The proof also gives a practical test: an unbiased linear estimator $a^{\mathsf T}y$ is the BLUE if and only if $a\in\mathcal C(X)$ (then $a=Ha=a^*$). For example, in any model whose design matrix has the group indicators among its columns (or in its column space), a difference of group means is a BLUE as soon as it is unbiased.

Three remarks. First, the theorem needs only the first two moments of $\varepsilon$; no normality and no independence beyond zero correlation. Second, the competition is restricted to linear unbiased estimators: a biased estimator (such as ridge regression) or a nonlinear one can have a smaller mean squared error. Third, the theorem has a matrix form: if $\Lambda^{\mathsf T}\beta$ is estimable and $T=A^{\mathsf T}y$ is any linear unbiased estimator of it, then $\operatorname{Cov}(T)-\operatorname{Cov}(\Lambda^{\mathsf T}\hat\beta)$ is positive semidefinite, because applying (3) to $c^{\mathsf T}\Lambda^{\mathsf T}\beta$ gives $c^{\mathsf T}\operatorname{Cov}(T)c\ge c^{\mathsf T}\operatorname{Cov}(\Lambda^{\mathsf T}\hat\beta)c$ for every $c$. In the full-rank case, $\operatorname{Cov}(\hat\beta)=\sigma^2(X^{\mathsf T}X)^{-1}$.

When the errors are correlated or have unequal variances, least squares remains unbiased but is no longer best. The remedy is to transform the model back to a Gauss–Markov model.

📐 Theorem 6.5 — Aitken's theorem (generalised least squares)

Let $y=X\beta+\varepsilon$ with $X$ of full rank $p$, $E[\varepsilon]=0$ and $\operatorname{Cov}(\varepsilon)=\sigma^2V$, where $V$ is a known positive definite matrix. Then the BLUE of $\beta$ is the generalised least squares estimator $$\tilde\beta=(X^{\mathsf T}V^{-1}X)^{-1}X^{\mathsf T}V^{-1}y,$$ $$\operatorname{Cov}(\tilde\beta)=\sigma^2(X^{\mathsf T}V^{-1}X)^{-1},$$ which minimises $(y-Xb)^{\mathsf T}V^{-1}(y-Xb)$. For $V=\operatorname{diag}(v_1,\dots,v_n)$ this is weighted least squares with weights $1/v_i$.

Proof. Let $V^{1/2}$ be the symmetric positive definite square root of $V$ and $V^{-1/2}$ its inverse. Then $z=V^{-1/2}y=V^{-1/2}X\beta+V^{-1/2}\varepsilon$ is a Gauss–Markov model with design $W=V^{-1/2}X$ (of rank $p$) and $\operatorname{Cov}(V^{-1/2}\varepsilon)=\sigma^2V^{-1/2}VV^{-1/2}=\sigma^2I$. Linear estimators based on $z$ and on $y$ are the same class, because $V^{-1/2}$ is invertible. By Theorem 6.4 the BLUE is $(W^{\mathsf T}W)^{-1}W^{\mathsf T}z=(X^{\mathsf T}V^{-1}X)^{-1}X^{\mathsf T}V^{-1}y$, with covariance $\sigma^2(W^{\mathsf T}W)^{-1}$, and it minimises $\|z-Wb\|^2=(y-Xb)^{\mathsf T}V^{-1}(y-Xb)$. ∎

✏️ Example 3 — Estimable or not? An incomplete two-way layout
In the additive model $y_{ij}=\mu+\alpha_i+\beta_j+\varepsilon_{ij}$ ($i,j\in\{1,2\}$), one observation is taken in each of the cells $(1,1)$, $(1,2)$ and $(2,1)$; cell $(2,2)$ is empty. (a) Decide whether $\alpha_1-\alpha_2$, $\beta_1-\beta_2$, $\alpha_1$ and the mean $\mu+\alpha_2+\beta_2$ of the empty cell are estimable, and give the BLUE where it exists. (b) If instead only cells $(1,1)$ and $(2,2)$ are observed, is $\alpha_1-\alpha_2$ estimable?
Rows of $X$ With $\beta=(\mu,\alpha_1,\alpha_2,\beta_1,\beta_2)^{\mathsf T}$ the rows are $r_{11}=(1,1,0,1,0)$, $r_{12}=(1,1,0,0,1)$ and $r_{21}=(1,0,1,1,0)$; they are independent, so $r=3$ and $\mathcal N(X)$ has dimension $2$. By Theorem 6.3, a function is estimable iff its coefficient vector is a combination of these rows.
(a) Estimable $r_{11}-r_{21}=(0,1,-1,0,0)$: $\alpha_1-\alpha_2$ is estimable with BLUE $y_{11}-y_{21}$. $r_{11}-r_{12}=(0,0,0,1,-1)$: $\beta_1-\beta_2$ is estimable with BLUE $y_{11}-y_{12}$. And $r_{12}+r_{21}-r_{11}=(1,0,1,0,1)$: the empty cell's mean is estimable, with BLUE $y_{12}+y_{21}-y_{11}$. With three observations and rank three, each $y$ is its own fitted value and these combinations are the unique unbiased estimators.
(a) Not estimable $\mathcal N(X)$ contains $v=(1,-1,-1,0,0)$ (check: $Xv=(1-1,\ 1-1,\ 1-1)=0$). Then $\lambda=(0,1,0,0,0)$ has $\lambda^{\mathsf T}v=-1\ne0$, so $\alpha_1$ is not estimable by criterion (4).
(b) The rows are $(1,1,0,1,0)$ and $(1,0,1,0,1)$. A combination $a(1,1,0,1,0)+b(1,0,1,0,1)$ equals $(0,1,-1,0,0)$ only if $a=1$, $b=-1$, which gives $(0,1,-1,1,-1)$: no. So $\alpha_1-\alpha_2$ is not estimable; only $(\alpha_1+\beta_1)-(\alpha_2+\beta_2)$ is. Row and column effects are confounded: the design is disconnected (Chapter 8 makes this notion precise).
✏️ Example 4 — Least squares against another unbiased slope
For the design $x=(1,2,3,4,5)$ of Example 1, the end-point slope $T=(y_5-y_1)/4$ is an unbiased estimator of $\beta_1$. Verify this, compute its variance and its efficiency relative to $\hat\beta_1$, and exhibit the orthogonal decomposition used in the proof of the Gauss–Markov theorem.
Unbiased $T=a^{\mathsf T}y$ with $a=(-\tfrac14,0,0,0,\tfrac14)$. Unbiasedness for $\beta_1$ means $X^{\mathsf T}a=(0,1)^{\mathsf T}$: $\sum a_i=0$ and $\sum a_ix_i=(5-1)/4=1$. ✓
Variances $\operatorname{Var}(T)=\sigma^2\|a\|^2=\sigma^2/8$. The BLUE has $a^*=(x_i-\bar x)/S_{xx}=(-0.2,-0.1,0,0.1,0.2)$ and $\operatorname{Var}(\hat\beta_1)=\sigma^2/S_{xx}=\sigma^2/10$. The efficiency of $T$ is $\frac{1/10}{1/8}=0.8$.
Decomposition $a-a^*=(-0.05,\ 0.1,\ 0,\ -0.1,\ 0.05)$. It is orthogonal to $\mathcal C(X)$: its entries sum to $0$ and $\sum x_i(a_i-a_i^*)=-0.05+0.2-0.4+0.25=0$. So $\|a\|^2=\|a^*\|^2+\|a-a^*\|^2$, that is, $0.125=0.1+0.025$. ✓
Interpretation Any unbiased linear estimator is the BLUE plus "noise" $(a-a^*)^{\mathsf T}y$ that is uncorrelated with it and has mean zero. Adding noise can only increase the variance. Interactive 2 shows this by simulation, and also what happens when the errors are not homoscedastic.
Interactive 2 — Gauss–Markov by Simulation: Three Unbiased Slopes

Data $y_i=2+0.5x_i+\varepsilon_i$ at $x=1,\dots,10$ are simulated again and again (seeded generator), and three linear unbiased estimators of the slope are computed each time: least squares $\hat\beta_1$ (blue), the grouping estimator $(\bar y_{\text{upper half}}-\bar y_{\text{lower half}})/5$ (green) and the end-point slope $(y_{10}-y_1)/9$ (red). All three histograms are centred at the true slope $0.5$ (dashed); the curves are normal densities with the theoretical variances $\sigma^2\|a\|^2$. Change the error law: with uncorrelated, equal-variance errors the least squares histogram is always the narrowest, whatever the shape of the errors. With heteroscedastic errors that is no longer guaranteed, and weighted least squares (Theorem 6.5) does better.

samples—
least squares: var (theory)—
grouping: var (theory)—
end points: var (theory)—

Estimating $\sigma^2$ and Distribution Theory under Normality

📐 Theorem 6.6 — Unbiased estimation of $\sigma^2$
  1. (Expectation of a quadratic form.) If $E[y]=\mu$, $\operatorname{Cov}(y)=\Sigma$ and $A$ is a constant symmetric matrix, then $E[y^{\mathsf T}Ay]=\operatorname{tr}(A\Sigma)+\mu^{\mathsf T}A\mu$.
  2. In the Gauss–Markov model, $E[\operatorname{RSS}]=(n-r)\sigma^2$, so $$\hat\sigma^2=\frac{\operatorname{RSS}}{n-r}$$ is an unbiased estimator of $\sigma^2$ (for $n\gt r$).
  3. $\operatorname{Cov}(\hat y,e)=0$, $\operatorname{Cov}(e)=\sigma^2(I-H)$; in particular $\operatorname{Var}(e_i)=\sigma^2(1-h_{ii})$.

Proof. (1) $y^{\mathsf T}Ay$ is a scalar, so it equals its trace, and $\operatorname{tr}(y^{\mathsf T}Ay)=\operatorname{tr}(Ayy^{\mathsf T})$. Taking expectations, $E[yy^{\mathsf T}]=\Sigma+\mu\mu^{\mathsf T}$, so $E[y^{\mathsf T}Ay]=\operatorname{tr}(A\Sigma)+\operatorname{tr}(A\mu\mu^{\mathsf T})=\operatorname{tr}(A\Sigma)+\mu^{\mathsf T}A\mu$.

(2) $\operatorname{RSS}=y^{\mathsf T}(I-H)y$ with $\mu=X\beta$ and $\Sigma=\sigma^2I$. Since $(I-H)X=0$, the second term vanishes, and $E[\operatorname{RSS}]=\sigma^2\operatorname{tr}(I-H)=(n-r)\sigma^2$ by Theorem 6.2.

(3) Using $H^{\mathsf T}=H=H^2$: $$\operatorname{Cov}(Hy,(I-H)y)=\sigma^2H(I-H)=\sigma^2(H-H^2)=0,$$ $$\operatorname{Cov}((I-H)y)=\sigma^2(I-H)^2=\sigma^2(I-H).\ ∎$$

The divisor is $n-r$, the residual degrees of freedom: $e$ lives in the $(n-r)$-dimensional space $\mathcal C(X)^\perp$. In Example 1, $\hat\sigma^2=1.10/3=0.3667$. Part (3) also warns that residuals are correlated and have unequal variances even when the errors are iid. Under normality much more is true.

📐 Theorem 6.7 — Distribution theory in the normal linear model

Let $y\sim N_n(X\beta,\sigma^2I)$ with $\operatorname{rank}X=r\lt n$, and let $\lambda^{\mathsf T}\beta$ be estimable. Then:

  1. $\hat y$ and $e$ are independent, hence $\lambda^{\mathsf T}\hat\beta$ and $\operatorname{RSS}$ are independent;
  2. $\lambda^{\mathsf T}\hat\beta\sim N\left(\lambda^{\mathsf T}\beta,\ \sigma^2\lambda^{\mathsf T}G\lambda\right)$, and for estimable $\Lambda^{\mathsf T}\beta$, $\Lambda^{\mathsf T}\hat\beta\sim N_q(\Lambda^{\mathsf T}\beta,\sigma^2\Lambda^{\mathsf T}G\Lambda)$;
  3. $\operatorname{RSS}/\sigma^2\sim\chi^2_{n-r}$;
  4. (full rank) the maximum likelihood estimators are $\hat\beta$ and $\hat\sigma^2_{\text{ML}}=\operatorname{RSS}/n$, and $(\hat\beta,\operatorname{RSS})$ is a complete sufficient statistic, so $\lambda^{\mathsf T}\hat\beta$ and $\operatorname{RSS}/(n-p)$ are UMVUEs.

Proof. Choose an orthonormal basis $q_1,\dots,q_r$ of $\mathcal C(X)$ and extend it by $q_{r+1},\dots,q_n$ to an orthonormal basis of $\mathbb R^n$; let $Q$ be the orthogonal matrix with columns $q_i$. Put $z=Q^{\mathsf T}y$. Because an orthogonal transformation maps $N_n(\mu,\sigma^2I)$ to $N_n(Q^{\mathsf T}\mu,\sigma^2Q^{\mathsf T}Q)=N_n(Q^{\mathsf T}\mu,\sigma^2I)$ (the same rotation invariance that gives the Helmert transformation in §4.1), the $z_i$ are independent $N(q_i^{\mathsf T}X\beta,\sigma^2)$. For $i\gt r$, $q_i\perp\mathcal C(X)$, so $E[z_i]=0$.

Now $y=\sum_iz_iq_i$, the projection onto $\mathcal C(X)$ keeps the first $r$ terms and $I-H$ keeps the rest: $$\hat y=\sum_{i=1}^rz_iq_i,\qquad e=\sum_{i=r+1}^nz_iq_i,\qquad\operatorname{RSS}=\sum_{i=r+1}^nz_i^2.$$ So $\hat y$ and $e$ are functions of disjoint sets of independent variables, which proves (1) (since $\lambda^{\mathsf T}\hat\beta=a_0^{\mathsf T}\hat y$). And $\operatorname{RSS}/\sigma^2$ is a sum of $n-r$ squares of independent $N(0,1)$ variables, which is (3). For (2), $\lambda^{\mathsf T}\hat\beta=\lambda^{\mathsf T}GX^{\mathsf T}y$ is a linear function of a normal vector, hence normal, with the mean and variance of Theorem 6.4; the vector case is the same.

(4) The log-likelihood is $-\frac n2\log(2\pi\sigma^2)-\frac{1}{2\sigma^2}\|y-X\beta\|^2$. For every $\sigma^2$ it is maximised in $\beta$ by the least squares estimate, and then $\frac{d}{d\sigma^2}$ of $-\frac n2\log\sigma^2-\frac{\operatorname{RSS}}{2\sigma^2}$ vanishes at $\sigma^2=\operatorname{RSS}/n$. Expanding $\|y-X\beta\|^2=y^{\mathsf T}y-2\beta^{\mathsf T}X^{\mathsf T}y+\beta^{\mathsf T}X^{\mathsf T}X\beta$ shows a full-rank exponential family in $(X^{\mathsf T}y,y^{\mathsf T}y)$, whose natural parameter $(\beta/\sigma^2,-1/(2\sigma^2))$ ranges over an open set; so $(X^{\mathsf T}y,y^{\mathsf T}y)$, equivalently $(\hat\beta,\operatorname{RSS})$, is complete and sufficient (§4.2). Unbiased functions of it are UMVUEs by the Lehmann–Scheffé theorem. ∎

📐 Theorem 6.8 — Confidence intervals in the normal linear model

Under the hypotheses of Theorem 6.7, with $\hat\sigma^2=\operatorname{RSS}/(n-r)$:

  1. $\dfrac{\lambda^{\mathsf T}\hat\beta-\lambda^{\mathsf T}\beta}{\hat\sigma\sqrt{\lambda^{\mathsf T}G\lambda}}\sim t_{n-r}$, so $\lambda^{\mathsf T}\hat\beta\pm t_{n-r,\alpha/2}\,\hat\sigma\sqrt{\lambda^{\mathsf T}G\lambda}$ is a $100(1-\alpha)\%$ confidence interval for $\lambda^{\mathsf T}\beta$;
  2. $\left[\dfrac{\operatorname{RSS}}{\chi^2_{n-r,\alpha/2}},\ \dfrac{\operatorname{RSS}}{\chi^2_{n-r,1-\alpha/2}}\right]$ is a $100(1-\alpha)\%$ confidence interval for $\sigma^2$;
  3. for estimable $\Lambda^{\mathsf T}\beta$ with $\Lambda$ of rank $q$, writing $u=\Lambda^{\mathsf T}\hat\beta-\theta$, the set $$\left\{\theta\in\mathbb R^q:\ u^{\mathsf T}(\Lambda^{\mathsf T}G\Lambda)^{-1}u\le q\hat\sigma^2F_{q,n-r,\alpha}\right\}$$ is a $100(1-\alpha)\%$ confidence ellipsoid for $\Lambda^{\mathsf T}\beta$.

Proof. (1) The numerator divided by $\sigma\sqrt{\lambda^{\mathsf T}G\lambda}$ is $N(0,1)$, $(n-r)\hat\sigma^2/\sigma^2\sim\chi^2_{n-r}$, and they are independent (Theorem 6.7). The ratio is $t_{n-r}$ by the definition of the $t$ law (§4.1), and the interval follows by the pivotal method (§4.3). (2) is the pivotal method with the pivot $\operatorname{RSS}/\sigma^2\sim\chi^2_{n-r}$. (3) The quadratic form divided by $\sigma^2$ is $\chi^2_q$ (proof of Theorem 6.9 below) and independent of $\operatorname{RSS}$, so the form divided by $q\hat\sigma^2$ is $F_{q,n-r}$. ∎

For the straight line, $\operatorname{Var}(\hat\beta_1)=\sigma^2/S_{xx}$, $\operatorname{Var}(\hat\beta_0)=\sigma^2\left(\frac1n+\frac{\bar x^2}{S_{xx}}\right)$, $\operatorname{Cov}(\hat\beta_0,\hat\beta_1)=-\bar x\sigma^2/S_{xx}$, and the mean response $\beta_0+\beta_1x_0$ has BLUE $\hat\beta_0+\hat\beta_1x_0$ with variance $\sigma^2\left(\frac1n+\frac{(x_0-\bar x)^2}{S_{xx}}\right)$. A new observation at $x_0$ is predicted with error variance $\sigma^2\left(1+\frac1n+\frac{(x_0-\bar x)^2}{S_{xx}}\right)$; §6.3 develops these prediction intervals.

✏️ Example 5 — Intervals for the sensor line
For the data of Example 1 (normal errors assumed), find $95\%$ confidence intervals for the slope $\beta_1$, for the mean reading at $x_0=4$ and for $\sigma^2$.
Ingredients $n-r=3$, $\hat\sigma^2=0.3667$, $\hat\sigma=0.6055$, $t_{3,0.025}=3.182$.
Slope $\operatorname{SE}(\hat\beta_1)=\sqrt{0.3667/10}=0.1915$, so $1.7\pm3.182\times0.1915=1.7\pm0.609$, i.e. $(1.09,\ 2.31)$. Zero is far outside: the slope is clearly positive ($t=8.88$, two-sided $p=0.003$).
Mean at $x_0=4$ $\hat\beta_0+4\hat\beta_1=7.7$ with $\operatorname{SE}=\sqrt{0.3667\left(\frac15+\frac1{10}\right)}=0.3317$, so $7.7\pm1.055$, i.e. $(6.64,\ 8.76)$.
Variance $\chi^2_{3,0.025}=9.348$ and $\chi^2_{3,0.975}=0.2158$: $\left[\frac{1.1}{9.348},\frac{1.1}{0.2158}\right]=(0.118,\ 5.10)$. With only $3$ degrees of freedom the interval for $\sigma^2$ is very wide, a ratio of about $43$ between its ends.

Testing Linear Hypotheses

📘 Definition 6.5 — Testable linear hypothesis

A linear hypothesis is $H_0:\Lambda^{\mathsf T}\beta=d$, where $\Lambda$ is a known $p\times q$ matrix of rank $q$ and $d\in\mathbb R^q$ is known. It is testable if $\Lambda^{\mathsf T}\beta$ is estimable, i.e. every column of $\Lambda$ lies in $\mathcal C(X^{\mathsf T})$; then $q\le r$.

Testability matters because a hypothesis about a non-estimable function is not a statement about the distribution of $y$. In the one-way model "$\alpha_1=0$" can be made true or false by changing the side condition, without changing $X\beta$; but "$\alpha_1=\alpha_2=\cdots=\alpha_k$" is the testable statement that all group means are equal ($q=k-1$ contrasts).

📐 Theorem 6.9 — The $F$-test of a general linear hypothesis

In the normal linear model with $\operatorname{rank}X=r\lt n$, let $H_0:\Lambda^{\mathsf T}\beta=d$ be testable, of rank $q$. Let $\operatorname{RSS}$ be the unrestricted residual sum of squares and $\operatorname{RSS}_0=\min\{\|y-Xb\|^2:\Lambda^{\mathsf T}b=d\}$ the restricted one. Then:

  1. $\Lambda^{\mathsf T}G\Lambda$ is nonsingular and $$\operatorname{RSS}_0-\operatorname{RSS}=(\Lambda^{\mathsf T}\hat\beta-d)^{\mathsf T}(\Lambda^{\mathsf T}G\Lambda)^{-1}(\Lambda^{\mathsf T}\hat\beta-d);$$
  2. the statistic $$F=\frac{(\operatorname{RSS}_0-\operatorname{RSS})/q}{\operatorname{RSS}/(n-r)}$$ has the $F_{q,n-r}$ distribution under $H_0$, and the size-$\alpha$ test rejects when $F\gt F_{q,n-r,\alpha}$;
  3. in general $F$ has the noncentral $F$ law $F_{q,n-r}(\delta)$ with noncentrality $\delta=(\Lambda^{\mathsf T}\beta-d)^{\mathsf T}(\Lambda^{\mathsf T}G\Lambda)^{-1}(\Lambda^{\mathsf T}\beta-d)/\sigma^2$, and the power increases with $\delta$;
  4. the test is the likelihood ratio test: $\Lambda(y)=(\operatorname{RSS}/\operatorname{RSS}_0)^{n/2}$ is a decreasing function of $F$.

For $q=1$, $H_0:\lambda^{\mathsf T}\beta=d$, $F=t^2$ with $t=(\lambda^{\mathsf T}\hat\beta-d)/(\hat\sigma\sqrt{\lambda^{\mathsf T}G\lambda})$.

Proof. We give the proof for a full-rank model, $G=(X^{\mathsf T}X)^{-1}$; the general case reduces to it by reparametrising with $r$ independent estimable functions. Put $W=\Lambda^{\mathsf T}(X^{\mathsf T}X)^{-1}\Lambda$, which is positive definite since $\Lambda$ has rank $q$ and $(X^{\mathsf T}X)^{-1}$ is positive definite.

(1) Minimise $\|y-Xb\|^2$ subject to $\Lambda^{\mathsf T}b=d$ with a Lagrange multiplier $2\gamma$: $X^{\mathsf T}Xb-X^{\mathsf T}y+\Lambda\gamma=0$, so $b=\hat\beta-(X^{\mathsf T}X)^{-1}\Lambda\gamma$, and the constraint gives $\Lambda^{\mathsf T}\hat\beta-W\gamma=d$. Hence $$\tilde\beta=\hat\beta-(X^{\mathsf T}X)^{-1}\Lambda W^{-1}(\Lambda^{\mathsf T}\hat\beta-d).$$ By Theorem 6.1(3), $\operatorname{RSS}_0=\|y-X\tilde\beta\|^2=\operatorname{RSS}+\|X(\hat\beta-\tilde\beta)\|^2$. Write $u=\Lambda^{\mathsf T}\hat\beta-d$ and $M=(X^{\mathsf T}X)^{-1}$. Then $X(\hat\beta-\tilde\beta)=XM\Lambda W^{-1}u$, and $$\|X(\hat\beta-\tilde\beta)\|^2=u^{\mathsf T}W^{-1}\Lambda^{\mathsf T}MX^{\mathsf T}XM\Lambda W^{-1}u,$$ which equals $u^{\mathsf T}W^{-1}WW^{-1}u=u^{\mathsf T}W^{-1}u$, since $MX^{\mathsf T}XM=M$ and $\Lambda^{\mathsf T}M\Lambda=W$. (The minimum is attained because $S$ is a convex quadratic and the identity $S(b)=\operatorname{RSS}+\|X(b-\hat\beta)\|^2$ shows that $\tilde\beta$, the point of the affine set closest to $\hat\beta$ in the $X^{\mathsf T}X$ metric, minimises it.)

(2), (3) By Theorem 6.7, $u=\Lambda^{\mathsf T}\hat\beta-d\sim N_q(\Lambda^{\mathsf T}\beta-d,\sigma^2W)$. Then $W^{-1/2}u/\sigma\sim N_q(W^{-1/2}(\Lambda^{\mathsf T}\beta-d)/\sigma,I_q)$, so $u^{\mathsf T}W^{-1}u/\sigma^2=\|W^{-1/2}u/\sigma\|^2$ is a sum of $q$ squares of independent unit-variance normals: $\chi^2_q$ under $H_0$, and noncentral $\chi^2_q(\delta)$ in general. It is a function of $\hat\beta$, hence independent of $\operatorname{RSS}/\sigma^2\sim\chi^2_{n-r}$. The ratio of independent $\chi^2$ variables divided by their degrees of freedom is $F_{q,n-r}$ (noncentral when $\delta\gt0$). A noncentral $\chi^2_q(\delta)$ is stochastically increasing in $\delta$, so the power increases with $\delta$.

(4) Maximising the normal likelihood with $\sigma^2$ free gives $\sup L=(2\pi e\,\operatorname{RSS}/n)^{-n/2}$ (Theorem 6.7(4)), and under $H_0$ the same with $\operatorname{RSS}_0$. So $\Lambda=(\operatorname{RSS}/\operatorname{RSS}_0)^{n/2}=\left(1+\frac{q}{n-r}F\right)^{-n/2}$, a decreasing function of $F$; rejecting for small $\Lambda$ is rejecting for large $F$ (as for the normal-theory tests in §5.1). For $q=1$, $W=\lambda^{\mathsf T}G\lambda$ and $F=u^2/(W\hat\sigma^2)=t^2$. ∎

🧭 Problem-solving strategy — testing a linear hypothesis
  1. Check testability. Each row of $\Lambda^{\mathsf T}$ must be a combination of rows of $X$; count the number $q$ of linearly independent constraints.
  2. Full model. Fit by least squares: $\operatorname{RSS}$ on $n-r$ degrees of freedom.
  3. Reduced model. Either substitute the constraints into the model and refit, giving $\operatorname{RSS}_0$ on $n-r+q$ degrees of freedom, or compute $\operatorname{RSS}_0-\operatorname{RSS}$ by the quadratic form of Theorem 6.9(1). For one constraint, use $t$.
  4. Decide. $F=\dfrac{(\operatorname{RSS}_0-\operatorname{RSS})/q}{\operatorname{RSS}/(n-r)}$ against $F_{q,n-r,\alpha}$. The numerator degrees of freedom are the difference of the residual degrees of freedom of the two models.
✏️ Example 6 — Two tests in a replicated $2^2$ design
A process is run twice at each combination of two coded factors $x_1,x_2\in\{-1,+1\}$. In the order $(x_1,x_2)=(-,-),(-,+),(+,-),(+,+)$ the first replicate gives $10,14,15,21$ and the second $12,13,17,20$. Fit $E[y]=\beta_0+\beta_1x_1+\beta_2x_2$ and test, at $5\%$, (a) $H_0:\beta_1=\beta_2$ and (b) $H_0:\beta_1=\beta_2=0$.
Fit The columns $\mathbf 1,x_1,x_2$ are orthogonal with squared lengths $8$, so $X^{\mathsf T}X=8I_3$ and $\hat\beta=X^{\mathsf T}y/8$: $\hat\beta_0=122/8=15.25$, $\hat\beta_1=24/8=3.0$, $\hat\beta_2=14/8=1.75$.
Residual SS $\sum(y-\bar y)^2=103.5$ and the regression part is $8(\hat\beta_1^2+\hat\beta_2^2)=8(9+3.0625)=96.5$, so $\operatorname{RSS}=7.0$ on $8-3=5$ degrees of freedom and $\hat\sigma^2=1.4$.
(a) $\lambda=(0,1,-1)^{\mathsf T}$, $\lambda^{\mathsf T}\hat\beta=1.25$, $\lambda^{\mathsf T}(X^{\mathsf T}X)^{-1}\lambda=\frac28$. So $\operatorname{RSS}_0-\operatorname{RSS}=\frac{1.25^2}{0.25}=6.25$ and $F=\frac{6.25}{1.4}=4.46\lt F_{1,5,0.05}=6.61$: do not reject ($p=0.088$). Check: refitting with the single regressor $x_1+x_2$ gives $\operatorname{RSS}_0=13.25=7.0+6.25$. ✓
(b) $q=2$: $\operatorname{RSS}_0=\sum(y-\bar y)^2=103.5$, so $F=\frac{96.5/2}{1.4}=34.5\gt F_{2,5,0.05}=5.79$: reject ($p=0.0012$). The factors matter, but the data do not show that their effects differ.
Note The residual SS contains the interaction $x_1x_2$ (its estimate is $0.5$, with sum of squares $8\times0.5^2=2.0$) and pure error ($5.0$ on $4$ df). If interaction is possible, it should be in the model; §8.4 treats $2^k$ designs in full.
📐 Theorem 6.10 — Cochran's theorem (statement)

Let $y\sim N_n(\mu,\sigma^2I_n)$ and suppose $y^{\mathsf T}y=\sum_{i=1}^kQ_i$, where $Q_i=y^{\mathsf T}A_iy$ with $A_i$ symmetric of rank $r_i$. Then the following are equivalent:

  1. $r_1+\cdots+r_k=n$;
  2. each $A_i$ is idempotent;
  3. $A_iA_j=0$ for all $i\ne j$.

When they hold, $Q_1,\dots,Q_k$ are independent and $Q_i/\sigma^2\sim\chi^2_{r_i}(\mu^{\mathsf T}A_i\mu/\sigma^2)$ (central when $A_i\mu=0$). More generally, for symmetric $A$, $y^{\mathsf T}Ay/\sigma^2$ has a (noncentral) $\chi^2$ law if and only if $A$ is idempotent, and then the degrees of freedom are $\operatorname{rank}A=\operatorname{tr}A$.

The proof, by simultaneous diagonalisation, is given with the distribution of quadratic forms in §7.1. Cochran's theorem is the bookkeeping device of the analysis of variance: whenever a total sum of squares is split into pieces whose degrees of freedom add up, the pieces are independent $\chi^2$ variables, and their ratios are $F$ variables. In this section, $y^{\mathsf T}y=y^{\mathsf T}Hy+y^{\mathsf T}(I-H)y$ with ranks $r+(n-r)=n$ is the simplest instance; §6.2 uses finer splittings.

✏️ Example 7 — True or false: residuals in the normal model
In the normal straight-line model with the design $x=(1,2,3,4,5)$ of Example 1, decide each statement. (a) The residuals $e_1,\dots,e_5$ are independent. (b) $e$ is independent of $\hat\beta$. (c) $e_1$ and $e_5$ have the same variance as the errors. (d) $\sum e_i^2/\sigma^2\sim\chi^2_5$.
(a) False $\operatorname{Cov}(e)=\sigma^2(I-H)$, and $h_{15}=\frac15+\frac{(1-3)(5-3)}{10}=-0.2$. So $\operatorname{Cov}(e_1,e_5)=0.2\sigma^2$ and, with $\operatorname{Var}(e_1)=\operatorname{Var}(e_5)=(1-0.6)\sigma^2$, $\operatorname{Corr}(e_1,e_5)=\frac{0.2}{0.4}=0.5$. Jointly normal and correlated, hence dependent. (The residuals also satisfy two linear constraints, $\sum e_i=\sum x_ie_i=0$.)
(b) True Theorem 6.7(1): $\hat\beta$ is a function of $\hat y$, which is independent of $e$.
(c) False $\operatorname{Var}(e_1)=\sigma^2(1-h_{11})=0.4\sigma^2$: the line is pulled towards the end points, so their residuals are small. Standardised residuals $e_i/(\hat\sigma\sqrt{1-h_{ii}})$ correct for this (§6.3).
(d) False $\operatorname{RSS}/\sigma^2\sim\chi^2_{n-r}=\chi^2_3$: two degrees of freedom are used by the two estimated parameters. In particular $E[\operatorname{RSS}]=3\sigma^2$, not $5\sigma^2$.
⚠️ Counterexamples and pitfalls
  1. Unique fit, non-unique coefficients. When $r\lt p$ the normal equations have infinitely many solutions (Example 2), but $\hat y$, $e$, $\operatorname{RSS}$ and every estimable $\lambda^{\mathsf T}\hat\beta$ are unique. A coefficient printed by software for a rank-deficient model depends on the constraint the program chose.
  2. Individual effects are not estimable. In $y_{ij}=\mu+\alpha_i+\varepsilon_{ij}$, neither $\mu$ nor $\alpha_i$ nor $\alpha_1+\alpha_2$ is estimable; $\mu+\alpha_i$ and contrasts $\sum c_i\alpha_i$ ($\sum c_i=0$) are. More data per group do not help.
  3. Gauss–Markov does not need normality — and does not give normality. It needs $E\varepsilon=0$ and $\operatorname{Cov}\varepsilon=\sigma^2I$ only (Interactive 2 with uniform or Laplace errors). The exact $t$- and $F$-distributions, however, do need normality.
  4. "Best" means best among linear unbiased estimators. A biased estimator can have smaller mean squared error: in $y_i=\beta+\varepsilon_i$, $0.9\bar y$ has MSE $0.81\sigma^2/n+0.01\beta^2$, smaller than $\sigma^2/n$ when $\beta^2\lt19\sigma^2/n$. With non-normal errors a nonlinear unbiased estimator can also win (for uniform errors the midrange beats $\bar y$ for a location parameter).
  5. Heteroscedastic or correlated errors. Least squares stays unbiased but is no longer BLUE, and the usual formula $\sigma^2\lambda^{\mathsf T}G\lambda$ for its variance is wrong. Use generalised or weighted least squares (Theorem 6.5).
  6. Residuals are not the errors. They are correlated, have variances $\sigma^2(1-h_{ii})$, and sum to zero when there is an intercept (Example 7). Without an intercept $\sum e_i$ need not be $0$ (Exercise I.3).
  7. Divide by $n-r$, not $n$ or $n-p$. $\operatorname{RSS}/n$ is the MLE and is biased; with a rank-deficient design the correct divisor uses the rank $r$, not the number of parameters $p$.
  8. Only testable hypotheses can be tested. "$\alpha_1=0$" in the one-way model has no meaning that is invariant under the choice of side condition; the numerator degrees of freedom of an $F$-test count independent estimable constraints.
  9. Joint and separate tests can disagree. With strongly correlated estimates, $H_0:\beta_1=\beta_2=0$ can be rejected by the $F$-test while neither individual $t$-test rejects, and vice versa: the confidence ellipse of Theorem 6.8(3) is not the rectangle of two intervals.
  10. Uncorrelated is not independent. The Gauss–Markov model asks only for uncorrelated errors; independence of $\hat\beta$ and $\operatorname{RSS}$ (Theorem 6.7) is a consequence of joint normality.

Exercises

I
Least squares, normal equations and projections.
  • 1 Fit $y=\beta_0+\beta_1x$ to $x=(0,1,2,3)$, $y=(1,3,2,6)$. Give $\hat\beta$, the residuals and $\operatorname{RSS}$.
  • 2 Fit the line through the origin $y=\beta x$ to the same data; give $\hat\beta$ and $\operatorname{RSS}$.
  • 3 Show that residuals sum to zero whenever $\mathbf 1\in\mathcal C(X)$. Using part 2, show that they need not sum to zero otherwise.
  • 4 Prove directly that $\mathcal N(X^{\mathsf T}X)=\mathcal N(X)$ and deduce $\operatorname{rank}(X^{\mathsf T}X)=\operatorname{rank}(X)$.
  • 5 Show that if $G$ is a g-inverse of a symmetric matrix $A$, so is $G^{\mathsf T}$, and so is $GAG^{\mathsf T}$, which is symmetric.
  • 6 Find a g-inverse and the Moore–Penrose inverse of $A=\begin{pmatrix}2&2\\2&2\end{pmatrix}$.
  • 7 Show that in the one-way model with $k$ groups of sizes $n_1,\dots,n_k$ the hat matrix is block diagonal with blocks $\frac{1}{n_i}J_{n_i}$ ($J$ the matrix of ones), so that $\hat y_{ij}=\bar y_i$ and $\operatorname{tr}H=k$.
  • 8 Prove that when the model has an intercept, $\sum(y_i-\bar y)^2=\sum(\hat y_i-\bar y)^2+\sum e_i^2$, and that $\sum\hat y_i=\sum y_i$.
1 $\bar x=1.5$, $\bar y=3$, $S_{xx}=5$, $S_{xy}=7$: $\hat\beta_1=1.4$, $\hat\beta_0=3-2.1=0.9$. Fitted $(0.9,2.3,3.7,5.1)$, residuals $(0.1,0.7,-1.7,0.9)$, $\operatorname{RSS}=4.2$.
2 $\hat\beta=\sum x_iy_i/\sum x_i^2=25/14=1.786$; $\operatorname{RSS}=\sum y_i^2-(\sum x_iy_i)^2/\sum x_i^2=50-625/14=5.357$ (larger, because the model is smaller).
3 $X^{\mathsf T}e=0$ and $\mathbf 1=Xc$ for some $c$ give $\mathbf 1^{\mathsf T}e=c^{\mathsf T}X^{\mathsf T}e=0$. In part 2 the residuals are $(1,\ 1.214,\ -1.571,\ 0.643)$, which sum to $1.286\ne0$; only $\sum x_ie_i=0$ holds.
4 $Xv=0\Rightarrow X^{\mathsf T}Xv=0$; and $X^{\mathsf T}Xv=0\Rightarrow\|Xv\|^2=v^{\mathsf T}X^{\mathsf T}Xv=0\Rightarrow Xv=0$. Both matrices have $p$ columns, so rank–nullity gives equal ranks.
5 Transposing $AGA=A$ gives $AG^{\mathsf T}A=A$. Then $A(GAG^{\mathsf T})A=(AGA)G^{\mathsf T}A=AG^{\mathsf T}A=A$, and $(GAG^{\mathsf T})^{\mathsf T}=GA^{\mathsf T}G^{\mathsf T}=GAG^{\mathsf T}$.
6 $G=\begin{pmatrix}1/2&0\\0&0\end{pmatrix}$: $AGA=\begin{pmatrix}1&0\\1&0\end{pmatrix}A=A$. ✓ Since $A=4uu^{\mathsf T}$ with $u=(1,1)^{\mathsf T}/\sqrt2$ a unit vector, $A^+=\frac14uu^{\mathsf T}=\begin{pmatrix}1/8&1/8\\1/8&1/8\end{pmatrix}$; it satisfies all four Penrose conditions.
7 Use the full-rank cell-means parametrisation $E[y_{ij}]=\mu_i$ (same $\mathcal C(X)$, so the same $H$): $X=\operatorname{blockdiag}(\mathbf 1_{n_i})$, $X^{\mathsf T}X=\operatorname{diag}(n_i)$ and $H=\operatorname{blockdiag}(\mathbf 1_{n_i}\mathbf 1_{n_i}^{\mathsf T}/n_i)$. So $\hat y_{ij}=\bar y_i$ and $\operatorname{tr}H=\sum_i n_i/n_i=k$.
8 $y-\bar y\mathbf 1=(\hat y-\bar y\mathbf 1)+e$; the first vector lies in $\mathcal C(X)$ (since $\mathbf 1$ does) and $e\perp\mathcal C(X)$, so Pythagoras applies. And $\sum\hat y_i=\sum y_i-\sum e_i=\sum y_i$ by part 3.
II
Estimability, BLUE and generalised least squares.
  • 1 In the one-way model with $k=3$ groups, which of $\mu$, $\alpha_1$, $\alpha_1-\alpha_2$, $\mu+\alpha_3$, $\alpha_1+\alpha_2-2\alpha_3$, $\alpha_1+\alpha_2$ are estimable?
  • 2 Prove that $\lambda^{\mathsf T}\beta$ is estimable if and only if $\lambda^{\mathsf T}GX^{\mathsf T}X=\lambda^{\mathsf T}$, where $G$ is any g-inverse of $X^{\mathsf T}X$.
  • 3 In the additive two-way model $y_{ij}=\mu+\alpha_i+\beta_j+\varepsilon_{ij}$ with $a=2$ rows, $b=3$ columns and one observation per cell, find the BLUE of $\alpha_1-\alpha_2$ and its variance.
  • 4 In $y_i=\beta x_i+\varepsilon_i$ (Gauss–Markov errors, $\sum x_i\ne0$), show that $\sum y_i/\sum x_i$ is unbiased and find its efficiency relative to the BLUE. Evaluate for $x=(1,2,3)$.
  • 5 Prove the matrix form of the Gauss–Markov theorem: for estimable $\Lambda^{\mathsf T}\beta$ and any linear unbiased $T=A^{\mathsf T}y$, $\operatorname{Cov}(T)-\operatorname{Cov}(\Lambda^{\mathsf T}\hat\beta)$ is positive semidefinite.
  • 6 In $y_i=\beta x_i+\varepsilon_i$ with $x_i\gt0$, uncorrelated errors and $\operatorname{Var}(\varepsilon_i)=\sigma^2x_i$, find the BLUE of $\beta$ and its variance.
  • 7 Suppose the model has an intercept, is of full rank, and $\operatorname{Cov}(\varepsilon)=\sigma^2\left[(1-\rho)I+\rho J\right]$ with $-\frac1{n-1}\lt\rho\lt1$ (equicorrelated errors). Prove that ordinary least squares coincides with generalised least squares.
  • 8 For the design $x=(1,2,3,4,5)$, show that $(y_4+y_5-y_1-y_2)/6$ is an unbiased estimator of the slope and find its efficiency.
  • 9 Prove that the BLUE is unique: if $a^{\mathsf T}y$ and $c^{\mathsf T}y$ are both BLUEs of $\lambda^{\mathsf T}\beta$, then $a=c$.
1 Criterion: $\lambda_0=\lambda_1+\lambda_2+\lambda_3$. Estimable: $\alpha_1-\alpha_2$ ($0=1-1$), $\mu+\alpha_3$ ($1=1$), $\alpha_1+\alpha_2-2\alpha_3$ ($0=0$). Not estimable: $\mu$ ($1\ne0$), $\alpha_1$ ($0\ne1$), $\alpha_1+\alpha_2$ ($0\ne2$).
2 Theorem 6.3, (2)⇔(3): if $\lambda=X^{\mathsf T}a$ then $\lambda^{\mathsf T}GX^{\mathsf T}X=a^{\mathsf T}XGX^{\mathsf T}X=a^{\mathsf T}X=\lambda^{\mathsf T}$; conversely the identity exhibits $\lambda=X^{\mathsf T}(XG^{\mathsf T}\lambda)\in\mathcal C(X^{\mathsf T})$.
3 $\bar y_{1\cdot}-\bar y_{2\cdot}$ is unbiased, since $E\bar y_{i\cdot}=\mu+\alpha_i+\bar\beta$ with $\bar\beta=\frac13\sum_j\beta_j$. Its coefficient vector is $a=\frac13(\text{row-1 indicator})-\frac13(\text{row-2 indicator})$, and the row indicators are columns of $X$, so $a\in\mathcal C(X)$ and $a^{\mathsf T}y$ is the BLUE (the criterion after Theorem 6.4). Its variance is $\sigma^2\|a\|^2=6\cdot\frac19\sigma^2=\frac{2\sigma^2}{3}$.
4 $E[\sum y_i/\sum x_i]=\beta$; variance $n\sigma^2/(\sum x_i)^2$. The BLUE $\sum x_iy_i/\sum x_i^2$ has variance $\sigma^2/\sum x_i^2$. Efficiency $(\sum x_i)^2/(n\sum x_i^2)\le1$ by Cauchy–Schwarz, with equality iff all $x_i$ are equal. For $x=(1,2,3)$: $36/42=0.857$.
5 For any $c\in\mathbb R^q$, $c^{\mathsf T}T$ is a linear unbiased estimator of the estimable function $(\Lambda c)^{\mathsf T}\beta$, whose BLUE is $c^{\mathsf T}\Lambda^{\mathsf T}\hat\beta$. So $c^{\mathsf T}\left[\operatorname{Cov}(T)-\operatorname{Cov}(\Lambda^{\mathsf T}\hat\beta)\right]c=\operatorname{Var}(c^{\mathsf T}T)-\operatorname{Var}(c^{\mathsf T}\Lambda^{\mathsf T}\hat\beta)\ge0$.
6 Theorem 6.5 with $V=\operatorname{diag}(x_i)$: $\tilde\beta=\frac{\sum x_iy_i/x_i}{\sum x_i^2/x_i}=\frac{\sum y_i}{\sum x_i}$ with variance $\sigma^2/\sum x_i$. (So the "naive" estimator of part 4 is best under this variance structure.)
7 $V=(1-\rho)I+\rho\mathbf 1\mathbf 1^{\mathsf T}$ is positive definite in that range. Let $\mathbf 1=Xc$. Then $VX=(1-\rho)X+\rho Xc\,\mathbf 1^{\mathsf T}X=XM$ with $M=(1-\rho)I+\rho c\mathbf 1^{\mathsf T}X$, and $M$ is invertible because $VX$ has rank $p$. Hence $V^{-1}X=XM^{-1}$ and $X^{\mathsf T}V^{-1}=(M^{-1})^{\mathsf T}X^{\mathsf T}$, so the GLS equations $X^{\mathsf T}V^{-1}Xb=X^{\mathsf T}V^{-1}y$ are $(M^{-1})^{\mathsf T}(X^{\mathsf T}Xb-X^{\mathsf T}y)=0$, i.e. the ordinary normal equations.
8 $a=\frac16(-1,-1,0,1,1)$: $\sum a_i=0$, $\sum a_ix_i=\frac{-1-2+4+5}6=1$. ✓ $\operatorname{Var}=\frac{4}{36}\sigma^2=\frac{\sigma^2}9$; efficiency $\frac{1/10}{1/9}=0.9$.
9 In the proof of Theorem 6.4, $\operatorname{Var}(a^{\mathsf T}y)=\sigma^2\|a^*\|^2+\sigma^2\|(I-H)a\|^2$ with $a^*=Ha$ common to all unbiased $a$. Minimum variance forces $(I-H)a=0$, i.e. $a=a^*$; likewise $c=a^*$.
III
Estimating $\sigma^2$, distributions and confidence intervals.
  • 1 Prove that $E[y^{\mathsf T}Ay]=\operatorname{tr}(A\Sigma)+\mu^{\mathsf T}A\mu$ for symmetric $A$, without assuming normality.
  • 2 Use part 1 to show $E[\operatorname{RSS}]=(n-r)\sigma^2$ in any Gauss–Markov model. What is $E[y^{\mathsf T}Hy]$?
  • 3 For the data of Example 1, find a $95\%$ confidence interval for the intercept $\beta_0$.
  • 4 (Calibration.) A spectrometer is calibrated with $n=12$ standards: $\bar x=5$, $S_{xx}=50$, $\hat\beta_0=0.4$, $\hat\beta_1=2.1$, $\operatorname{RSS}=4.0$ (absorbance units). Assuming normal errors, find $\hat\sigma^2$, a $95\%$ interval for the slope, a $95\%$ interval for the mean absorbance at $x_0=8$, and a $95\%$ prediction interval for one new reading at $x_0=8$.
  • 5 Show that the MLE $\operatorname{RSS}/n$ of $\sigma^2$ has bias $-r\sigma^2/n$ and compare its MSE with that of $\hat\sigma^2=\operatorname{RSS}/(n-r)$ under normality.
  • 6 Show that $\operatorname{Cov}(\hat\beta,e)=0$ in every full-rank Gauss–Markov model. Give an example in which $\hat\beta$ and $e$ are uncorrelated but not independent.
  • 7 Find a $90\%$ confidence interval for $\sigma^2$ for the data of Example 1.
  • 8 In simple regression show that $\operatorname{Cov}(\hat\beta_0,\hat\beta_1)=-\bar x\sigma^2/S_{xx}$. For which designs are $\hat\beta_0$ and $\hat\beta_1$ uncorrelated? Find $\operatorname{Corr}(\hat\beta_0,\hat\beta_1)$ for $x=(1,2,3,4,5)$.
1 $y^{\mathsf T}Ay=\operatorname{tr}(Ayy^{\mathsf T})$; expectation and trace commute, and $E[yy^{\mathsf T}]=\Sigma+\mu\mu^{\mathsf T}$, so $E[y^{\mathsf T}Ay]=\operatorname{tr}(A\Sigma)+\mu^{\mathsf T}A\mu$.
2 $A=I-H$, $\Sigma=\sigma^2I$, $\mu=X\beta$: $\sigma^2(n-r)+0$. For $A=H$: $E[y^{\mathsf T}Hy]=r\sigma^2+\beta^{\mathsf T}X^{\mathsf T}X\beta$.
3 $\operatorname{SE}(\hat\beta_0)=\sqrt{0.3667\left(\frac15+\frac9{10}\right)}=0.6351$; $0.9\pm3.182\times0.6351=(-1.12,\ 2.92)$. The intercept is not significantly different from zero.
4 $\hat\sigma^2=4/10=0.4$, $t_{10,0.025}=2.228$. Slope: $2.1\pm2.228\sqrt{0.4/50}=2.1\pm0.199$, i.e. $(1.90,\ 2.30)$. Mean at $8$: $17.2\pm2.228\sqrt{0.4\left(\frac1{12}+\frac9{50}\right)}=17.2\pm0.723$, i.e. $(16.48,\ 17.92)$. Prediction: $17.2\pm2.228\sqrt{0.4\left(1+\frac1{12}+\frac{9}{50}\right)}=17.2\pm1.584$, i.e. $(15.62,\ 18.78)$ absorbance units.
5 $E[\operatorname{RSS}/n]=(n-r)\sigma^2/n$, bias $-r\sigma^2/n$. Under normality $\operatorname{Var}(\operatorname{RSS})=2(n-r)\sigma^4$, so $\operatorname{MSE}(\operatorname{RSS}/n)=\frac{2(n-r)+r^2}{n^2}\sigma^4$ and $\operatorname{MSE}(\hat\sigma^2)=\frac{2}{n-r}\sigma^4$. With $m=n-r$, the MLE has the smaller MSE iff $m(2m+r^2)\lt2(m+r)^2$, i.e. $m(r-4)\lt2r$: always when $r\le4$ (e.g. $r=1$: $(2n-1)/n^2\lt2/(n-1)$), but for $r\ge5$ only when $n-r\lt2r/(r-4)$. The best multiple of $\operatorname{RSS}$ in MSE is $\operatorname{RSS}/(n-r+2)$.
6 $\operatorname{Cov}((X^{\mathsf T}X)^{-1}X^{\mathsf T}y,(I-H)y)=\sigma^2(X^{\mathsf T}X)^{-1}X^{\mathsf T}(I-H)=0$. Example: $y_i=\mu+\varepsilon_i$, $n=2$, with $\varepsilon_i$ iid taking values $\pm1$ with probability $\frac12$. Then $\hat\mu=\bar y$ and $e_1=(\varepsilon_1-\varepsilon_2)/2$; $e_1\ne0$ forces $\varepsilon_1=-\varepsilon_2$ and hence $\bar y=\mu$, so they are dependent although uncorrelated.
7 $\chi^2_{3,0.05}=7.815$, $\chi^2_{3,0.95}=0.3518$: $\left[\frac{1.1}{7.815},\frac{1.1}{0.3518}\right]=(0.141,\ 3.13)$.
8 $\hat\beta_0=\bar y-\hat\beta_1\bar x$, and $\operatorname{Cov}(\bar y,\hat\beta_1)=\sigma^2\sum\frac1n\frac{x_i-\bar x}{S_{xx}}=0$, so $\operatorname{Cov}(\hat\beta_0,\hat\beta_1)=-\bar x\operatorname{Var}(\hat\beta_1)=-\bar x\sigma^2/S_{xx}$. Uncorrelated iff $\bar x=0$ (centred design). For $x=(1,\dots,5)$: $\frac{-0.3}{\sqrt{1.1\times0.1}}=-0.905$.
IV
Linear hypotheses and Cochran's theorem.
  • 1 For the $2^2$ data of Example 6, test $H_0:\beta_2=0$ at $5\%$ by a $t$-test and by the $F$-test, and check $F=t^2$.
  • 2 Prove in general that for one constraint the $F$ statistic of Theorem 6.9 equals $t^2$.
  • 3 Write $H_0:\alpha_1=\alpha_2=\alpha_3$ in the one-way model as $\Lambda^{\mathsf T}\beta=0$ with $\Lambda$ of rank $2$ and verify that it is testable. What are the degrees of freedom of the $F$-test if $N=15$?
  • 4 Show that $H_0:\alpha_1=0$ is not testable in the one-way model, by exhibiting two parameter vectors with the same $X\beta$, one satisfying $H_0$ and one not.
  • 5 Prove that the likelihood ratio for $H_0:\Lambda^{\mathsf T}\beta=d$ in the normal model is $\left(1+\frac{q}{n-r}F\right)^{-n/2}$.
  • 6 (Two calibration lines.) Two instruments are each calibrated at $x=1,\dots,5$. Instrument A gives $S_{xx}=10$, $S_{xy}=17$, $S_{yy}=30$; instrument B gives $S_{xx}=10$, $S_{xy}=12$, $S_{yy}=16$ (sums of squares about each instrument's own means). Fitting separate lines, test at $5\%$ whether the slopes are equal.
  • 7 In the one-way model, write the identity $$\sum_{ij}y_{ij}^2=N\bar y^2+\sum_in_i(\bar y_i-\bar y)^2+\sum_{ij}(y_{ij}-\bar y_i)^2$$ as $y^{\mathsf T}A_1y+y^{\mathsf T}A_2y+y^{\mathsf T}A_3y$, give the ranks, and state what Cochran's theorem yields.
  • 8 For the design of Example 1 with $\sigma=1$ and true slope $\beta_1=0.5$, find the noncentrality of the $F$-test of $\beta_1=0$ and its power at $5\%$ (use software for the noncentral $F$).
1 $\operatorname{SE}(\hat\beta_2)=\sqrt{1.4/8}=0.4183$, $t=1.75/0.4183=4.18\gt t_{5,0.025}=2.571$: reject. $F=8\times1.75^2/1.4=17.5=4.18^2$. ✓
2 With $q=1$, $\operatorname{RSS}_0-\operatorname{RSS}=(\lambda^{\mathsf T}\hat\beta-d)^2/(\lambda^{\mathsf T}G\lambda)$, so $F=\frac{(\lambda^{\mathsf T}\hat\beta-d)^2}{\hat\sigma^2\lambda^{\mathsf T}G\lambda}=t^2$; and $t_{\nu}^2\sim F_{1,\nu}$ (§4.1), with $t^2_{\nu,\alpha/2}=F_{1,\nu,\alpha}$.
3 $\Lambda^{\mathsf T}=\begin{pmatrix}0&1&-1&0\\0&1&0&-1\end{pmatrix}$ (rows $\alpha_1-\alpha_2$, $\alpha_1-\alpha_3$), rank $2$; each row has $\lambda_0=0=\sum\lambda_i$, so both are estimable. Degrees of freedom $(q,N-r)=(2,15-3)=(2,12)$.
4 $\beta=(\mu,\alpha_1,\alpha_2,\alpha_3)=(5,0,1,2)$ and $(4,1,2,3)$ give the same group means $5,6,7$, hence the same $X\beta$ and the same distribution of $y$; the first satisfies $\alpha_1=0$, the second does not. No test can distinguish them.
5 $\sup_{\beta,\sigma^2}L=(2\pi\operatorname{RSS}/n)^{-n/2}e^{-n/2}$ and $\sup_{H_0}L=(2\pi\operatorname{RSS}_0/n)^{-n/2}e^{-n/2}$, so $\Lambda=(\operatorname{RSS}/\operatorname{RSS}_0)^{n/2}$ and $\operatorname{RSS}_0/\operatorname{RSS}=1+\frac{q}{n-r}F$.
6 Separate fits: $\operatorname{RSS}_A=30-17^2/10=1.1$, $\operatorname{RSS}_B=16-12^2/10=1.6$, total $2.7$ on $10-4=6$ df. Common slope (separate intercepts): $b=\frac{17+12}{10+10}=1.45$ and $\operatorname{RSS}_0=46-29^2/20=3.95$ on $7$ df. $F=\frac{1.25/1}{2.7/6}=2.78\lt F_{1,6,0.05}=5.99$ ($p=0.147$): equal slopes are not rejected.
7 $A_1=\frac1NJ_N$ (rank $1$), $A_2=H-\frac1NJ_N$ with $H=\operatorname{blockdiag}(J_{n_i}/n_i)$ (rank $k-1$), $A_3=I-H$ (rank $N-k$). The ranks add to $N$, so the three forms are independent, $\sum(y_{ij}-\bar y_i)^2/\sigma^2\sim\chi^2_{N-k}$, and under equal means $\sum n_i(\bar y_i-\bar y)^2/\sigma^2\sim\chi^2_{k-1}$: the one-way $F$-test of §6.2.
8 $\delta=\beta_1^2/(\sigma^2\lambda^{\mathsf T}G\lambda)=\beta_1^2S_{xx}/\sigma^2=0.25\times10=2.5$. Power $=P(F_{1,3}(2.5)\gt10.13)=0.202$: five points and three error degrees of freedom give little power.
🎯 Exam corner CSIR Part BCSIR Part CGATE ST

E1. CSIR Part B GATE ST In the model $y_i=\beta_0+\beta_1x_i+\varepsilon_i$, $i=1,\dots,5$, with uncorrelated errors of common variance and $x=(1,2,3,4,5)$, the correlation between the least squares estimators $\hat\beta_0$ and $\hat\beta_1$ is closest to (A) $0$; (B) $-0.30$; (C) $-0.90$; (D) $-0.55$.

Idea $\operatorname{Cov}(\hat\beta_0,\hat\beta_1)=-\bar x\sigma^2/S_{xx}$, $\operatorname{Var}(\hat\beta_1)=\sigma^2/S_{xx}$, $\operatorname{Var}(\hat\beta_0)=\sigma^2(1/n+\bar x^2/S_{xx})$.
Compute $\bar x=3$, $S_{xx}=10$: covariance $-0.3\sigma^2$, variances $0.1\sigma^2$ and $1.1\sigma^2$. Correlation $-0.3/\sqrt{0.11}=-0.905$.
Answer (C). (Option (B) is the covariance in units of $\sigma^2$, not the correlation.)

E2. CSIR Part B Let $y_1,y_2,y_3$ be uncorrelated with common variance and $E[y_1]=\beta_1+\beta_2$, $E[y_2]=\beta_2+\beta_3$, $E[y_3]=\beta_1+2\beta_2+\beta_3$. Which of the following parametric functions is estimable? (A) $\beta_2$; (B) $\beta_1+\beta_3$; (C) $\beta_1-\beta_3$; (D) $\beta_1$.

Idea Theorem 6.3: $\lambda$ must lie in the row space of $X$, spanned by $(1,1,0)$ and $(0,1,1)$ (the third row is their sum).
Check $a(1,1,0)+b(0,1,1)=(a,a+b,b)$. (C) $(1,0,-1)$: $a=1$, $b=-1$, middle entry $0$ ✓. (A) $(0,1,0)$ needs $a=b=0$: no. (B) $(1,0,1)$ needs $a=b=1$, middle $2$: no. (D) $(1,0,0)$ needs $a=1,b=0$, middle $1$: no. Equivalently, $\mathcal N(X)$ is spanned by $(1,-1,1)$ and only (C) is orthogonal to it.
Answer (C), with unbiased estimator $y_1-y_2$.

E3. CSIR Part B GATE ST In a normal linear model with $n=25$ observations and a design matrix of rank $5$, the residual sum of squares is $60$. Imposing two linearly independent testable constraints raises the residual sum of squares to $84$. The $F$ statistic for testing the constraints, and its degrees of freedom, are (A) $4.0$ on $(2,20)$; (B) $2.0$ on $(2,20)$; (C) $4.0$ on $(2,23)$; (D) $0.4$ on $(20,2)$.

Idea Theorem 6.9: $F=\frac{(\operatorname{RSS}_0-\operatorname{RSS})/q}{\operatorname{RSS}/(n-r)}$ on $(q,n-r)$.
Compute $\frac{24/2}{60/20}=\frac{12}{3}=4.0$ on $(2,20)$. (Since $F_{2,20,0.05}=3.49$, the constraints are rejected at $5\%$.)
Answer (A).

E4. CSIR Part C Consider $y=X\beta+\varepsilon$ with $X$ an $n\times p$ matrix of rank $r\lt p\lt n$, $E\varepsilon=0$, $\operatorname{Cov}\varepsilon=\sigma^2I$. Which of the following are true? (One or more options may be correct.) (A) $X^{\mathsf T}X$ is singular. (B) All solutions of the normal equations give the same vector of fitted values. (C) Every coordinate $\beta_j$ is estimable. (D) $\operatorname{RSS}/(n-r)$ is an unbiased estimator of $\sigma^2$.

(A) True $\operatorname{rank}(X^{\mathsf T}X)=r\lt p$ (Theorem 6.1(1)).
(B) True Two solutions differ by a vector of $\mathcal N(X^{\mathsf T}X)=\mathcal N(X)$, so $X\hat\beta$ is unique (Theorem 6.1(4)).
(C) False The estimable functions form $\mathcal C(X^{\mathsf T})$, of dimension $r\lt p$; it cannot contain all $p$ unit vectors. At least one $\beta_j$ is not estimable (in the one-way model, none is).
(D) True Theorem 6.6: $E[\operatorname{RSS}]=\sigma^2\operatorname{tr}(I-H)=(n-r)\sigma^2$; no normality needed.
Answer (A), (B) and (D).

E5. CSIR Part C GATE ST In the normal linear model $y\sim N_n(X\beta,\sigma^2I)$ with $X$ of full rank $p\lt n$, let $\hat\beta$ be the least squares estimator and $e$ the residual vector. Which of the following are true? (A) $\hat\beta$ and $e$ are independent. (B) $e_1,\dots,e_n$ are independent. (C) $\|e\|^2/\sigma^2\sim\chi^2_{n-p}$. (D) Each $\hat\beta_j$ is the UMVUE of $\beta_j$.

(A) True Theorem 6.7(1): $\hat\beta$ is a function of $\hat y$, and $(\hat y,e)$ are jointly normal and uncorrelated.
(B) False $\operatorname{Cov}(e)=\sigma^2(I-H)$ is not diagonal (in Example 7, $\operatorname{Corr}(e_1,e_5)=0.5$); indeed $e$ lies in an $(n-p)$-dimensional subspace, so its $n$ coordinates cannot be independent nondegenerate normals.
(C) True Theorem 6.7(3) with $r=p$.
(D) True $(\hat\beta,\operatorname{RSS})$ is complete sufficient (Theorem 6.7(4)) and $\hat\beta_j$ is an unbiased function of it; Lehmann–Scheffé.
Answer (A), (C) and (D).

E6. CSIR Part C Let $y_i=\beta x_i+\varepsilon_i$, $i=1,\dots,n$ ($n\ge2$), with known $x_i\gt0$ not all equal and uncorrelated errors of mean $0$ and variance $\sigma^2$. Which of the following are true? (A) $\sum y_i/\sum x_i$ is unbiased for $\beta$. (B) $\frac1n\sum y_i/x_i$ is unbiased for $\beta$. (C) $\sum x_iy_i/\sum x_i^2$ has variance $\sigma^2/\sum x_i^2$. (D) $\operatorname{Var}\left(\sum y_i/\sum x_i\right)\le\operatorname{Var}\left(\sum x_iy_i/\sum x_i^2\right)$.

(A) True $E\sum y_i=\beta\sum x_i$.
(B) True $E[y_i/x_i]=\beta$ for each $i$.
(C) True $\operatorname{Var}=\sigma^2\sum x_i^2/(\sum x_i^2)^2$. It is the least squares estimator, hence the BLUE.
(D) False $\operatorname{Var}(\sum y_i/\sum x_i)=n\sigma^2/(\sum x_i)^2\ge\sigma^2/\sum x_i^2$ by Cauchy–Schwarz, with equality only when all $x_i$ are equal, which is excluded. Gauss–Markov gives the opposite inequality, strictly.
Answer (A), (B) and (C).
🧑‍🏫 For the teacher

Suggested plan — 3 lectures × 60 min.

  1. Least squares as projection. Definition 6.1 with the three standard designs written out as matrices. Theorem 6.1 in full (the Pythagoras argument), then Example 1 on the board. Project Interactive 1, move $y_2$ and let the class predict where $\hat y$ goes. Generalised inverses and Theorem 6.2 (prove (1) and (2); state (3)–(4)); finish with Example 2.
  2. Estimability and Gauss–Markov. Theorem 6.3 with the identifiability interpretation; one-way contrasts; Example 3 as a class exercise (ten minutes in pairs). Theorem 6.4 with the full projection proof, Example 4, then Interactive 2: normal, then Laplace errors (least squares still wins), then heteroscedastic (it does not). State Aitken's theorem (Theorem 6.5) and prove it by whitening.
  3. Normal theory and tests. Theorem 6.6 via the trace lemma; Theorem 6.7 via the orthonormal basis; Example 5. Theorem 6.9 with the Lagrange computation; Example 6, including the check by refitting. State Cochran's theorem and point forward to §6.2. Close with Example 7 and E5.

Discussion questions. (1) Software reports $\hat\alpha_3=0$ for a one-way model. Is the third treatment ineffective? What would you report instead? (2) Gauss–Markov says least squares is best. Why do statisticians use ridge regression or robust regression at all? (3) Why is the right divisor for $\sigma^2$ the rank of $X$ and not the number of columns?

Misconceptions to watch for. Believing that Gauss–Markov requires normal errors, or that it compares least squares with all estimators; treating residuals as independent copies of the errors; dividing $\operatorname{RSS}$ by $n$; testing or estimating non-estimable effects; counting the numerator degrees of freedom as the number of parameters mentioned in $H_0$ rather than the number of independent estimable constraints.

Suggested homework. I(1, 3, 6, 7), II(1, 3, 4, 6, 7), III(2, 4, 6, 8), IV(1, 3, 4, 6); Exam corner E2, E4, E6.

🔬 Research–Teaching NexusResearch-tutored · optional, about 60 minutes

Research question. How much precision does a well-chosen design buy? For a straight line on the interval $[0,1]$ with $n=10$ observations, compare designs by $\operatorname{Var}(\hat\beta_1)=\sigma^2/S_{xx}$ and by the worst-case variance of the fitted mean, $\max_{x_0\in[0,1]}\operatorname{Var}(\hat\beta_0+\hat\beta_1x_0)$.

Why it matters. In dose-finding studies, materials testing and sensor calibration, the settings $x_i$ are chosen by the experimenter before any data are seen. The Gauss–Markov formulas show that the precision of the estimates depends on the design through $X^{\mathsf T}X$ alone, so designs can be compared, and optimised, before the experiment. This is the starting point of optimal design theory (D-, A- and G-optimality), widely used in engineering and pharmaceutical research.

What to do.

  1. For three designs — equally spaced points $0,\frac19,\dots,1$; five points at each end; and three at each end with four in the middle — compute $S_{xx}$, $\operatorname{Var}(\hat\beta_1)/\sigma^2$ and $\det(X^{\mathsf T}X)$.
  2. For each design plot $\operatorname{Var}(\hat\beta_0+\hat\beta_1x_0)/\sigma^2=\frac1n+\frac{(x_0-\bar x)^2}{S_{xx}}$ on $[0,1]$ and record its maximum.
  3. Now suppose the truth is quadratic, $E[y]=1+x+cx^2$. Simulate each design with $c=0,\ 0.5,\ 1$ and record how often the lack of linearity can be detected at all. (With two support points it never can.)
  4. Write a paragraph on the trade-off between efficiency under the assumed model and robustness to a wrong model.

Tools: a spreadsheet, R or Python (numpy.linalg); Interactive 2 for intuition about variances of linear estimators.

Hand in: A table of the three designs' criteria, the variance curves, and a recommendation with reasons.

Going further. Read about the Kiefer–Wolfowitz equivalence theorem, which shows that D-optimal and G-optimal approximate designs coincide, and find the D-optimal design for a quadratic on $[-1,1]$ (weights $\frac13$ at $-1,0,1$).

Instructor note: $S_{xx}=1.0185$ for the equally spaced design, $2.5$ for the two-point design (the maximum on $[0,1]$ for $n=10$) and $1.5$ for the 3–4–3 design; the corresponding maximum fitted-mean variances are $0.1+0.25/S_{xx}$, i.e. $0.345$, $0.2$ and $0.267$ (times $\sigma^2$). The two-point design is optimal for the line but cannot detect curvature, which is the point of step 3.

🧪 Self-check quiz

0/5 correct

Q1. In the linear model $E[y]=X\beta$, the function $\lambda^{\mathsf T}\beta$ is estimable if and only if

Q2. In a Gauss–Markov model with $n$ observations and $\operatorname{rank}X=r$, $E[\operatorname{RSS}]$ equals

Q3. The hat matrix $H=X(X^{\mathsf T}X)^-X^{\mathsf T}$

Q4. In a normal linear model with $n=30$ and $\operatorname{rank}X=6$, a testable hypothesis consisting of $3$ linearly independent constraints is tested by an $F$ statistic with degrees of freedom

Q5. The Gauss–Markov theorem requires