6.1 Gauss–Markov Models, Estimability and BLUE
The linear model $y=X\beta+\varepsilon$: least squares as orthogonal projection, normal equations and generalised inverses, which parametric functions can be estimated at all, the Gauss–Markov theorem with proof, the unbiased estimate of $\sigma^2$, exact distribution theory under normality, confidence intervals, and the $F$-test of a general linear hypothesis.
Almost every classical statistical method of the twentieth century is a special case of one model. A chemist calibrating an instrument fits a straight line; an agronomist compares four fertilisers on twenty plots; an economist explains wages by education, experience and region; a clinical trial adjusts a treatment effect for the patients' baseline values. In each case the expected response is a linear function of unknown parameters, $E[y]=X\beta$, where the matrix $X$ records the design: the values of the regressors, or zeros and ones saying which treatment each unit received. Gauss and Legendre fitted such models by least squares two hundred years ago. The questions that make the method a theory are newer. Which functions of $\beta$ can the data estimate at all, when $X$ does not have full column rank (as in every analysis of variance)? In what sense is least squares best? How is the error variance estimated, and what are the exact distributions needed for intervals and tests?
This section answers these questions in one framework, which the rest of the chapter uses. We show that least squares is the orthogonal projection of $y$ onto the column space of $X$, characterise the estimable functions $\lambda^{\mathsf T}\beta$ by the row space of $X$, and prove the Gauss–Markov theorem: the least squares estimator of an estimable function has the smallest variance among all linear unbiased estimators. Adding normality, we derive the $t$-intervals and the $F$-test of a general linear hypothesis $\Lambda^{\mathsf T}\beta=d$. Analysis of variance (§6.2), regression (§6.3) and the designs of Chapter 8 are applications. CSIR NET Part B questions ask for a variance, a degrees-of-freedom count or an $F$ value; Part C questions ask which functions are estimable, which statements about residuals and fitted values are true, and what Gauss–Markov does and does not assert. Quantiles are upper points as in §4.3: $t_{\nu,\alpha}$, $\chi^2_{\nu,\alpha}$ and $F_{\nu_1,\nu_2,\alpha}$ are exceeded with probability $\alpha$. Vectors are columns, $^{\mathsf T}$ denotes the transpose, $\mathcal C(A)$ the column space of a matrix $A$, $\mathbf 1$ a vector of ones and $\|v\|^2=v^{\mathsf T}v$.
The Linear Model and Least Squares
Observations $y=(y_1,\dots,y_n)^{\mathsf T}$ follow the linear model $$y=X\beta+\varepsilon,$$ where $X$ is a known $n\times p$ design matrix of rank $r\le p$, $\beta\in\mathbb R^p$ is an unknown parameter vector and $\varepsilon$ is a random error vector. It is a Gauss–Markov model if $$E[\varepsilon]=0,\qquad\operatorname{Cov}(\varepsilon)=\sigma^2I_n,$$ with $\sigma^2\gt0$ unknown: the errors have mean zero, a common variance, and are uncorrelated. It is the normal linear model if, in addition, $\varepsilon\sim N_n(0,\sigma^2I_n)$, that is, $\varepsilon_1,\dots,\varepsilon_n$ are iid $N(0,\sigma^2)$. The model has full rank if $r=p$.
The word linear refers to the parameters, not to the regressors: $E[y_i]=\beta_0+\beta_1x_i+\beta_2x_i^2$ and $E[y_i]=\beta_0+\beta_1\log x_i$ are linear models. Three examples recur.
- Simple linear regression: $y_i=\beta_0+\beta_1x_i+\varepsilon_i$; $X$ has columns $\mathbf 1$ and $x=(x_1,\dots,x_n)^{\mathsf T}$, and has rank $2$ unless all $x_i$ are equal.
- One-way classification: $y_{ij}=\mu+\alpha_i+\varepsilon_{ij}$ ($i=1,\dots,k$, $j=1,\dots,n_i$). With $\beta=(\mu,\alpha_1,\dots,\alpha_k)^{\mathsf T}$ the columns of $X$ are $\mathbf 1$ and the $k$ group indicators. The indicators add up to $\mathbf 1$, so $p=k+1$ but $r=k$: the model is not of full rank.
- Multiple regression: $y_i=\beta_0+\beta_1x_{i1}+\cdots+\beta_{p-1}x_{i,p-1}+\varepsilon_i$, usually of full rank.
A vector $\hat\beta$ is a least squares estimate of $\beta$ if it minimises $$S(b)=\|y-Xb\|^2=\sum_{i=1}^n\left(y_i-x_i^{\mathsf T}b\right)^2$$ over $b\in\mathbb R^p$, where $x_i^{\mathsf T}$ is the $i$-th row of $X$. The equations $$X^{\mathsf T}Xb=X^{\mathsf T}y$$ are the normal equations. For a least squares estimate, $\hat y=X\hat\beta$ is the vector of fitted values, $e=y-\hat y$ the vector of residuals and $\operatorname{RSS}=\|e\|^2$ the residual sum of squares.
- $\mathcal N(X^{\mathsf T}X)=\mathcal N(X)$, so $\operatorname{rank}(X^{\mathsf T}X)=\operatorname{rank}X=r$ and $\mathcal C(X^{\mathsf T}X)=\mathcal C(X^{\mathsf T})$.
- The normal equations are consistent for every $y$.
- $b$ minimises $S$ if and only if $b$ solves the normal equations. For any solution $\hat\beta$ and any $b$, $$S(b)=S(\hat\beta)+\|X(b-\hat\beta)\|^2.$$
- The fitted vector $\hat y=X\hat\beta$ and the residual vector $e$ are the same for every solution: $\hat y$ is the orthogonal projection of $y$ onto $\mathcal C(X)$, and $X^{\mathsf T}e=0$.
- If $r=p$, the solution is unique: $\hat\beta=(X^{\mathsf T}X)^{-1}X^{\mathsf T}y$.
Proof. (1) If $Xv=0$ then $X^{\mathsf T}Xv=0$. Conversely, if $X^{\mathsf T}Xv=0$ then $\|Xv\|^2=v^{\mathsf T}X^{\mathsf T}Xv=0$, so $Xv=0$. Equal null spaces give equal ranks by the rank–nullity theorem. Since $\mathcal C(X^{\mathsf T}X)\subseteq\mathcal C(X^{\mathsf T})$ and the two spaces have the same dimension $r$, they are equal.
(2) $X^{\mathsf T}y\in\mathcal C(X^{\mathsf T})=\mathcal C(X^{\mathsf T}X)$ by (1), so $X^{\mathsf T}Xb=X^{\mathsf T}y$ has a solution.
(3) Let $\hat\beta$ solve the normal equations and put $e=y-X\hat\beta$, so $X^{\mathsf T}e=0$. For any $b$, $$y-Xb=e+X(\hat\beta-b),$$ and the cross term $2(\hat\beta-b)^{\mathsf T}X^{\mathsf T}e$ vanishes. This gives the displayed identity, so $\hat\beta$ minimises $S$. If $b$ is another minimiser, then $\|X(b-\hat\beta)\|^2=0$, so $Xb=X\hat\beta$ and $X^{\mathsf T}Xb=X^{\mathsf T}X\hat\beta=X^{\mathsf T}y$: $b$ solves the normal equations.
(4) Two solutions $b,\hat\beta$ satisfy $X^{\mathsf T}X(b-\hat\beta)=0$, hence $X(b-\hat\beta)=0$ by (1), so the fitted vectors coincide. The relation $X^{\mathsf T}e=0$ says that $e$ is orthogonal to every column of $X$, so $y=\hat y+e$ with $\hat y\in\mathcal C(X)$ and $e\perp\mathcal C(X)$: this is the orthogonal decomposition of $y$, and $\hat y$ is the projection.
(5) If $r=p$, $X^{\mathsf T}X$ is a $p\times p$ matrix of rank $p$, hence invertible. ∎
When $r\lt p$ the normal equations have infinitely many solutions: if $\hat\beta$ is one, so is $\hat\beta+v$ for every $v\in\mathcal N(X)$. To write them down we need a substitute for the inverse.
A generalised inverse (g-inverse) of a matrix $A$ is any matrix $A^-$ with $AA^-A=A$. The Moore–Penrose inverse $A^+$ is the unique g-inverse that also satisfies $A^+AA^+=A^+$ and makes $AA^+$ and $A^+A$ symmetric. Write $G=(X^{\mathsf T}X)^-$ for a g-inverse of $X^{\mathsf T}X$. The matrix $$H=X(X^{\mathsf T}X)^-X^{\mathsf T}$$ is the hat matrix (it "puts the hat on $y$": $\hat y=Hy$).
A g-inverse always exists. If $A$ has rank $r$ and is written as $A=P\begin{pmatrix}D&0\\0&0\end{pmatrix}Q$ with $P,Q$ invertible and $D$ an invertible $r\times r$ matrix, then $Q^{-1}\begin{pmatrix}D^{-1}&0\\0&0\end{pmatrix}P^{-1}$ is a g-inverse. In practice, for symmetric $A=X^{\mathsf T}X$: pick an invertible $r\times r$ principal submatrix, invert it, put the inverse in the same position and fill the rest with zeros. For any g-inverse $G$, $b=GX^{\mathsf T}y$ solves the normal equations, because $X^{\mathsf T}XGX^{\mathsf T}y=X^{\mathsf T}XGX^{\mathsf T}Xc=X^{\mathsf T}Xc=X^{\mathsf T}y$, where $c$ is any solution.
For any g-inverse $G$ of $X^{\mathsf T}X$:
- $XGX^{\mathsf T}X=X$ and $X^{\mathsf T}XGX^{\mathsf T}=X^{\mathsf T}$;
- $H=XGX^{\mathsf T}$ does not depend on the choice of $G$, and $H$ is symmetric and idempotent ($H^2=H$);
- $HX=X$, $\mathcal C(H)=\mathcal C(X)$ and $\operatorname{rank}H=\operatorname{tr}H=r$;
- $I-H$ is symmetric and idempotent, of rank $n-r$, $(I-H)X=0$, and $\hat y=Hy$, $e=(I-H)y$, $\operatorname{RSS}=y^{\mathsf T}(I-H)y$.
So $H$ is the orthogonal projection onto $\mathcal C(X)$ and $I-H$ the orthogonal projection onto its orthogonal complement.
Proof. (1) Put $B=XGX^{\mathsf T}X-X$. Then $X^{\mathsf T}B=X^{\mathsf T}XGX^{\mathsf T}X-X^{\mathsf T}X=0$. Since $B^{\mathsf T}=X^{\mathsf T}XG^{\mathsf T}X^{\mathsf T}-X^{\mathsf T}$, we get $B^{\mathsf T}B=X^{\mathsf T}XG^{\mathsf T}(X^{\mathsf T}B)-X^{\mathsf T}B=0$, and a matrix with $B^{\mathsf T}B=0$ is zero (its columns have zero length). The second identity is the transpose of the first applied to the g-inverse $G^{\mathsf T}$ (transposing $X^{\mathsf T}XGX^{\mathsf T}X=X^{\mathsf T}X$ shows that $G^{\mathsf T}$ is also a g-inverse of the symmetric matrix $X^{\mathsf T}X$).
(2) Let $G_1,G_2$ be g-inverses. By (1) applied to $G_2^{\mathsf T}$, $X^{\mathsf T}=X^{\mathsf T}XG_2^{\mathsf T}X^{\mathsf T}$. Hence $$XG_1X^{\mathsf T}=XG_1\left(X^{\mathsf T}XG_2^{\mathsf T}X^{\mathsf T}\right)=\left(XG_1X^{\mathsf T}X\right)G_2^{\mathsf T}X^{\mathsf T}=XG_2^{\mathsf T}X^{\mathsf T}.$$ With $G_1=G_2=G$ this says $XGX^{\mathsf T}=XG^{\mathsf T}X^{\mathsf T}=(XGX^{\mathsf T})^{\mathsf T}$, so $H$ is symmetric; then for general $G_1,G_2$, $XG_1X^{\mathsf T}=XG_2^{\mathsf T}X^{\mathsf T}=XG_2X^{\mathsf T}$, so $H$ does not depend on $G$. Idempotence: $H^2=(XGX^{\mathsf T}X)GX^{\mathsf T}=XGX^{\mathsf T}=H$ by (1).
(3) $HX=X$ is (1). Thus $\mathcal C(X)\subseteq\mathcal C(H)$, and $\mathcal C(H)\subseteq\mathcal C(X)$ because $H=X(GX^{\mathsf T})$. For an idempotent matrix the eigenvalues are $0$ or $1$ and, $H$ being symmetric, it is diagonalisable, so $\operatorname{tr}H$ is the number of unit eigenvalues, which is $\operatorname{rank}H=r$.
(4) $(I-H)^2=I-2H+H^2=I-H$, $\operatorname{tr}(I-H)=n-r$ and $(I-H)X=X-X=0$. Since $\hat\beta=GX^{\mathsf T}y$ is a solution, $\hat y=XGX^{\mathsf T}y=Hy$, and $\operatorname{RSS}=y^{\mathsf T}(I-H)^{\mathsf T}(I-H)y=y^{\mathsf T}(I-H)y$. A symmetric idempotent matrix $P$ is the orthogonal projection onto $\mathcal C(P)$, because $v=Pv+(I-P)v$ and $(Pv)^{\mathsf T}(I-P)w=v^{\mathsf T}(P-P^2)w=0$. ∎
For a full-rank model, $H=X(X^{\mathsf T}X)^{-1}X^{\mathsf T}$, and its diagonal entries $h_{ii}$, the leverages, satisfy $0\le h_{ii}\le1$ and $\sum h_{ii}=p$; §6.3 uses them in regression diagnostics. Theorem 6.2(4) gives the Pythagorean decomposition $$\|y\|^2=\|\hat y\|^2+\|e\|^2,$$ and, when $\mathbf 1\in\mathcal C(X)$ (the model has an intercept), $\sum e_i=\mathbf 1^{\mathsf T}e=0$ and $$\sum_i(y_i-\bar y)^2=\sum_i(\hat y_i-\bar y)^2+\sum_ie_i^2.$$
Estimability and the Gauss–Markov Theorem
Example 2 showed that in a model of less than full rank some functions of $\beta$, such as $\alpha_2-\alpha_1$, are determined by the data while others, such as $\alpha_1$, are not. The reason is not a shortage of data: even with infinitely many observations per group we would learn $\mu+\alpha_1$ and $\mu+\alpha_2$ and nothing more. The parameters $(\mu,\alpha_1,\alpha_2)$ and $(\mu+c,\alpha_1-c,\alpha_2-c)$ give the same distribution of $y$; they are not identifiable.
A linear parametric function $\lambda^{\mathsf T}\beta$ ($\lambda\in\mathbb R^p$) is estimable if it has a linear unbiased estimator: there is $a\in\mathbb R^n$ with $E[a^{\mathsf T}y]=\lambda^{\mathsf T}\beta$ for every $\beta\in\mathbb R^p$. A vector $\Lambda^{\mathsf T}\beta$ ($\Lambda$ a $p\times q$ matrix) is estimable if each of its components is.
In the linear model $E[y]=X\beta$, the following are equivalent:
- $\lambda^{\mathsf T}\beta$ is estimable;
- $\lambda\in\mathcal C(X^{\mathsf T})$, i.e. $\lambda^{\mathsf T}$ is a linear combination of the rows of $X$: $\lambda=X^{\mathsf T}a$ for some $a$;
- $\lambda^{\mathsf T}GX^{\mathsf T}X=\lambda^{\mathsf T}$ for some (equivalently, every) g-inverse $G$ of $X^{\mathsf T}X$;
- $\lambda\perp\mathcal N(X)$: $\lambda^{\mathsf T}v=0$ whenever $Xv=0$;
- $\lambda^{\mathsf T}\beta_1=\lambda^{\mathsf T}\beta_2$ whenever $X\beta_1=X\beta_2$.
Consequently every component of $X\beta$ is estimable, linear combinations of estimable functions are estimable, all of $\beta$ is estimable if and only if $r=p$, and there are exactly $r$ linearly independent estimable functions.
Proof. (1)⇔(2): $E[a^{\mathsf T}y]=a^{\mathsf T}X\beta$, and $a^{\mathsf T}X\beta=\lambda^{\mathsf T}\beta$ for all $\beta$ exactly when $X^{\mathsf T}a=\lambda$.
(2)⇒(3): If $\lambda=X^{\mathsf T}a$, then $\lambda^{\mathsf T}GX^{\mathsf T}X=a^{\mathsf T}XGX^{\mathsf T}X=a^{\mathsf T}X=\lambda^{\mathsf T}$ by Theorem 6.2(1), for every $G$. (3)⇒(2): $\lambda=X^{\mathsf T}(XG^{\mathsf T}\lambda)$.
(2)⇔(4): $\mathcal C(X^{\mathsf T})$ is the orthogonal complement of $\mathcal N(X)$ in $\mathbb R^p$ (the row space is orthogonal to the null space, and their dimensions $r$ and $p-r$ add up to $p$).
(4)⇔(5): put $v=\beta_1-\beta_2$.
The consequences: the $i$-th component of $X\beta$ is $x_i^{\mathsf T}\beta$ and $x_i\in\mathcal C(X^{\mathsf T})$; $\mathcal C(X^{\mathsf T})$ is a subspace; it equals $\mathbb R^p$ iff $r=p$; and its dimension is $r$. ∎
Condition (5) is the identifiability statement: an estimable function is one that is determined by the mean vector $X\beta$. In the one-way model, $\mathcal N(X)$ is spanned by $(1,-1,\dots,-1)$, so by (4), $\lambda^{\mathsf T}\beta=\lambda_0\mu+\sum\lambda_i\alpha_i$ is estimable iff $\lambda_0=\sum_{i=1}^k\lambda_i$. Hence $\mu+\alpha_i$ is estimable, every contrast $\sum c_i\alpha_i$ with $\sum c_i=0$ is estimable, and $\mu$, $\alpha_i$ and $\alpha_1+\alpha_2$ are not. Imposing a side condition such as $\sum n_i\alpha_i=0$ or $\alpha_k=0$ picks one solution of the normal equations and makes every parameter "estimable" in the constrained model, but the meaning of $\alpha_i$ then depends on the constraint; estimable functions have the same estimate under every constraint.
In the Gauss–Markov model of Definition 6.1, let $\lambda^{\mathsf T}\beta$ be estimable and let $\hat\beta$ be any solution of the normal equations. Then:
- $\lambda^{\mathsf T}\hat\beta$ has the same value for every solution $\hat\beta$, and $\lambda^{\mathsf T}\hat\beta=\lambda^{\mathsf T}GX^{\mathsf T}y$ for every g-inverse $G$;
- $\lambda^{\mathsf T}\hat\beta$ is unbiased for $\lambda^{\mathsf T}\beta$, with $\operatorname{Var}(\lambda^{\mathsf T}\hat\beta)=\sigma^2\lambda^{\mathsf T}G\lambda$ (also independent of $G$);
- if $a^{\mathsf T}y$ is any linear unbiased estimator of $\lambda^{\mathsf T}\beta$, then $\operatorname{Var}(a^{\mathsf T}y)\ge\operatorname{Var}(\lambda^{\mathsf T}\hat\beta)$, with equality if and only if $a^{\mathsf T}y=\lambda^{\mathsf T}\hat\beta$ (i.e. $a=XG\lambda$).
So $\lambda^{\mathsf T}\hat\beta$ is the unique best linear unbiased estimator (BLUE) of $\lambda^{\mathsf T}\beta$. For estimable $\lambda_1^{\mathsf T}\beta,\lambda_2^{\mathsf T}\beta$, $\operatorname{Cov}(\lambda_1^{\mathsf T}\hat\beta,\lambda_2^{\mathsf T}\hat\beta)=\sigma^2\lambda_1^{\mathsf T}G\lambda_2$.
Proof. Write $\lambda=X^{\mathsf T}a_0$ (Theorem 6.3). (1) $\lambda^{\mathsf T}\hat\beta=a_0^{\mathsf T}X\hat\beta=a_0^{\mathsf T}\hat y$, and $\hat y$ is the same for every solution (Theorem 6.1(4)). Since $GX^{\mathsf T}y$ is a solution, $\lambda^{\mathsf T}\hat\beta=\lambda^{\mathsf T}GX^{\mathsf T}y$.
(3) first. A linear estimator $a^{\mathsf T}y$ is unbiased iff $X^{\mathsf T}a=\lambda$, and then $\operatorname{Var}(a^{\mathsf T}y)=a^{\mathsf T}(\sigma^2I)a=\sigma^2\|a\|^2$. Split $a=Ha+(I-H)a$, an orthogonal decomposition. The first part is the same for every unbiased $a$: $$Ha=XGX^{\mathsf T}a=XG\lambda=:a^*.$$ Therefore $$\|a\|^2=\|a^*\|^2+\|(I-H)a\|^2\ge\|a^*\|^2,$$ with equality iff $(I-H)a=0$, i.e. $a=a^*$. And $a^*$ itself is unbiased ($X^{\mathsf T}a^*=X^{\mathsf T}Ha=X^{\mathsf T}a=\lambda$, using $X^{\mathsf T}H=X^{\mathsf T}$), with $$a^{*\mathsf T}y=\lambda^{\mathsf T}G^{\mathsf T}X^{\mathsf T}y=\lambda^{\mathsf T}\hat\beta,$$ because $G^{\mathsf T}$ is a g-inverse too and (1) holds for every g-inverse. So the minimum variance is attained by $\lambda^{\mathsf T}\hat\beta$ and only by it.
(2) Unbiasedness: $$E[\lambda^{\mathsf T}\hat\beta]=a_0^{\mathsf T}E[\hat y]=a_0^{\mathsf T}HX\beta=a_0^{\mathsf T}X\beta=\lambda^{\mathsf T}\beta.$$ Variance, using $a^*=Ha_0$ (take $a=a_0$ above): $$\begin{aligned}\sigma^2\|a^*\|^2&=\sigma^2a_0^{\mathsf T}H^{\mathsf T}Ha_0=\sigma^2a_0^{\mathsf T}Ha_0\\&=\sigma^2a_0^{\mathsf T}XGX^{\mathsf T}a_0=\sigma^2\lambda^{\mathsf T}G\lambda.\end{aligned}$$ The covariance formula follows in the same way from $\operatorname{Cov}(a_1^{*\mathsf T}y,a_2^{*\mathsf T}y)=\sigma^2a_1^{*\mathsf T}a_2^*$. ∎
The proof also gives a practical test: an unbiased linear estimator $a^{\mathsf T}y$ is the BLUE if and only if $a\in\mathcal C(X)$ (then $a=Ha=a^*$). For example, in any model whose design matrix has the group indicators among its columns (or in its column space), a difference of group means is a BLUE as soon as it is unbiased.
Three remarks. First, the theorem needs only the first two moments of $\varepsilon$; no normality and no independence beyond zero correlation. Second, the competition is restricted to linear unbiased estimators: a biased estimator (such as ridge regression) or a nonlinear one can have a smaller mean squared error. Third, the theorem has a matrix form: if $\Lambda^{\mathsf T}\beta$ is estimable and $T=A^{\mathsf T}y$ is any linear unbiased estimator of it, then $\operatorname{Cov}(T)-\operatorname{Cov}(\Lambda^{\mathsf T}\hat\beta)$ is positive semidefinite, because applying (3) to $c^{\mathsf T}\Lambda^{\mathsf T}\beta$ gives $c^{\mathsf T}\operatorname{Cov}(T)c\ge c^{\mathsf T}\operatorname{Cov}(\Lambda^{\mathsf T}\hat\beta)c$ for every $c$. In the full-rank case, $\operatorname{Cov}(\hat\beta)=\sigma^2(X^{\mathsf T}X)^{-1}$.
When the errors are correlated or have unequal variances, least squares remains unbiased but is no longer best. The remedy is to transform the model back to a Gauss–Markov model.
Let $y=X\beta+\varepsilon$ with $X$ of full rank $p$, $E[\varepsilon]=0$ and $\operatorname{Cov}(\varepsilon)=\sigma^2V$, where $V$ is a known positive definite matrix. Then the BLUE of $\beta$ is the generalised least squares estimator $$\tilde\beta=(X^{\mathsf T}V^{-1}X)^{-1}X^{\mathsf T}V^{-1}y,$$ $$\operatorname{Cov}(\tilde\beta)=\sigma^2(X^{\mathsf T}V^{-1}X)^{-1},$$ which minimises $(y-Xb)^{\mathsf T}V^{-1}(y-Xb)$. For $V=\operatorname{diag}(v_1,\dots,v_n)$ this is weighted least squares with weights $1/v_i$.
Proof. Let $V^{1/2}$ be the symmetric positive definite square root of $V$ and $V^{-1/2}$ its inverse. Then $z=V^{-1/2}y=V^{-1/2}X\beta+V^{-1/2}\varepsilon$ is a Gauss–Markov model with design $W=V^{-1/2}X$ (of rank $p$) and $\operatorname{Cov}(V^{-1/2}\varepsilon)=\sigma^2V^{-1/2}VV^{-1/2}=\sigma^2I$. Linear estimators based on $z$ and on $y$ are the same class, because $V^{-1/2}$ is invertible. By Theorem 6.4 the BLUE is $(W^{\mathsf T}W)^{-1}W^{\mathsf T}z=(X^{\mathsf T}V^{-1}X)^{-1}X^{\mathsf T}V^{-1}y$, with covariance $\sigma^2(W^{\mathsf T}W)^{-1}$, and it minimises $\|z-Wb\|^2=(y-Xb)^{\mathsf T}V^{-1}(y-Xb)$. ∎
Estimating $\sigma^2$ and Distribution Theory under Normality
- (Expectation of a quadratic form.) If $E[y]=\mu$, $\operatorname{Cov}(y)=\Sigma$ and $A$ is a constant symmetric matrix, then $E[y^{\mathsf T}Ay]=\operatorname{tr}(A\Sigma)+\mu^{\mathsf T}A\mu$.
- In the Gauss–Markov model, $E[\operatorname{RSS}]=(n-r)\sigma^2$, so $$\hat\sigma^2=\frac{\operatorname{RSS}}{n-r}$$ is an unbiased estimator of $\sigma^2$ (for $n\gt r$).
- $\operatorname{Cov}(\hat y,e)=0$, $\operatorname{Cov}(e)=\sigma^2(I-H)$; in particular $\operatorname{Var}(e_i)=\sigma^2(1-h_{ii})$.
Proof. (1) $y^{\mathsf T}Ay$ is a scalar, so it equals its trace, and $\operatorname{tr}(y^{\mathsf T}Ay)=\operatorname{tr}(Ayy^{\mathsf T})$. Taking expectations, $E[yy^{\mathsf T}]=\Sigma+\mu\mu^{\mathsf T}$, so $E[y^{\mathsf T}Ay]=\operatorname{tr}(A\Sigma)+\operatorname{tr}(A\mu\mu^{\mathsf T})=\operatorname{tr}(A\Sigma)+\mu^{\mathsf T}A\mu$.
(2) $\operatorname{RSS}=y^{\mathsf T}(I-H)y$ with $\mu=X\beta$ and $\Sigma=\sigma^2I$. Since $(I-H)X=0$, the second term vanishes, and $E[\operatorname{RSS}]=\sigma^2\operatorname{tr}(I-H)=(n-r)\sigma^2$ by Theorem 6.2.
(3) Using $H^{\mathsf T}=H=H^2$: $$\operatorname{Cov}(Hy,(I-H)y)=\sigma^2H(I-H)=\sigma^2(H-H^2)=0,$$ $$\operatorname{Cov}((I-H)y)=\sigma^2(I-H)^2=\sigma^2(I-H).\ ∎$$
The divisor is $n-r$, the residual degrees of freedom: $e$ lives in the $(n-r)$-dimensional space $\mathcal C(X)^\perp$. In Example 1, $\hat\sigma^2=1.10/3=0.3667$. Part (3) also warns that residuals are correlated and have unequal variances even when the errors are iid. Under normality much more is true.
Let $y\sim N_n(X\beta,\sigma^2I)$ with $\operatorname{rank}X=r\lt n$, and let $\lambda^{\mathsf T}\beta$ be estimable. Then:
- $\hat y$ and $e$ are independent, hence $\lambda^{\mathsf T}\hat\beta$ and $\operatorname{RSS}$ are independent;
- $\lambda^{\mathsf T}\hat\beta\sim N\left(\lambda^{\mathsf T}\beta,\ \sigma^2\lambda^{\mathsf T}G\lambda\right)$, and for estimable $\Lambda^{\mathsf T}\beta$, $\Lambda^{\mathsf T}\hat\beta\sim N_q(\Lambda^{\mathsf T}\beta,\sigma^2\Lambda^{\mathsf T}G\Lambda)$;
- $\operatorname{RSS}/\sigma^2\sim\chi^2_{n-r}$;
- (full rank) the maximum likelihood estimators are $\hat\beta$ and $\hat\sigma^2_{\text{ML}}=\operatorname{RSS}/n$, and $(\hat\beta,\operatorname{RSS})$ is a complete sufficient statistic, so $\lambda^{\mathsf T}\hat\beta$ and $\operatorname{RSS}/(n-p)$ are UMVUEs.
Proof. Choose an orthonormal basis $q_1,\dots,q_r$ of $\mathcal C(X)$ and extend it by $q_{r+1},\dots,q_n$ to an orthonormal basis of $\mathbb R^n$; let $Q$ be the orthogonal matrix with columns $q_i$. Put $z=Q^{\mathsf T}y$. Because an orthogonal transformation maps $N_n(\mu,\sigma^2I)$ to $N_n(Q^{\mathsf T}\mu,\sigma^2Q^{\mathsf T}Q)=N_n(Q^{\mathsf T}\mu,\sigma^2I)$ (the same rotation invariance that gives the Helmert transformation in §4.1), the $z_i$ are independent $N(q_i^{\mathsf T}X\beta,\sigma^2)$. For $i\gt r$, $q_i\perp\mathcal C(X)$, so $E[z_i]=0$.
Now $y=\sum_iz_iq_i$, the projection onto $\mathcal C(X)$ keeps the first $r$ terms and $I-H$ keeps the rest: $$\hat y=\sum_{i=1}^rz_iq_i,\qquad e=\sum_{i=r+1}^nz_iq_i,\qquad\operatorname{RSS}=\sum_{i=r+1}^nz_i^2.$$ So $\hat y$ and $e$ are functions of disjoint sets of independent variables, which proves (1) (since $\lambda^{\mathsf T}\hat\beta=a_0^{\mathsf T}\hat y$). And $\operatorname{RSS}/\sigma^2$ is a sum of $n-r$ squares of independent $N(0,1)$ variables, which is (3). For (2), $\lambda^{\mathsf T}\hat\beta=\lambda^{\mathsf T}GX^{\mathsf T}y$ is a linear function of a normal vector, hence normal, with the mean and variance of Theorem 6.4; the vector case is the same.
(4) The log-likelihood is $-\frac n2\log(2\pi\sigma^2)-\frac{1}{2\sigma^2}\|y-X\beta\|^2$. For every $\sigma^2$ it is maximised in $\beta$ by the least squares estimate, and then $\frac{d}{d\sigma^2}$ of $-\frac n2\log\sigma^2-\frac{\operatorname{RSS}}{2\sigma^2}$ vanishes at $\sigma^2=\operatorname{RSS}/n$. Expanding $\|y-X\beta\|^2=y^{\mathsf T}y-2\beta^{\mathsf T}X^{\mathsf T}y+\beta^{\mathsf T}X^{\mathsf T}X\beta$ shows a full-rank exponential family in $(X^{\mathsf T}y,y^{\mathsf T}y)$, whose natural parameter $(\beta/\sigma^2,-1/(2\sigma^2))$ ranges over an open set; so $(X^{\mathsf T}y,y^{\mathsf T}y)$, equivalently $(\hat\beta,\operatorname{RSS})$, is complete and sufficient (§4.2). Unbiased functions of it are UMVUEs by the Lehmann–Scheffé theorem. ∎
Under the hypotheses of Theorem 6.7, with $\hat\sigma^2=\operatorname{RSS}/(n-r)$:
- $\dfrac{\lambda^{\mathsf T}\hat\beta-\lambda^{\mathsf T}\beta}{\hat\sigma\sqrt{\lambda^{\mathsf T}G\lambda}}\sim t_{n-r}$, so $\lambda^{\mathsf T}\hat\beta\pm t_{n-r,\alpha/2}\,\hat\sigma\sqrt{\lambda^{\mathsf T}G\lambda}$ is a $100(1-\alpha)\%$ confidence interval for $\lambda^{\mathsf T}\beta$;
- $\left[\dfrac{\operatorname{RSS}}{\chi^2_{n-r,\alpha/2}},\ \dfrac{\operatorname{RSS}}{\chi^2_{n-r,1-\alpha/2}}\right]$ is a $100(1-\alpha)\%$ confidence interval for $\sigma^2$;
- for estimable $\Lambda^{\mathsf T}\beta$ with $\Lambda$ of rank $q$, writing $u=\Lambda^{\mathsf T}\hat\beta-\theta$, the set $$\left\{\theta\in\mathbb R^q:\ u^{\mathsf T}(\Lambda^{\mathsf T}G\Lambda)^{-1}u\le q\hat\sigma^2F_{q,n-r,\alpha}\right\}$$ is a $100(1-\alpha)\%$ confidence ellipsoid for $\Lambda^{\mathsf T}\beta$.
Proof. (1) The numerator divided by $\sigma\sqrt{\lambda^{\mathsf T}G\lambda}$ is $N(0,1)$, $(n-r)\hat\sigma^2/\sigma^2\sim\chi^2_{n-r}$, and they are independent (Theorem 6.7). The ratio is $t_{n-r}$ by the definition of the $t$ law (§4.1), and the interval follows by the pivotal method (§4.3). (2) is the pivotal method with the pivot $\operatorname{RSS}/\sigma^2\sim\chi^2_{n-r}$. (3) The quadratic form divided by $\sigma^2$ is $\chi^2_q$ (proof of Theorem 6.9 below) and independent of $\operatorname{RSS}$, so the form divided by $q\hat\sigma^2$ is $F_{q,n-r}$. ∎
For the straight line, $\operatorname{Var}(\hat\beta_1)=\sigma^2/S_{xx}$, $\operatorname{Var}(\hat\beta_0)=\sigma^2\left(\frac1n+\frac{\bar x^2}{S_{xx}}\right)$, $\operatorname{Cov}(\hat\beta_0,\hat\beta_1)=-\bar x\sigma^2/S_{xx}$, and the mean response $\beta_0+\beta_1x_0$ has BLUE $\hat\beta_0+\hat\beta_1x_0$ with variance $\sigma^2\left(\frac1n+\frac{(x_0-\bar x)^2}{S_{xx}}\right)$. A new observation at $x_0$ is predicted with error variance $\sigma^2\left(1+\frac1n+\frac{(x_0-\bar x)^2}{S_{xx}}\right)$; §6.3 develops these prediction intervals.
Testing Linear Hypotheses
A linear hypothesis is $H_0:\Lambda^{\mathsf T}\beta=d$, where $\Lambda$ is a known $p\times q$ matrix of rank $q$ and $d\in\mathbb R^q$ is known. It is testable if $\Lambda^{\mathsf T}\beta$ is estimable, i.e. every column of $\Lambda$ lies in $\mathcal C(X^{\mathsf T})$; then $q\le r$.
Testability matters because a hypothesis about a non-estimable function is not a statement about the distribution of $y$. In the one-way model "$\alpha_1=0$" can be made true or false by changing the side condition, without changing $X\beta$; but "$\alpha_1=\alpha_2=\cdots=\alpha_k$" is the testable statement that all group means are equal ($q=k-1$ contrasts).
In the normal linear model with $\operatorname{rank}X=r\lt n$, let $H_0:\Lambda^{\mathsf T}\beta=d$ be testable, of rank $q$. Let $\operatorname{RSS}$ be the unrestricted residual sum of squares and $\operatorname{RSS}_0=\min\{\|y-Xb\|^2:\Lambda^{\mathsf T}b=d\}$ the restricted one. Then:
- $\Lambda^{\mathsf T}G\Lambda$ is nonsingular and $$\operatorname{RSS}_0-\operatorname{RSS}=(\Lambda^{\mathsf T}\hat\beta-d)^{\mathsf T}(\Lambda^{\mathsf T}G\Lambda)^{-1}(\Lambda^{\mathsf T}\hat\beta-d);$$
- the statistic $$F=\frac{(\operatorname{RSS}_0-\operatorname{RSS})/q}{\operatorname{RSS}/(n-r)}$$ has the $F_{q,n-r}$ distribution under $H_0$, and the size-$\alpha$ test rejects when $F\gt F_{q,n-r,\alpha}$;
- in general $F$ has the noncentral $F$ law $F_{q,n-r}(\delta)$ with noncentrality $\delta=(\Lambda^{\mathsf T}\beta-d)^{\mathsf T}(\Lambda^{\mathsf T}G\Lambda)^{-1}(\Lambda^{\mathsf T}\beta-d)/\sigma^2$, and the power increases with $\delta$;
- the test is the likelihood ratio test: $\Lambda(y)=(\operatorname{RSS}/\operatorname{RSS}_0)^{n/2}$ is a decreasing function of $F$.
For $q=1$, $H_0:\lambda^{\mathsf T}\beta=d$, $F=t^2$ with $t=(\lambda^{\mathsf T}\hat\beta-d)/(\hat\sigma\sqrt{\lambda^{\mathsf T}G\lambda})$.
Proof. We give the proof for a full-rank model, $G=(X^{\mathsf T}X)^{-1}$; the general case reduces to it by reparametrising with $r$ independent estimable functions. Put $W=\Lambda^{\mathsf T}(X^{\mathsf T}X)^{-1}\Lambda$, which is positive definite since $\Lambda$ has rank $q$ and $(X^{\mathsf T}X)^{-1}$ is positive definite.
(1) Minimise $\|y-Xb\|^2$ subject to $\Lambda^{\mathsf T}b=d$ with a Lagrange multiplier $2\gamma$: $X^{\mathsf T}Xb-X^{\mathsf T}y+\Lambda\gamma=0$, so $b=\hat\beta-(X^{\mathsf T}X)^{-1}\Lambda\gamma$, and the constraint gives $\Lambda^{\mathsf T}\hat\beta-W\gamma=d$. Hence $$\tilde\beta=\hat\beta-(X^{\mathsf T}X)^{-1}\Lambda W^{-1}(\Lambda^{\mathsf T}\hat\beta-d).$$ By Theorem 6.1(3), $\operatorname{RSS}_0=\|y-X\tilde\beta\|^2=\operatorname{RSS}+\|X(\hat\beta-\tilde\beta)\|^2$. Write $u=\Lambda^{\mathsf T}\hat\beta-d$ and $M=(X^{\mathsf T}X)^{-1}$. Then $X(\hat\beta-\tilde\beta)=XM\Lambda W^{-1}u$, and $$\|X(\hat\beta-\tilde\beta)\|^2=u^{\mathsf T}W^{-1}\Lambda^{\mathsf T}MX^{\mathsf T}XM\Lambda W^{-1}u,$$ which equals $u^{\mathsf T}W^{-1}WW^{-1}u=u^{\mathsf T}W^{-1}u$, since $MX^{\mathsf T}XM=M$ and $\Lambda^{\mathsf T}M\Lambda=W$. (The minimum is attained because $S$ is a convex quadratic and the identity $S(b)=\operatorname{RSS}+\|X(b-\hat\beta)\|^2$ shows that $\tilde\beta$, the point of the affine set closest to $\hat\beta$ in the $X^{\mathsf T}X$ metric, minimises it.)
(2), (3) By Theorem 6.7, $u=\Lambda^{\mathsf T}\hat\beta-d\sim N_q(\Lambda^{\mathsf T}\beta-d,\sigma^2W)$. Then $W^{-1/2}u/\sigma\sim N_q(W^{-1/2}(\Lambda^{\mathsf T}\beta-d)/\sigma,I_q)$, so $u^{\mathsf T}W^{-1}u/\sigma^2=\|W^{-1/2}u/\sigma\|^2$ is a sum of $q$ squares of independent unit-variance normals: $\chi^2_q$ under $H_0$, and noncentral $\chi^2_q(\delta)$ in general. It is a function of $\hat\beta$, hence independent of $\operatorname{RSS}/\sigma^2\sim\chi^2_{n-r}$. The ratio of independent $\chi^2$ variables divided by their degrees of freedom is $F_{q,n-r}$ (noncentral when $\delta\gt0$). A noncentral $\chi^2_q(\delta)$ is stochastically increasing in $\delta$, so the power increases with $\delta$.
(4) Maximising the normal likelihood with $\sigma^2$ free gives $\sup L=(2\pi e\,\operatorname{RSS}/n)^{-n/2}$ (Theorem 6.7(4)), and under $H_0$ the same with $\operatorname{RSS}_0$. So $\Lambda=(\operatorname{RSS}/\operatorname{RSS}_0)^{n/2}=\left(1+\frac{q}{n-r}F\right)^{-n/2}$, a decreasing function of $F$; rejecting for small $\Lambda$ is rejecting for large $F$ (as for the normal-theory tests in §5.1). For $q=1$, $W=\lambda^{\mathsf T}G\lambda$ and $F=u^2/(W\hat\sigma^2)=t^2$. ∎
- Check testability. Each row of $\Lambda^{\mathsf T}$ must be a combination of rows of $X$; count the number $q$ of linearly independent constraints.
- Full model. Fit by least squares: $\operatorname{RSS}$ on $n-r$ degrees of freedom.
- Reduced model. Either substitute the constraints into the model and refit, giving $\operatorname{RSS}_0$ on $n-r+q$ degrees of freedom, or compute $\operatorname{RSS}_0-\operatorname{RSS}$ by the quadratic form of Theorem 6.9(1). For one constraint, use $t$.
- Decide. $F=\dfrac{(\operatorname{RSS}_0-\operatorname{RSS})/q}{\operatorname{RSS}/(n-r)}$ against $F_{q,n-r,\alpha}$. The numerator degrees of freedom are the difference of the residual degrees of freedom of the two models.
Let $y\sim N_n(\mu,\sigma^2I_n)$ and suppose $y^{\mathsf T}y=\sum_{i=1}^kQ_i$, where $Q_i=y^{\mathsf T}A_iy$ with $A_i$ symmetric of rank $r_i$. Then the following are equivalent:
- $r_1+\cdots+r_k=n$;
- each $A_i$ is idempotent;
- $A_iA_j=0$ for all $i\ne j$.
When they hold, $Q_1,\dots,Q_k$ are independent and $Q_i/\sigma^2\sim\chi^2_{r_i}(\mu^{\mathsf T}A_i\mu/\sigma^2)$ (central when $A_i\mu=0$). More generally, for symmetric $A$, $y^{\mathsf T}Ay/\sigma^2$ has a (noncentral) $\chi^2$ law if and only if $A$ is idempotent, and then the degrees of freedom are $\operatorname{rank}A=\operatorname{tr}A$.
The proof, by simultaneous diagonalisation, is given with the distribution of quadratic forms in §7.1. Cochran's theorem is the bookkeeping device of the analysis of variance: whenever a total sum of squares is split into pieces whose degrees of freedom add up, the pieces are independent $\chi^2$ variables, and their ratios are $F$ variables. In this section, $y^{\mathsf T}y=y^{\mathsf T}Hy+y^{\mathsf T}(I-H)y$ with ranks $r+(n-r)=n$ is the simplest instance; §6.2 uses finer splittings.
- Unique fit, non-unique coefficients. When $r\lt p$ the normal equations have infinitely many solutions (Example 2), but $\hat y$, $e$, $\operatorname{RSS}$ and every estimable $\lambda^{\mathsf T}\hat\beta$ are unique. A coefficient printed by software for a rank-deficient model depends on the constraint the program chose.
- Individual effects are not estimable. In $y_{ij}=\mu+\alpha_i+\varepsilon_{ij}$, neither $\mu$ nor $\alpha_i$ nor $\alpha_1+\alpha_2$ is estimable; $\mu+\alpha_i$ and contrasts $\sum c_i\alpha_i$ ($\sum c_i=0$) are. More data per group do not help.
- Gauss–Markov does not need normality — and does not give normality. It needs $E\varepsilon=0$ and $\operatorname{Cov}\varepsilon=\sigma^2I$ only (Interactive 2 with uniform or Laplace errors). The exact $t$- and $F$-distributions, however, do need normality.
- "Best" means best among linear unbiased estimators. A biased estimator can have smaller mean squared error: in $y_i=\beta+\varepsilon_i$, $0.9\bar y$ has MSE $0.81\sigma^2/n+0.01\beta^2$, smaller than $\sigma^2/n$ when $\beta^2\lt19\sigma^2/n$. With non-normal errors a nonlinear unbiased estimator can also win (for uniform errors the midrange beats $\bar y$ for a location parameter).
- Heteroscedastic or correlated errors. Least squares stays unbiased but is no longer BLUE, and the usual formula $\sigma^2\lambda^{\mathsf T}G\lambda$ for its variance is wrong. Use generalised or weighted least squares (Theorem 6.5).
- Residuals are not the errors. They are correlated, have variances $\sigma^2(1-h_{ii})$, and sum to zero when there is an intercept (Example 7). Without an intercept $\sum e_i$ need not be $0$ (Exercise I.3).
- Divide by $n-r$, not $n$ or $n-p$. $\operatorname{RSS}/n$ is the MLE and is biased; with a rank-deficient design the correct divisor uses the rank $r$, not the number of parameters $p$.
- Only testable hypotheses can be tested. "$\alpha_1=0$" in the one-way model has no meaning that is invariant under the choice of side condition; the numerator degrees of freedom of an $F$-test count independent estimable constraints.
- Joint and separate tests can disagree. With strongly correlated estimates, $H_0:\beta_1=\beta_2=0$ can be rejected by the $F$-test while neither individual $t$-test rejects, and vice versa: the confidence ellipse of Theorem 6.8(3) is not the rectangle of two intervals.
- Uncorrelated is not independent. The Gauss–Markov model asks only for uncorrelated errors; independence of $\hat\beta$ and $\operatorname{RSS}$ (Theorem 6.7) is a consequence of joint normality.
Exercises
- 1 Fit $y=\beta_0+\beta_1x$ to $x=(0,1,2,3)$, $y=(1,3,2,6)$. Give $\hat\beta$, the residuals and $\operatorname{RSS}$.
- 2 Fit the line through the origin $y=\beta x$ to the same data; give $\hat\beta$ and $\operatorname{RSS}$.
- 3 Show that residuals sum to zero whenever $\mathbf 1\in\mathcal C(X)$. Using part 2, show that they need not sum to zero otherwise.
- 4 Prove directly that $\mathcal N(X^{\mathsf T}X)=\mathcal N(X)$ and deduce $\operatorname{rank}(X^{\mathsf T}X)=\operatorname{rank}(X)$.
- 5 Show that if $G$ is a g-inverse of a symmetric matrix $A$, so is $G^{\mathsf T}$, and so is $GAG^{\mathsf T}$, which is symmetric.
- 6 Find a g-inverse and the Moore–Penrose inverse of $A=\begin{pmatrix}2&2\\2&2\end{pmatrix}$.
- 7 Show that in the one-way model with $k$ groups of sizes $n_1,\dots,n_k$ the hat matrix is block diagonal with blocks $\frac{1}{n_i}J_{n_i}$ ($J$ the matrix of ones), so that $\hat y_{ij}=\bar y_i$ and $\operatorname{tr}H=k$.
- 8 Prove that when the model has an intercept, $\sum(y_i-\bar y)^2=\sum(\hat y_i-\bar y)^2+\sum e_i^2$, and that $\sum\hat y_i=\sum y_i$.
- 1 In the one-way model with $k=3$ groups, which of $\mu$, $\alpha_1$, $\alpha_1-\alpha_2$, $\mu+\alpha_3$, $\alpha_1+\alpha_2-2\alpha_3$, $\alpha_1+\alpha_2$ are estimable?
- 2 Prove that $\lambda^{\mathsf T}\beta$ is estimable if and only if $\lambda^{\mathsf T}GX^{\mathsf T}X=\lambda^{\mathsf T}$, where $G$ is any g-inverse of $X^{\mathsf T}X$.
- 3 In the additive two-way model $y_{ij}=\mu+\alpha_i+\beta_j+\varepsilon_{ij}$ with $a=2$ rows, $b=3$ columns and one observation per cell, find the BLUE of $\alpha_1-\alpha_2$ and its variance.
- 4 In $y_i=\beta x_i+\varepsilon_i$ (Gauss–Markov errors, $\sum x_i\ne0$), show that $\sum y_i/\sum x_i$ is unbiased and find its efficiency relative to the BLUE. Evaluate for $x=(1,2,3)$.
- 5 Prove the matrix form of the Gauss–Markov theorem: for estimable $\Lambda^{\mathsf T}\beta$ and any linear unbiased $T=A^{\mathsf T}y$, $\operatorname{Cov}(T)-\operatorname{Cov}(\Lambda^{\mathsf T}\hat\beta)$ is positive semidefinite.
- 6 In $y_i=\beta x_i+\varepsilon_i$ with $x_i\gt0$, uncorrelated errors and $\operatorname{Var}(\varepsilon_i)=\sigma^2x_i$, find the BLUE of $\beta$ and its variance.
- 7 Suppose the model has an intercept, is of full rank, and $\operatorname{Cov}(\varepsilon)=\sigma^2\left[(1-\rho)I+\rho J\right]$ with $-\frac1{n-1}\lt\rho\lt1$ (equicorrelated errors). Prove that ordinary least squares coincides with generalised least squares.
- 8 For the design $x=(1,2,3,4,5)$, show that $(y_4+y_5-y_1-y_2)/6$ is an unbiased estimator of the slope and find its efficiency.
- 9 Prove that the BLUE is unique: if $a^{\mathsf T}y$ and $c^{\mathsf T}y$ are both BLUEs of $\lambda^{\mathsf T}\beta$, then $a=c$.
- 1 Prove that $E[y^{\mathsf T}Ay]=\operatorname{tr}(A\Sigma)+\mu^{\mathsf T}A\mu$ for symmetric $A$, without assuming normality.
- 2 Use part 1 to show $E[\operatorname{RSS}]=(n-r)\sigma^2$ in any Gauss–Markov model. What is $E[y^{\mathsf T}Hy]$?
- 3 For the data of Example 1, find a $95\%$ confidence interval for the intercept $\beta_0$.
- 4 (Calibration.) A spectrometer is calibrated with $n=12$ standards: $\bar x=5$, $S_{xx}=50$, $\hat\beta_0=0.4$, $\hat\beta_1=2.1$, $\operatorname{RSS}=4.0$ (absorbance units). Assuming normal errors, find $\hat\sigma^2$, a $95\%$ interval for the slope, a $95\%$ interval for the mean absorbance at $x_0=8$, and a $95\%$ prediction interval for one new reading at $x_0=8$.
- 5 Show that the MLE $\operatorname{RSS}/n$ of $\sigma^2$ has bias $-r\sigma^2/n$ and compare its MSE with that of $\hat\sigma^2=\operatorname{RSS}/(n-r)$ under normality.
- 6 Show that $\operatorname{Cov}(\hat\beta,e)=0$ in every full-rank Gauss–Markov model. Give an example in which $\hat\beta$ and $e$ are uncorrelated but not independent.
- 7 Find a $90\%$ confidence interval for $\sigma^2$ for the data of Example 1.
- 8 In simple regression show that $\operatorname{Cov}(\hat\beta_0,\hat\beta_1)=-\bar x\sigma^2/S_{xx}$. For which designs are $\hat\beta_0$ and $\hat\beta_1$ uncorrelated? Find $\operatorname{Corr}(\hat\beta_0,\hat\beta_1)$ for $x=(1,2,3,4,5)$.
- 1 For the $2^2$ data of Example 6, test $H_0:\beta_2=0$ at $5\%$ by a $t$-test and by the $F$-test, and check $F=t^2$.
- 2 Prove in general that for one constraint the $F$ statistic of Theorem 6.9 equals $t^2$.
- 3 Write $H_0:\alpha_1=\alpha_2=\alpha_3$ in the one-way model as $\Lambda^{\mathsf T}\beta=0$ with $\Lambda$ of rank $2$ and verify that it is testable. What are the degrees of freedom of the $F$-test if $N=15$?
- 4 Show that $H_0:\alpha_1=0$ is not testable in the one-way model, by exhibiting two parameter vectors with the same $X\beta$, one satisfying $H_0$ and one not.
- 5 Prove that the likelihood ratio for $H_0:\Lambda^{\mathsf T}\beta=d$ in the normal model is $\left(1+\frac{q}{n-r}F\right)^{-n/2}$.
- 6 (Two calibration lines.) Two instruments are each calibrated at $x=1,\dots,5$. Instrument A gives $S_{xx}=10$, $S_{xy}=17$, $S_{yy}=30$; instrument B gives $S_{xx}=10$, $S_{xy}=12$, $S_{yy}=16$ (sums of squares about each instrument's own means). Fitting separate lines, test at $5\%$ whether the slopes are equal.
- 7 In the one-way model, write the identity $$\sum_{ij}y_{ij}^2=N\bar y^2+\sum_in_i(\bar y_i-\bar y)^2+\sum_{ij}(y_{ij}-\bar y_i)^2$$ as $y^{\mathsf T}A_1y+y^{\mathsf T}A_2y+y^{\mathsf T}A_3y$, give the ranks, and state what Cochran's theorem yields.
- 8 For the design of Example 1 with $\sigma=1$ and true slope $\beta_1=0.5$, find the noncentrality of the $F$-test of $\beta_1=0$ and its power at $5\%$ (use software for the noncentral $F$).
E1. CSIR Part B GATE ST In the model $y_i=\beta_0+\beta_1x_i+\varepsilon_i$, $i=1,\dots,5$, with uncorrelated errors of common variance and $x=(1,2,3,4,5)$, the correlation between the least squares estimators $\hat\beta_0$ and $\hat\beta_1$ is closest to (A) $0$; (B) $-0.30$; (C) $-0.90$; (D) $-0.55$.
E2. CSIR Part B Let $y_1,y_2,y_3$ be uncorrelated with common variance and $E[y_1]=\beta_1+\beta_2$, $E[y_2]=\beta_2+\beta_3$, $E[y_3]=\beta_1+2\beta_2+\beta_3$. Which of the following parametric functions is estimable? (A) $\beta_2$; (B) $\beta_1+\beta_3$; (C) $\beta_1-\beta_3$; (D) $\beta_1$.
E3. CSIR Part B GATE ST In a normal linear model with $n=25$ observations and a design matrix of rank $5$, the residual sum of squares is $60$. Imposing two linearly independent testable constraints raises the residual sum of squares to $84$. The $F$ statistic for testing the constraints, and its degrees of freedom, are (A) $4.0$ on $(2,20)$; (B) $2.0$ on $(2,20)$; (C) $4.0$ on $(2,23)$; (D) $0.4$ on $(20,2)$.
E4. CSIR Part C Consider $y=X\beta+\varepsilon$ with $X$ an $n\times p$ matrix of rank $r\lt p\lt n$, $E\varepsilon=0$, $\operatorname{Cov}\varepsilon=\sigma^2I$. Which of the following are true? (One or more options may be correct.) (A) $X^{\mathsf T}X$ is singular. (B) All solutions of the normal equations give the same vector of fitted values. (C) Every coordinate $\beta_j$ is estimable. (D) $\operatorname{RSS}/(n-r)$ is an unbiased estimator of $\sigma^2$.
E5. CSIR Part C GATE ST In the normal linear model $y\sim N_n(X\beta,\sigma^2I)$ with $X$ of full rank $p\lt n$, let $\hat\beta$ be the least squares estimator and $e$ the residual vector. Which of the following are true? (A) $\hat\beta$ and $e$ are independent. (B) $e_1,\dots,e_n$ are independent. (C) $\|e\|^2/\sigma^2\sim\chi^2_{n-p}$. (D) Each $\hat\beta_j$ is the UMVUE of $\beta_j$.
E6. CSIR Part C Let $y_i=\beta x_i+\varepsilon_i$, $i=1,\dots,n$ ($n\ge2$), with known $x_i\gt0$ not all equal and uncorrelated errors of mean $0$ and variance $\sigma^2$. Which of the following are true? (A) $\sum y_i/\sum x_i$ is unbiased for $\beta$. (B) $\frac1n\sum y_i/x_i$ is unbiased for $\beta$. (C) $\sum x_iy_i/\sum x_i^2$ has variance $\sigma^2/\sum x_i^2$. (D) $\operatorname{Var}\left(\sum y_i/\sum x_i\right)\le\operatorname{Var}\left(\sum x_iy_i/\sum x_i^2\right)$.
Suggested plan — 3 lectures × 60 min.
- Least squares as projection. Definition 6.1 with the three standard designs written out as matrices. Theorem 6.1 in full (the Pythagoras argument), then Example 1 on the board. Project Interactive 1, move $y_2$ and let the class predict where $\hat y$ goes. Generalised inverses and Theorem 6.2 (prove (1) and (2); state (3)–(4)); finish with Example 2.
- Estimability and Gauss–Markov. Theorem 6.3 with the identifiability interpretation; one-way contrasts; Example 3 as a class exercise (ten minutes in pairs). Theorem 6.4 with the full projection proof, Example 4, then Interactive 2: normal, then Laplace errors (least squares still wins), then heteroscedastic (it does not). State Aitken's theorem (Theorem 6.5) and prove it by whitening.
- Normal theory and tests. Theorem 6.6 via the trace lemma; Theorem 6.7 via the orthonormal basis; Example 5. Theorem 6.9 with the Lagrange computation; Example 6, including the check by refitting. State Cochran's theorem and point forward to §6.2. Close with Example 7 and E5.
Discussion questions. (1) Software reports $\hat\alpha_3=0$ for a one-way model. Is the third treatment ineffective? What would you report instead? (2) Gauss–Markov says least squares is best. Why do statisticians use ridge regression or robust regression at all? (3) Why is the right divisor for $\sigma^2$ the rank of $X$ and not the number of columns?
Misconceptions to watch for. Believing that Gauss–Markov requires normal errors, or that it compares least squares with all estimators; treating residuals as independent copies of the errors; dividing $\operatorname{RSS}$ by $n$; testing or estimating non-estimable effects; counting the numerator degrees of freedom as the number of parameters mentioned in $H_0$ rather than the number of independent estimable constraints.
Suggested homework. I(1, 3, 6, 7), II(1, 3, 4, 6, 7), III(2, 4, 6, 8), IV(1, 3, 4, 6); Exam corner E2, E4, E6.
Research question. How much precision does a well-chosen design buy? For a straight line on the interval $[0,1]$ with $n=10$ observations, compare designs by $\operatorname{Var}(\hat\beta_1)=\sigma^2/S_{xx}$ and by the worst-case variance of the fitted mean, $\max_{x_0\in[0,1]}\operatorname{Var}(\hat\beta_0+\hat\beta_1x_0)$.
Why it matters. In dose-finding studies, materials testing and sensor calibration, the settings $x_i$ are chosen by the experimenter before any data are seen. The Gauss–Markov formulas show that the precision of the estimates depends on the design through $X^{\mathsf T}X$ alone, so designs can be compared, and optimised, before the experiment. This is the starting point of optimal design theory (D-, A- and G-optimality), widely used in engineering and pharmaceutical research.
What to do.
- For three designs — equally spaced points $0,\frac19,\dots,1$; five points at each end; and three at each end with four in the middle — compute $S_{xx}$, $\operatorname{Var}(\hat\beta_1)/\sigma^2$ and $\det(X^{\mathsf T}X)$.
- For each design plot $\operatorname{Var}(\hat\beta_0+\hat\beta_1x_0)/\sigma^2=\frac1n+\frac{(x_0-\bar x)^2}{S_{xx}}$ on $[0,1]$ and record its maximum.
- Now suppose the truth is quadratic, $E[y]=1+x+cx^2$. Simulate each design with $c=0,\ 0.5,\ 1$ and record how often the lack of linearity can be detected at all. (With two support points it never can.)
- Write a paragraph on the trade-off between efficiency under the assumed model and robustness to a wrong model.
Tools: a spreadsheet, R or Python (numpy.linalg); Interactive 2 for intuition about variances of linear estimators.
Hand in: A table of the three designs' criteria, the variance curves, and a recommendation with reasons.
Going further. Read about the Kiefer–Wolfowitz equivalence theorem, which shows that D-optimal and G-optimal approximate designs coincide, and find the D-optimal design for a quadratic on $[-1,1]$ (weights $\frac13$ at $-1,0,1$).
🧪 Self-check quiz
Q1. In the linear model $E[y]=X\beta$, the function $\lambda^{\mathsf T}\beta$ is estimable if and only if
Q2. In a Gauss–Markov model with $n$ observations and $\operatorname{rank}X=r$, $E[\operatorname{RSS}]$ equals
Q3. The hat matrix $H=X(X^{\mathsf T}X)^-X^{\mathsf T}$
Q4. In a normal linear model with $n=30$ and $\operatorname{rank}X=6$, a testable hypothesis consisting of $3$ linearly independent constraints is tested by an $F$ statistic with degrees of freedom
Q5. The Gauss–Markov theorem requires