Chapter 10 — Linear Models, Gauss–Markov & ANOVA
One model behind many methods: least squares and why it is optimal, which parameters a design can estimate at all, exact tests of linear hypotheses, and the analysis of variance and covariance with fixed, random and mixed effects — all as statements about $y=X\beta+\varepsilon$.
In 1805 Legendre proposed to fit the orbit of a comet by choosing the parameters that make the sum of squared errors as small as possible, and Gauss soon explained why this "method of least squares" is the natural choice when the errors are random: among all linear unbiased estimates it has the smallest variance. More than a century later, at the Rothamsted agricultural station, R. A. Fisher faced a problem that looks quite different: how to compare many fertilisers and varieties on plots of land that differ among themselves. His answer, the analysis of variance, splits the total variation of the yields into parts due to treatments, to blocks of land and to error, and compares them with an $F$ ratio. Both ideas are the same mathematics. The expected response is a linear function of unknown parameters, $E[y]=X\beta$, and every estimate, interval and test is a statement about projecting the data vector $y$ onto the column space of the design matrix $X$.
This chapter develops that single framework and its first great application. Section 10.1 sets up the Gauss–Markov model, derives the normal equations, decides which parametric functions are estimable when $X$ does not have full rank, proves the Gauss–Markov theorem (least squares gives the best linear unbiased estimators) and Aitken's extension to correlated errors, and, under normality, derives the $t$-intervals and the $F$-test of a general linear hypothesis, with Cochran's theorem in the background. Section 10.2 applies the theory to designed comparisons: one- and two-way analysis of variance, contrasts and multiple comparisons, interaction, fixed, random and mixed effects with expected mean squares and variance components, and the analysis of covariance. The same machinery runs regression with continuous predictors in Chapter 11 and the classical experimental designs in Chapter 13.
- write ANOVA, regression and covariance models in the form $y=X\beta+\varepsilon$, solve the normal equations with a generalised inverse, and use the hat matrix and its properties;
- decide whether a parametric function is estimable by the row-space criterion, and find its best linear unbiased estimator and its variance;
- prove the Gauss–Markov theorem, state exactly what it assumes, and use generalised least squares (Aitken's theorem) when the errors are correlated or heteroscedastic;
- estimate $\sigma^2$ without bias, derive the exact distributions under normality, and construct $t$-intervals and $F$-tests for testable linear hypotheses;
- carry out one- and two-way analyses of variance, with contrasts, multiple comparisons (Bonferroni, Scheffé, Tukey) and tests for interaction, and interpret an interaction correctly;
- write down expected mean squares for fixed, random and mixed models, choose the correct $F$ denominator, estimate variance components and the intraclass correlation, and adjust treatment comparisons for a covariate by the analysis of covariance.
How this is examined
Linear models and the analysis of variance appear regularly in CSIR NET papers. Part B questions usually ask for a number: the variance of a least squares estimator or of a contrast, an unbiased estimate of $\sigma^2$ with the right degrees of freedom, the rank of a design matrix, a value of $F$ from a partly filled ANOVA table, the degrees of freedom of an interaction or error line, or an estimate of a variance component or of the intraclass correlation. Part C questions, where several options may be correct, test the structure of the theory: which parametric functions are estimable in a design that is not of full rank; which statements about fitted values, residuals and the hat matrix are true; what the Gauss–Markov theorem assumes and what it concludes; whether a hypothesis is testable; which mean square is the correct denominator in a random or mixed model; and what a covariate adjustment changes. The wrong options come from the ⚠️ boxes of the two sections: individual effects treated as estimable, residuals treated as independent errors, $\operatorname{RSS}$ divided by $n$ instead of $n-r$, many $t$-tests used in place of one ANOVA, "no main effect" read as "no effect" despite a strong interaction, and MSE used where MSAB is needed. GATE ST covers the same material, often with numerical answers.
Sections in this chapter
Gauss–Markov Models, Estimability and BLUE
The linear model, least squares as orthogonal projection, normal equations and generalised inverses, the hat matrix, estimable functions, the Gauss–Markov and Aitken theorems, unbiased estimation of $\sigma^2$, distribution theory under normality, confidence intervals, the $F$-test of a linear hypothesis and Cochran's theorem.
Analysis of Variance and Covariance
One-way ANOVA with contrasts and multiple comparisons (Bonferroni, Scheffé, Tukey); two-way layouts with one and with several observations per cell, and interaction; fixed, random and mixed effects, expected mean squares and variance components; analysis of covariance with one covariate.
This chapter uses the inference chapters throughout: the $\chi^2$, $t$ and $F$ laws, the independence of $\bar X$ and $S^2$ for normal samples and the Helmert transformation (§6.1); maximum likelihood, sufficiency, completeness and the Lehmann–Scheffé theorem (§7.1); pivotal quantities and confidence intervals (§7.2); and likelihood ratio tests with the normal-theory $t$- and $F$-tests (§8.1). From probability you need expectations, covariances and correlation (§2.2). From linear algebra you need rank, null space and column space, orthogonal projections, symmetric matrices and their eigenvalues, and positive definite matrices; the sister course CSIR NET Unit 1: Analysis and Linear Algebra covers them. The multivariate normal distribution and the full theory of quadratic forms, including Cochran's theorem, come later in §12.1; this chapter uses only the facts it states.