Chapter 6 · Introduction

Chapter 6 — Linear Models and Regression

One model, many methods: least squares and its optimality, comparing several means by the analysis of variance, random and mixed effects, adjusting for covariates, and fitting, checking and extending regression models — all as instances of $y=X\beta+\varepsilon$.

In 1805 Legendre proposed to fit the orbit of a comet by choosing the parameters that make the sum of squared errors as small as possible, and a few years later Gauss explained why this "method of least squares" is the natural choice when errors are random. A century later, at the Rothamsted agricultural station, R. A. Fisher faced a different-looking problem: how to compare many fertilisers and varieties on plots of land that differ among themselves. His answer, the analysis of variance, splits the total variation of the yields into parts due to treatments, to blocks of land and to error, and compares them with an $F$ ratio. Both ideas turn out to be the same mathematics. The expected response is a linear function of unknown parameters, $E[y]=X\beta$, and every estimate, interval and test is a statement about projecting the data vector $y$ onto the column space of the design matrix $X$.

This chapter develops that single framework and its main uses. Section 6.1 sets up the Gauss–Markov model, derives the normal equations, decides which parametric functions are estimable when $X$ does not have full rank, proves the Gauss–Markov theorem (least squares gives the best linear unbiased estimators), and, under normality, derives the $t$-intervals and the $F$-test of a general linear hypothesis with Cochran's theorem in the background. Section 6.2 applies the theory to designed comparisons: one- and two-way analysis of variance, contrasts and multiple comparisons, interaction, fixed, random and mixed effects with expected mean squares and variance components, and the analysis of covariance. Section 6.3 turns to regression with continuous predictors: simple and multiple regression, $R^2$, prediction intervals, diagnostics such as leverage and Cook's distance, and logistic regression for binary responses.

Linear model y = Xβ + ε→Normal equations→Projection & hat matrix→Estimability→Gauss–Markov BLUE→σ̂² and normal theory→F-test of Λᵀβ = d→One- & two-way ANOVA→Random & mixed effects→ANCOVA→Regression & diagnostics→Logistic regression
🎯 By the end of this chapter you can…
  • write ANOVA, regression and covariance models in the form $y=X\beta+\varepsilon$, solve the normal equations with a generalised inverse, and use the hat matrix and its properties;
  • decide whether a parametric function is estimable by the row-space criterion, and find its best linear unbiased estimator and variance;
  • prove the Gauss–Markov theorem, state exactly what it assumes, and use generalised least squares when the errors are correlated or heteroscedastic;
  • estimate $\sigma^2$ without bias, derive the exact distributions under normality, and construct $t$-intervals and $F$-tests for testable linear hypotheses;
  • carry out one- and two-way analyses of variance, with contrasts, multiple comparisons and tests for interaction, and interpret an interaction correctly;
  • write down expected mean squares for fixed, random and mixed models, choose the correct $F$ denominator, and estimate variance components and the intraclass correlation;
  • adjust treatment comparisons for a covariate by the analysis of covariance;
  • fit and diagnose simple and multiple regression models and interpret a logistic regression.
Teaser: The Residual Sum of Squares Has One Lowest Point

Eight calibration points and a line through them with slope $b$; the intercept is always the best one for that slope, $\bar y-b\bar x$. The red segments are the residuals. On the right, the residual sum of squares $\operatorname{RSS}(b)=S_{yy}-2bS_{xy}+b^2S_{xx}$ is a parabola in $b$, and the moving dot shows where the current line sits on it. Press Play to sweep the slope, or drag the slider: the lowest point is the least squares slope $\hat b=S_{xy}/S_{xx}$, where the residuals become orthogonal to the regressor. Section 6.1 proves that this works for any number of parameters, and why the result is the best linear unbiased estimate.

0.30
line—
RSS(b)—
Σ xᵢeᵢ (0 at the optimum)—
least squares—

How this is examined

Linear models appear in almost every CSIR NET paper. Part B questions ask for a number: the variance of a least squares estimator or of a contrast, the correlation between $\hat\beta_0$ and $\hat\beta_1$, an unbiased estimate of $\sigma^2$ with the right degrees of freedom, a value of $F$ from a partly filled ANOVA table, the degrees of freedom of an interaction or error line, an estimate of a variance component, or $R^2$ from given sums of squares. Part C questions, where several options may be correct, test understanding of the structure: which parametric functions are estimable in a design that is not of full rank; which statements about fitted values, residuals and the hat matrix are true; what the Gauss–Markov theorem assumes and what it concludes; whether a hypothesis is testable; which mean square is the correct denominator in a random or mixed model; and what leverage, Cook's distance or a logistic coefficient means. The wrong options come from the ⚠️ boxes: individual effects treated as estimable, residuals treated as independent, $\operatorname{RSS}$ divided by $n$, main effects interpreted despite interaction, and MSE used where MSAB is needed. GATE ST covers the same material, often with numerical answers.

Sections in this chapter

✅ Before you start

This chapter uses Chapters 4 and 5 throughout: the $\chi^2$, $t$ and $F$ laws, the independence of $\bar X$ and $S^2$ for normal samples and the Helmert transformation (§4.1); maximum likelihood, sufficiency, completeness and the Lehmann–Scheffé theorem (§4.2); pivotal quantities and confidence intervals (§4.3); and likelihood ratio tests with the normal-theory $t$- and $F$-tests (§5.1). From probability you need expectations, covariances and correlation (§1.4). From linear algebra you need rank, null space and column space, orthogonal projections, symmetric matrices and their eigenvalues, and positive definite matrices; the sister course CSIR NET Unit 1: Analysis and Linear Algebra covers them. The multivariate normal distribution and the full theory of quadratic forms, including the proof of Cochran's theorem, come later in §7.1; this chapter uses only the facts it states.