Chapter 11 — Regression — Linear, Logistic, Diagnostics
The linear model put to work: fitting simple and multiple regressions with honest standard errors and prediction intervals, judging a fit by $R^2$, residuals, leverage and Cook's distance, and modelling a yes-or-no response by logistic regression.
In 1886 Francis Galton plotted the heights of adult children against the heights of their parents and noticed that the children of very tall parents were, on average, tall but less extreme than their parents. He called the effect "regression towards mediocrity", and the word stuck to the whole method of fitting a line that predicts one variable from another. Almost a century later, in 1973, Francis Anscombe published four small data sets that share the same means, the same correlation and the same least squares line, yet look completely different when plotted: one is a clean linear trend, one is a curve, one is a line spoiled by a single outlier, and one owes its entire slope to one isolated point. The moral is the subject of this chapter. Fitting a regression takes one formula; trusting it takes diagnostics.
Section 11.1 starts from the linear model of Chapter 10 and turns it into a working method. It fits the simple and multiple linear regression models, attaches $t$-tests and confidence intervals to every coefficient, separates a confidence interval for the mean response from a prediction interval for a new observation, and measures the fit by $R^2$ and adjusted $R^2$, with multicollinearity and the variance inflation factor as a warning. It then develops the diagnostics that would have caught Anscombe's traps: residual plots, leverages, studentised and deleted residuals, Cook's distance and the Box–Cox transformation. Finally it leaves the normal model behind. When the response is binary, a straight line for a probability makes no sense, and logistic regression (named after Joseph Berkson's "logit" of 1944) models the log-odds as linear instead; its coefficients are log odds ratios and its maximum likelihood estimate is found by Newton–Raphson, which is a sequence of weighted least squares fits.
- fit simple and multiple regressions from data or from summary statistics, give the variances and covariance of the estimators, and carry out $t$- and $F$-tests on the coefficients;
- build a confidence interval for a mean response and a prediction interval for a new observation, and explain why both widen away from $\bar x$ and why extrapolation is dangerous;
- interpret $R^2$ and adjusted $R^2$, say what each does when a regressor is added, and recognise multicollinearity through the variance inflation factor and its effect on the tests;
- use residual plots, leverages, studentised and deleted residuals and Cook's distance to find outliers and influential points, and choose a transformation such as Box–Cox to repair curvature or unequal variance;
- write down the logistic regression model, interpret its coefficients as log odds ratios, fit it by Newton–Raphson (iteratively reweighted least squares), and recognise separation, when the maximum likelihood estimate does not exist;
- test logistic coefficients by Wald and likelihood ratio (deviance) tests, and connect a single binary regressor to the odds ratio of a $2\times2$ table.
How this is examined
Regression questions appear regularly in CSIR NET papers. Part B questions usually ask for a number computed from summary statistics: a slope or intercept, the standard error of $\hat\beta_1$, the covariance or correlation between $\hat\beta_0$ and $\hat\beta_1$, $R^2$ or the regression $F$ statistic from sums of squares (or $F$ from $R^2$), an adjusted $R^2$, a variance inflation factor, a leverage or Cook's distance, or a fitted probability, odds or odds ratio from a logistic model. Part C questions, where several options may be correct, test the theory behind those numbers: which residual identities need an intercept in the model; what $R^2$ and adjusted $R^2$ do when a regressor is added; how multicollinearity affects individual $t$-tests and the overall $F$-test; whether a point with a small residual can still be influential; how a logistic coefficient changes the odds; and when the logistic MLE fails to exist. The wrong options come from the ⚠️ box of Section 11.1: a high $R^2$ read as proof of a correct model, a small residual read as a harmless point, residuals treated as the errors, prediction intervals expected to shrink to zero, an odds ratio read as a risk ratio, and logistic coefficients read as changes in probability. GATE ST covers the same material, often with numerical answers.
Sections in this chapter
This chapter applies the theory of §10.1 directly: the linear model $y=X\beta+\varepsilon$, the normal equations, the hat matrix $H$ and its properties, the Gauss–Markov theorem, the unbiased estimator $\hat\sigma^2=\operatorname{RSS}/(n-r)$ and the $F$-test of a linear hypothesis; the ANOVA table of §10.2 reappears as the regression ANOVA. From earlier chapters you need the correlation coefficient and the least-squares line as descriptive tools (§1.1); covariance and correlation of random variables (§2.2); the delta method (§4.1); the $\chi^2$, $t$ and $F$ laws (§6.1); maximum likelihood and Fisher information (§7.1); confidence intervals (§7.2); likelihood ratio tests and Wilks' theorem (§8.1); and odds ratios in $2\times2$ tables (§9.1). From linear algebra you need rank, inverses, orthogonal projections and positive definite matrices (sister course CSIR NET Unit 1: Analysis and Linear Algebra); Newton's method itself belongs to numerical analysis (sister course CSIR NET Unit 3), but the section explains all it uses. Partial and multiple correlation coefficients, which describe regression in the population, follow in §12.2.