6.3 Regression, Diagnostics and Logistic Regression
Fitting and judging regressions: least squares in the simple and multiple models, $t$ and $F$ inference, confidence and prediction intervals, $R^2$ and adjusted $R^2$, multicollinearity, leverage, studentised residuals and Cook's distance, transformations — and logistic regression for a binary response, fitted by Newton–Raphson.
Regression is the part of statistics that most people outside statistics actually use. An agronomist relates yield to fertiliser and irrigation, a hospital relates length of stay to age and diagnosis, an economist relates demand to price and income, and an epidemiologist relates the chance of a disease to smoking and age. In §6.1 we met the general linear model $y=X\beta+\varepsilon$ and proved that least squares gives the best linear unbiased estimators; this section turns that theory into a working method. We fit straight lines and planes, attach standard errors, confidence intervals and tests to every coefficient, predict new observations with honest error bars, and measure how much of the variation the model explains. Then comes the step that separates a careful analyst from a button-presser: diagnostics. Is the relation really linear? Is the variance constant? Does a single observation control the whole fit? Leverage, studentised residuals and Cook's distance answer these questions with exact formulas, and suitable transformations often repair what they reveal.
The last part of the section leaves the normal model behind. When the response is binary — cured or not, defaulted or not, died or survived — a straight line for the probability makes no sense, and the right model is logistic regression: the log-odds of success is linear in the regressors. Its coefficients are log odds ratios, its likelihood is concave, and its maximum likelihood estimate is computed by Newton–Raphson, which turns out to be a sequence of weighted least-squares fits. CSIR Part B questions typically ask for a slope, a standard error, an $R^2$ or $F$ value from summary statistics, a leverage or Cook's distance, or a fitted probability or odds ratio. Part C questions test the theory behind them: which residual identities need an intercept, what $R^2$ and adjusted $R^2$ do when a variable is added, how multicollinearity affects $t$ and $F$ tests, when a point is influential, and when the logistic MLE fails to exist.
Simple Linear Regression: Estimation and Inference
We write $S_{xx}=\sum(x_i-\bar x)^2$, $S_{xy}=\sum(x_i-\bar x)(y_i-\bar y)$ and $S_{yy}=\sum(y_i-\bar y)^2$, and the same with capital letters when the $Y_i$ are random. Quantiles are upper-$\alpha$ points as in §4.3: $t_{\nu,\alpha}$, $\chi^2_{\nu,\alpha}$ and $F_{\nu_1,\nu_2,\alpha}$ are exceeded with probability $\alpha$ by a $t_\nu$, $\chi^2_\nu$ and $F_{\nu_1,\nu_2}$ variable, and $z_\alpha$ by an $N(0,1)$ variable. The descriptive least-squares line of §1.1 reappears here as an estimator inside a probability model.
Observations $(x_1,Y_1),\dots,(x_n,Y_n)$ follow the simple linear regression model if
$$Y_i=\beta_0+\beta_1x_i+\varepsilon_i,\qquad i=1,\dots,n,$$where the $x_i$ are known constants, not all equal (so $S_{xx}\gt0$), $\beta_0,\beta_1\in\mathbb{R}$ and $\sigma^2\gt0$ are unknown parameters, and the errors satisfy one of two sets of assumptions:
- (GM) Gauss–Markov assumptions: $E[\varepsilon_i]=0$, $\operatorname{Var}(\varepsilon_i)=\sigma^2$, $\operatorname{Cov}(\varepsilon_i,\varepsilon_j)=0$ for $i\ne j$;
- (N) normal-theory assumptions: $\varepsilon_1,\dots,\varepsilon_n$ iid $N(0,\sigma^2)$.
"Linear" means linear in the parameters: $Y=\beta_0+\beta_1\log x+\varepsilon$ and $Y=\beta_0+\beta_1x+\beta_2x^2+\varepsilon$ are linear models, $Y=\beta_0e^{\beta_1x}+\varepsilon$ is not. When the $x_i$ are themselves random, all statements are made conditionally on the observed $x_i$.
Under (GM), the function $Q(\beta_0,\beta_1)=\sum_{i=1}^n(Y_i-\beta_0-\beta_1x_i)^2$ has a unique minimiser, the least-squares estimator
$$\hat\beta_1=\frac{S_{xY}}{S_{xx}}=\sum_{i=1}^nc_iY_i,\quad c_i=\frac{x_i-\bar x}{S_{xx}},$$ $$\hat\beta_0=\bar Y-\hat\beta_1\bar x.$$Both are unbiased, and
$$\operatorname{Var}(\hat\beta_1)=\frac{\sigma^2}{S_{xx}},$$ $$\operatorname{Var}(\hat\beta_0)=\sigma^2\Big(\frac1n+\frac{\bar x^2}{S_{xx}}\Big),$$ $$\operatorname{Cov}(\hat\beta_0,\hat\beta_1)=-\frac{\bar x\,\sigma^2}{S_{xx}},\quad \operatorname{Cov}(\bar Y,\hat\beta_1)=0.$$Moreover $\hat\beta_1$ has the smallest variance among all linear unbiased estimators $\sum a_iY_i$ of $\beta_1$ (it is the BLUE; the general statement is the Gauss–Markov theorem of §6.1).
Proof. $Q$ is a quadratic function of $(\beta_0,\beta_1)$. Setting its partial derivatives to zero gives the normal equations $\sum(Y_i-\beta_0-\beta_1x_i)=0$ and $\sum x_i(Y_i-\beta_0-\beta_1x_i)=0$. The first gives $\beta_0=\bar Y-\beta_1\bar x$; substituting into the second and subtracting $\bar x$ times the first gives $\sum(x_i-\bar x)\big(Y_i-\bar Y-\beta_1(x_i-\bar x)\big)=0$, that is $\beta_1=S_{xY}/S_{xx}$. The Hessian of $Q$ is $2\begin{pmatrix}n&\sum x_i\\ \sum x_i&\sum x_i^2\end{pmatrix}$, whose determinant is $4nS_{xx}\gt0$ and whose diagonal is positive, so it is positive definite and the stationary point is the unique global minimum.
Because $\sum(x_i-\bar x)=0$ we have $S_{xY}=\sum(x_i-\bar x)Y_i$, so $\hat\beta_1=\sum c_iY_i$ with $\sum c_i=0$, $\sum c_ix_i=\sum c_i(x_i-\bar x)=1$ and $\sum c_i^2=1/S_{xx}$. Hence $E[\hat\beta_1]=\sum c_i(\beta_0+\beta_1x_i)=\beta_1$ and, the $Y_i$ being uncorrelated with variance $\sigma^2$, $\operatorname{Var}(\hat\beta_1)=\sigma^2\sum c_i^2=\sigma^2/S_{xx}$. Next $\operatorname{Cov}(\bar Y,\hat\beta_1)=\sigma^2\sum\frac1n c_i=0$. Then $E[\hat\beta_0]=\beta_0+\beta_1\bar x-\beta_1\bar x=\beta_0$, $\operatorname{Var}(\hat\beta_0)=\operatorname{Var}(\bar Y)+\bar x^2\operatorname{Var}(\hat\beta_1)=\sigma^2(\frac1n+\frac{\bar x^2}{S_{xx}})$ and $\operatorname{Cov}(\hat\beta_0,\hat\beta_1)=\operatorname{Cov}(\bar Y,\hat\beta_1)-\bar x\operatorname{Var}(\hat\beta_1)$, which is $-\bar x\sigma^2/S_{xx}$.
Finally let $\sum a_iY_i$ be unbiased for $\beta_1$ for all $\beta_0,\beta_1$. Then $\sum a_i(\beta_0+\beta_1x_i)=\beta_1$ identically, i.e. $\sum a_i=0$ and $\sum a_ix_i=1$. Hence $\sum a_ic_i=\frac{\sum a_ix_i-\bar x\sum a_i}{S_{xx}}=\frac1{S_{xx}}=\sum c_i^2$, so $\sum(a_i-c_i)c_i=0$ and $\sum a_i^2=\sum c_i^2+\sum(a_i-c_i)^2$. Thus $\operatorname{Var}(\sum a_iY_i)=\sigma^2\sum a_i^2\ge\sigma^2\sum c_i^2$, with equality only if $a_i=c_i$ for all $i$. ∎
The fitted values are $\hat Y_i=\hat\beta_0+\hat\beta_1x_i$ and the residuals are $e_i=Y_i-\hat Y_i$. The total, regression and error (residual) sums of squares are
$$SST=\sum(Y_i-\bar Y)^2,$$ $$SSR=\sum(\hat Y_i-\bar Y)^2,\qquad SSE=\sum e_i^2 .$$The residual mean square is $s^2=MSE=SSE/(n-2)$, and the coefficient of determination is $R^2=SSR/SST$, the proportion of the variation of $Y$ about its mean that is accounted for by the fitted line.
- $\sum e_i=0$, $\sum x_ie_i=0$ and $\sum\hat Y_ie_i=0$; the fitted line passes through $(\bar x,\bar Y)$.
- $SST=SSR+SSE$, with $SSR=\hat\beta_1^2S_{xx}=S_{xY}^2/S_{xx}$ and $SSE=S_{YY}-S_{xY}^2/S_{xx}$. In particular $R^2=r_{xY}^2$, the square of the sample correlation coefficient.
- Under (GM), $E[SSE]=(n-2)\sigma^2$, so $s^2$ is an unbiased estimator of $\sigma^2$. No normality is needed.
Proof. (1) The two normal equations say exactly $\sum e_i=0$ and $\sum x_ie_i=0$; since $\hat Y_i=\hat\beta_0+\hat\beta_1x_i$, $\sum\hat Y_ie_i=\hat\beta_0\sum e_i+\hat\beta_1\sum x_ie_i=0$. Also $\hat\beta_0+\hat\beta_1\bar x=\bar Y$ by the formula for $\hat\beta_0$.
(2) Write $Y_i-\bar Y=(\hat Y_i-\bar Y)+e_i$, square and sum. The cross term is $2\sum(\hat Y_i-\bar Y)e_i=2\sum\hat Y_ie_i-2\bar Y\sum e_i=0$. Since $\hat Y_i-\bar Y=\hat\beta_1(x_i-\bar x)$, $SSR=\hat\beta_1^2S_{xx}=S_{xY}^2/S_{xx}$, and $SSE=SST-SSR$. Then $R^2=S_{xY}^2/(S_{xx}S_{YY})=r_{xY}^2$.
(3) $Y_i-\bar Y=\beta_1(x_i-\bar x)+(\varepsilon_i-\bar\varepsilon)$, so $E[S_{YY}]=\beta_1^2S_{xx}+E\sum(\varepsilon_i-\bar\varepsilon)^2$, which equals $\beta_1^2S_{xx}+(n-1)\sigma^2$ (the cross term has mean $0$). Also $E[\hat\beta_1^2]=\operatorname{Var}(\hat\beta_1)+\beta_1^2=\sigma^2/S_{xx}+\beta_1^2$. Hence
$$E[SSE]=E[S_{YY}]-S_{xx}E[\hat\beta_1^2]$$ $$=\beta_1^2S_{xx}+(n-1)\sigma^2-\sigma^2-\beta_1^2S_{xx}=(n-2)\sigma^2.$$∎
We lose two degrees of freedom because two parameters were estimated: the $n$ residuals satisfy the two linear constraints in (1). The same count gives $n-p$ in a model with $p$ coefficients.
Under (N):
- $(\hat\beta_0,\hat\beta_1)$ is bivariate normal with the means, variances and covariance of Theorem 6.19;
- $SSE/\sigma^2\sim\chi^2_{n-2}$, and $SSE$ is independent of $(\bar Y,\hat\beta_0,\hat\beta_1)$;
- consequently
Hence $\hat\beta_1\pm t_{n-2,\alpha/2}\,s/\sqrt{S_{xx}}$ is a $100(1-\alpha)\%$ confidence interval for $\beta_1$; the size-$\alpha$ test of $H_0:\beta_1=b$ against $\beta_1\ne b$ rejects when $|T_1|\gt t_{n-2,\alpha/2}$ with $\beta_1=b$ in $T_1$; and $\big[SSE/\chi^2_{n-2,\alpha/2},\ SSE/\chi^2_{n-2,1-\alpha/2}\big]$ is a confidence interval for $\sigma^2$. The least-squares estimators are also the maximum likelihood estimators of $\beta_0,\beta_1$; the MLE of $\sigma^2$ is $SSE/n$, which is biased.
Proof. Let $\mathbf 1=(1,\dots,1)^\top$ and $\mathbf x=(x_1,\dots,x_n)^\top$. The vectors $u_1=\mathbf 1/\sqrt n$ and $u_2=(\mathbf x-\bar x\mathbf 1)/\sqrt{S_{xx}}$ are orthonormal and span the same plane as $\mathbf 1$ and $\mathbf x$. Extend them to an orthonormal basis $u_1,\dots,u_n$ of $\mathbb{R}^n$ and let $Q$ be the orthogonal matrix with rows $u_k^\top$. Put $Z=Qy$, where $y=(Y_1,\dots,Y_n)^\top$. Since $y\sim N_n(\beta_0\mathbf 1+\beta_1\mathbf x,\sigma^2I)$ and $Q$ is orthogonal, $Z\sim N_n(Q(\beta_0\mathbf 1+\beta_1\mathbf x),\sigma^2I)$: the $Z_k$ are independent normal with variance $\sigma^2$. For $k\ge3$, $u_k\perp\mathbf 1,\mathbf x$, so $E[Z_k]=0$. Also $Z_1=\sqrt n\,\bar Y$ and $Z_2=S_{xY}/\sqrt{S_{xx}}=\sqrt{S_{xx}}\,\hat\beta_1$.
The fitted vector $\hat y$ is the orthogonal projection of $y$ onto the plane spanned by $\mathbf 1,\mathbf x$ (least squares minimises the distance to that plane), so $\hat y=Z_1u_1+Z_2u_2$ and $e=y-\hat y=\sum_{k\ge3}Z_ku_k$. Therefore $SSE=\|e\|^2=\sum_{k=3}^nZ_k^2$, a sum of $n-2$ squares of independent $N(0,\sigma^2)$ variables, so $SSE/\sigma^2\sim\chi^2_{n-2}$; it is a function of $Z_3,\dots,Z_n$ and hence independent of $(Z_1,Z_2)$, i.e. of $(\bar Y,\hat\beta_1)$ and of $\hat\beta_0=\bar Y-\hat\beta_1\bar x$. Part (1) holds because $(\hat\beta_0,\hat\beta_1)$ is a linear function of the normal vector $y$. For (3), $(\hat\beta_1-\beta_1)\sqrt{S_{xx}}/\sigma\sim N(0,1)$ is independent of $SSE/\sigma^2\sim\chi^2_{n-2}$, and dividing by $\sqrt{SSE/(\sigma^2(n-2))}=s/\sigma$ gives a $t_{n-2}$ variable; $T_0$ is the same. The interval and the test follow by inverting the pivots. Finally, the normal log-likelihood is $-\frac n2\log(2\pi\sigma^2)-\frac{Q(\beta_0,\beta_1)}{2\sigma^2}$; for each $\sigma^2$ it is maximised by minimising $Q$, and then maximising $-\frac n2\log\sigma^2-\frac{SSE}{2\sigma^2}$ gives $\hat\sigma^2=SSE/n$. ∎
| $x$ | 2 | 3 | 5 | 6 | 8 | 9 | 11 | 12 |
|---|---|---|---|---|---|---|---|---|
| $y$ | 52 | 55 | 61 | 60 | 68 | 67 | 74 | 79 |
Under (N), fix $x_0$ and let $\hat\mu_0=\hat\beta_0+\hat\beta_1x_0$.
- A $100(1-\alpha)\%$ confidence interval for the mean response $\mu_0=\beta_0+\beta_1x_0$ is
- If $Y_0=\beta_0+\beta_1x_0+\varepsilon_0$ is a future observation with $\varepsilon_0\sim N(0,\sigma^2)$ independent of $Y_1,\dots,Y_n$, then the prediction interval
contains $Y_0$ with probability exactly $1-\alpha$. Both intervals are narrowest at $x_0=\bar x$ and widen as $x_0$ moves away from $\bar x$.
Proof. $\hat\mu_0=\bar Y+\hat\beta_1(x_0-\bar x)$, and $\bar Y$, $\hat\beta_1$ are uncorrelated (Theorem 6.19), so $E[\hat\mu_0]=\mu_0$ and $\operatorname{Var}(\hat\mu_0)=\sigma^2\big(\frac1n+\frac{(x_0-\bar x)^2}{S_{xx}}\big)$. It is normal and, as a function of $(\bar Y,\hat\beta_1)$, independent of $SSE$ (Theorem 6.21). So $(\hat\mu_0-\mu_0)/\big(s\sqrt{\frac1n+\frac{(x_0-\bar x)^2}{S_{xx}}}\big)\sim t_{n-2}$, which gives (1). For (2), $Y_0-\hat\mu_0$ is normal with mean $0$ and, because $\varepsilon_0$ is independent of the sample, variance $\sigma^2+\operatorname{Var}(\hat\mu_0)=\sigma^2\big(1+\frac1n+\frac{(x_0-\bar x)^2}{S_{xx}}\big)$; it is independent of $SSE$ (both $\varepsilon_0$ and $\hat\mu_0$ are), so the standardised ratio is again $t_{n-2}$, and $P(|T|\le t_{n-2,\alpha/2})=1-\alpha$. ∎
The extra "$1+$" is the variance of the new observation's own error, and it never goes away: as $n\to\infty$ with $S_{xx}/n$ bounded away from $0$, the confidence interval shrinks to the point $\mu_0$ while the prediction interval tends to $\mu_0\pm z_{\alpha/2}\sigma$. Plotted against $x_0$, the two intervals trace out hyperbolic bands around the fitted line (Interactive 1 draws them). These are pointwise statements; a band that covers the whole true line simultaneously with probability $1-\alpha$ (the Working–Hotelling band) replaces $t_{n-2,\alpha/2}$ by $\sqrt{2F_{2,n-2,\alpha}}$.
Under (N), the statistic for $H_0:\beta_1=0$
$$F=\frac{SSR/1}{SSE/(n-2)}$$has the $F_{1,n-2}$ distribution under $H_0$, and $F=T_1^2$ where $T_1=\hat\beta_1\sqrt{S_{xx}}/s$; so the $F$ test and the two-sided $t$ test are the same test. Moreover $F=\dfrac{(n-2)R^2}{1-R^2}$, and $T_1=\dfrac{r\sqrt{n-2}}{\sqrt{1-r^2}}$ with $r=r_{xY}$.
Proof. By Theorem 6.20(2), $SSR=\hat\beta_1^2S_{xx}$, so $F=\hat\beta_1^2S_{xx}/s^2=T_1^2$. Under $H_0$, $T_1\sim t_{n-2}$ (Theorem 6.21), and the square of a $t_\nu$ variable is $F_{1,\nu}$ (§4.1). Dividing numerator and denominator of $F$ by $SST$ gives $F=\frac{R^2}{(1-R^2)/(n-2)}$, and with $R^2=r^2$ (Theorem 6.20) $T_1^2=\frac{(n-2)r^2}{1-r^2}$; the signs of $T_1$ and $r$ agree because both are the sign of $S_{xY}$. ∎
The decomposition $SST=SSR+SSE$ is reported as an analysis of variance table. For Example 1:
| Source | df | SS | MS | $F$ |
|---|---|---|---|---|
| Regression | 1 | 580.011 | 580.011 | 193.45 |
| Error | 6 | 17.989 | 2.998 | |
| Total | 7 | 598.000 |
and $193.45=13.909^2=T_1^2$, while $F_{1,6,0.05}=5.99=t_{6,0.025}^2$.
Multiple Regression, $R^2$ and Multicollinearity
With several regressors the algebra is cleanest in matrix form, and most of it was done in §6.1. We recall what we need, then concentrate on what is special to regression practice: how to judge the fit as a whole, how to compare nested models, and what happens when regressors are nearly linearly dependent.
The multiple linear regression model with $k$ regressors is
$$y=X\beta+\varepsilon,\qquad X=\begin{pmatrix}1&x_{11}&\cdots&x_{1k}\\ \vdots&\vdots&&\vdots\\ 1&x_{n1}&\cdots&x_{nk}\end{pmatrix},$$with $\beta=(\beta_0,\dots,\beta_k)^\top$, $p=k+1$ coefficients, $X$ a known $n\times p$ matrix of full column rank $p\lt n$, and errors satisfying (GM): $E[\varepsilon]=0$, $\operatorname{Cov}(\varepsilon)=\sigma^2I_n$, or (N): $\varepsilon\sim N_n(0,\sigma^2I_n)$. The $i$-th row of $X$ is written $x_i^\top$. Here $y=(Y_1,\dots,Y_n)^\top$ is the response vector $y$ of §6.1, and in this full-rank regression setting $\operatorname{rank}X=p$, so the rank $r$ of §6.1 equals $p$ and §6.1's unbiased estimator $\hat\sigma^2=\operatorname{RSS}/(n-r)$ (Theorem 6.6, with $\operatorname{RSS}=SSE$) is the residual mean square $s^2=SSE/(n-p)$ of this section. The hat matrix is
$$H=X(X^\top X)^{-1}X^\top,$$so called because the fitted values are $\hat y=X\hat\beta=Hy$ ("$H$ puts the hat on $y$"); the residuals are $e=y-\hat y=(I-H)y$. Regressors may be powers ($x,x^2$), logarithms, products (interactions) or 0–1 indicator ("dummy") variables for categories.
- (GM) The least-squares estimator is $\hat\beta=(X^\top X)^{-1}X^\top y$; it is unbiased with $\operatorname{Cov}(\hat\beta)=\sigma^2(X^\top X)^{-1}$, and $c^\top\hat\beta$ is the BLUE of $c^\top\beta$ (§6.1).
- $H$ and $I-H$ are symmetric and idempotent with traces $p$ and $n-p$; $X^\top e=0$, so $\sum e_i=0$ when the model has an intercept; and $E[SSE]=(n-p)\sigma^2$, so $s^2=SSE/(n-p)$ is unbiased.
- (N) $\hat\beta\sim N_p\big(\beta,\sigma^2(X^\top X)^{-1}\big)$ and $SSE/\sigma^2\sim\chi^2_{n-p}$, independently. With $c_{jj}$ the $j$-th diagonal entry of $(X^\top X)^{-1}$, $T_j=(\hat\beta_j-\beta_j)/(s\sqrt{c_{jj}})\sim t_{n-p}$. At a new point $x_0$, the confidence and prediction intervals are
- (N) Extra sum of squares. Let a reduced model be obtained by setting $q$ of the coefficients to $0$, with error sum of squares $SSE_R$, and let $SSE_F$ be that of the full model. Under $H_0$ (the $q$ coefficients are $0$) $$F=\frac{(SSE_R-SSE_F)/q}{SSE_F/(n-p)}\sim F_{q,\,n-p}.$$ For $q=1$, $F=T_j^2$. For the overall test $H_0:\beta_1=\cdots=\beta_k=0$ the reduced model is $Y_i=\beta_0+\varepsilon_i$, $SSE_R=SST$, and $$F=\frac{SSR/k}{SSE/(n-k-1)}$$ $$=\frac{R^2/k}{(1-R^2)/(n-k-1)}\sim F_{k,\,n-k-1}.$$
Proof. The details are in §6.1; here is the thread. The normal equations are $X^\top X\beta=X^\top y$, and $X^\top X$ is invertible because $X$ has full column rank. $E[\hat\beta]=(X^\top X)^{-1}X^\top X\beta=\beta$ and $\operatorname{Cov}(\hat\beta)=(X^\top X)^{-1}X^\top(\sigma^2I)X(X^\top X)^{-1}$, which is $\sigma^2(X^\top X)^{-1}$. $H^\top=H$ and $H^2=X(X^\top X)^{-1}X^\top X(X^\top X)^{-1}X^\top=H$; $\operatorname{tr}H=\operatorname{tr}\big((X^\top X)^{-1}X^\top X\big)=p$. $X^\top e=X^\top(I-H)y=0$ since $X^\top H=X^\top$; the first row of $X^\top$ is $\mathbf 1^\top$. Because $(I-H)X=0$, $e=(I-H)\varepsilon$ and $E[SSE]=E[\varepsilon^\top(I-H)\varepsilon]$, which is $\sigma^2\operatorname{tr}(I-H)=(n-p)\sigma^2$.
Under (N), $\hat\beta$ and $e$ are jointly normal with $\operatorname{Cov}(\hat\beta,e)=(X^\top X)^{-1}X^\top\sigma^2(I-H)=0$, hence independent. Writing the symmetric idempotent $I-H=PDP^\top$ with $P$ orthogonal and $D$ diagonal with $n-p$ ones and $p$ zeros, $SSE/\sigma^2=\varepsilon^\top(I-H)\varepsilon/\sigma^2$ is a sum of $n-p$ squares of independent $N(0,1)$ variables (the coordinates of $P^\top\varepsilon/\sigma$). The $t$ pivots and the intervals follow exactly as in Theorems 6.21 and 6.22, with $\operatorname{Var}(x_0^\top\hat\beta)=\sigma^2x_0^\top(X^\top X)^{-1}x_0$.
For (4), let $H_F$, $H_R$ be the hat matrices of the two models. The column space of the reduced model lies inside that of the full one, so $H_FH_R=H_R$, and $M=H_F-H_R$ is symmetric idempotent of rank $q$ with $M(I-H_F)=0$. Now $SSE_R-SSE_F=y^\top(I-H_R)y-y^\top(I-H_F)y$, which is $\|My\|^2$, and under $H_0$ the mean $X\beta$ lies in the reduced column space, so $MX\beta=0$ and $\|My\|^2=\varepsilon^\top M\varepsilon\sim\sigma^2\chi^2_q$. Since $M\varepsilon$ and $(I-H_F)\varepsilon$ are jointly normal and uncorrelated, the numerator and denominator are independent, which gives $F_{q,n-p}$. The case $q=1$ is proved after Theorem 6.26, and the overall test is the case of the intercept-only reduced model. ∎
For a model with an intercept, the coefficient of determination and the adjusted $R^2$ are
$$R^2=1-\frac{SSE}{SST},$$ $$\bar R^2=1-\frac{SSE/(n-p)}{SST/(n-1)}=1-(1-R^2)\frac{n-1}{n-p}.$$$R=\sqrt{R^2}\ge0$ is the multiple correlation coefficient of $y$ with the regressors (studied further in §7.2). $\bar R^2$ replaces the two sums of squares by their unbiased variance estimates; it penalises extra parameters.
Consider models with an intercept.
- $SST=SSR+SSE$ with $SSR=\sum(\hat Y_i-\bar Y)^2$; hence $0\le R^2\le1$, and $R^2=[\operatorname{corr}(y,\hat y)]^2$, the squared sample correlation between observed and fitted values.
- Adding regressors never decreases $R^2$.
- $\bar R^2\le R^2$, and $\bar R^2$ can be negative. Adding $q$ regressors to a model with $p$ coefficients increases $\bar R^2$ if and only if the extra-sum-of-squares statistic $F$ of Theorem 6.24(4) exceeds $1$; for a single added regressor, iff its $|T_j|\gt1$.
Proof. (1) Both $\hat y$ and $\bar Y\mathbf 1$ lie in the column space of $X$ (the intercept column is $\mathbf 1$), while $e$ is orthogonal to that space; so $e\perp\hat y-\bar Y\mathbf 1$ and Pythagoras gives $$\|y-\bar Y\mathbf 1\|^2=\|\hat y-\bar Y\mathbf 1\|^2+\|e\|^2.$$ Next, $\sum(Y_i-\bar Y)(\hat Y_i-\bar Y)$ equals $\sum(\hat Y_i-\bar Y+e_i)(\hat Y_i-\bar Y)=SSR$, and the mean of the $\hat Y_i$ is $\bar Y$ (because $\sum e_i=0$), so $[\operatorname{corr}(y,\hat y)]^2=\frac{SSR^2}{SST\cdot SSR}=R^2$.
(2) The larger model minimises the same sum of squares over a larger set of coefficient vectors (the smaller model is the special case with the new coefficients $0$), so $SSE_F\le SSE_R$, and $SST$ does not change.
(3) $\bar R^2\le R^2$ because $\frac{n-1}{n-p}\ge1$; for example $R^2=0$ gives $\bar R^2=-\frac{p-1}{n-p}\lt0$ when $p\ge2$. Let the reduced model have $p$ and the full one $p+q$ coefficients. Since $SST$ is common, $\bar R^2$ increases iff $\frac{SSE_F}{n-p-q}\lt\frac{SSE_R}{n-p}$. By definition of $F$, $SSE_R=SSE_F+\frac{qF\,SSE_F}{n-p-q}=SSE_F\cdot\frac{n-p-q+qF}{n-p-q}$, so the condition becomes $\frac{n-p-q+qF}{n-p}\gt1$, i.e. $F\gt1$. For $q=1$, $F=T_j^2$. ∎
So $\bar R^2$ is a very lenient model-selection rule: it keeps any variable whose $|t|$ exceeds $1$, whereas a $5\%$ test needs $|t|$ near $2$. Neither $R^2$ nor $\bar R^2$ says whether the model is correct; that is the job of the diagnostics of the next subsection.
For the regressor $x_j$ ($1\le j\le k$), let $R_j^2$ be the $R^2$ of the regression of $x_j$ on the other $k-1$ regressors (with an intercept) and $S_{jj}=\sum_i(x_{ij}-\bar x_j)^2$. The variance inflation factor of $x_j$ is
$$\operatorname{VIF}_j=\frac1{1-R_j^2}.$$The regressors are multicollinear when some $R_j^2$ is close to $1$ — equivalently, some linear combination of the columns of $X$ is nearly zero and $X^\top X$ has an eigenvalue close to $0$. Exact collinearity ($R_j^2=1$) means $X$ is not of full rank and the individual $\beta_j$ are not estimable (§6.1). A common rule of thumb treats $\operatorname{VIF}_j\gt10$ as serious.
In a full-rank model with an intercept, write $X=[Z\ \ x_j]$ where $Z$ holds the other columns (including $\mathbf 1$), $H_Z=Z(Z^\top Z)^{-1}Z^\top$, and let $\tilde x_j=(I-H_Z)x_j$ be the residual vector from regressing $x_j$ on $Z$. Then
$$\hat\beta_j=\frac{\tilde x_j^\top y}{\tilde x_j^\top\tilde x_j},$$ $$\operatorname{Var}(\hat\beta_j)=\frac{\sigma^2}{\|\tilde x_j\|^2}=\frac{\sigma^2}{S_{jj}(1-R_j^2)}=\frac{\sigma^2}{S_{jj}}\operatorname{VIF}_j.$$With two regressors, $R_1^2=R_2^2=r_{12}^2$ (the squared correlation between them) and $\operatorname{VIF}_1=\operatorname{VIF}_2=1/(1-r_{12}^2)$. If $x_j$ is orthogonal to the other centred regressors, $\operatorname{VIF}_j=1$ and $\hat\beta_j$ equals the slope of the simple regression of $y$ on $x_j$.
Proof. Write the normal equations in blocks, with $\gamma$ the coefficients of $Z$:
$$Z^\top Z\gamma+Z^\top x_j\beta_j=Z^\top y,$$ $$x_j^\top Z\gamma+x_j^\top x_j\beta_j=x_j^\top y.$$The first gives $Z\gamma=H_Z(y-x_j\beta_j)$. Substituting into the second, $x_j^\top H_Z(y-x_j\beta_j)+x_j^\top x_j\beta_j=x_j^\top y$, i.e. $x_j^\top(I-H_Z)x_j\,\beta_j=x_j^\top(I-H_Z)y$. Since $I-H_Z$ is symmetric idempotent, $x_j^\top(I-H_Z)x_j=\|\tilde x_j\|^2$ and $x_j^\top(I-H_Z)y=\tilde x_j^\top y$, which is the formula for $\hat\beta_j$. It is a linear combination $\sum a_iY_i$ with $a=\tilde x_j/\|\tilde x_j\|^2$, so $$\operatorname{Var}(\hat\beta_j)=\sigma^2\|a\|^2=\sigma^2/\|\tilde x_j\|^2.$$ Finally $\|\tilde x_j\|^2$ is the residual sum of squares of the regression of $x_j$ on $Z$, which has an intercept, so it equals $(1-R_j^2)$ times the total sum of squares $S_{jj}$ of $x_j$. With two regressors, $Z=[\mathbf 1\ x_{\text{other}}]$ and $R_j^2=r_{12}^2$ by Theorem 6.20(2). If $x_j-\bar x_j\mathbf 1$ is orthogonal to the columns of $Z$ other than $\mathbf 1$, then $\tilde x_j=x_j-\bar x_j\mathbf 1$ and $\hat\beta_j=S_{x_jY}/S_{jj}$. ∎
The same argument proves $F=T_j^2$ in Theorem 6.24(4): $H_F-H_Z$ is the projection onto $\tilde x_j$, namely $\tilde x_j\tilde x_j^\top/\|\tilde x_j\|^2$, so $SSE_R-SSE_F=(\tilde x_j^\top y)^2/\|\tilde x_j\|^2$, which is $\hat\beta_j^2\|\tilde x_j\|^2=\hat\beta_j^2/c_{jj}$, and dividing by $s^2$ gives $T_j^2$. Theorem 6.26 explains the symptoms of multicollinearity: when $x_j$ is nearly a linear combination of the others, $\tilde x_j$ is short, $\hat\beta_j$ is computed from a tiny "private" part of $x_j$, and its standard error is inflated by the factor $\sqrt{\operatorname{VIF}_j}$. The fitted values and predictions inside the range of the data are unaffected; only the attribution of the effect to individual regressors becomes unstable. Remedies: drop or combine nearly redundant regressors, centre $x$ before forming $x^2$ or interactions, collect data in which the regressors vary separately, or accept a little bias for much less variance with ridge regression $\hat\beta_\lambda=(X^\top X+\lambda I)^{-1}X^\top y$ or with principal-component regression (§7.3).
| $x_1$ | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 |
|---|---|---|---|---|---|---|---|---|
| $x_2$ | 2 | 3 | 3 | 5 | 6 | 6 | 8 | 9 |
| $y$ | 10 | 13 | 13 | 17 | 18 | 20 | 22 | 25 |
Regression Diagnostics and Transformations
Every interval and test above rests on assumptions: the mean of $Y$ is linear in the regressors, the errors have constant variance, they are uncorrelated (independent under (N)) and normal — and, implicitly, no single observation dictates the fit. Diagnostics check these assumptions with the only proxies we have for the unobservable errors, the residuals. The first lesson is that residuals are not the errors: they are correlated, they have unequal variances, and the observations that pull hardest on the fit tend to have the smallest residuals.
In the full-rank model of Definition 6.13, write $h_{ij}$ for the entries of $H$.
- Under (GM), $E[e]=0$ and $\operatorname{Cov}(e)=\sigma^2(I-H)$: $\operatorname{Var}(e_i)=\sigma^2(1-h_{ii})$ and $\operatorname{Cov}(e_i,e_j)=-\sigma^2h_{ij}$. Also $\operatorname{Cov}(e,\hat y)=0$.
- $0\le h_{ii}\le1$ and $\sum_ih_{ii}=p$. If the model has an intercept, $h_{ii}\ge\frac1n$.
- $\hat Y_i=h_{ii}Y_i+\sum_{j\ne i}h_{ij}Y_j$, so $h_{ii}=\partial\hat Y_i/\partial Y_i$. If $h_{ii}=1$ then $h_{ij}=0$ for $j\ne i$ and $e_i=0$ whatever the data: the fit is forced through the $i$-th point.
- In simple regression, $h_{ij}=\dfrac1n+\dfrac{(x_i-\bar x)(x_j-\bar x)}{S_{xx}}$; in particular $h_{ii}=\dfrac1n+\dfrac{(x_i-\bar x)^2}{S_{xx}}$.
Proof. (1) $(I-H)X=0$ gives $e=(I-H)y=(I-H)\varepsilon$, so $E[e]=0$ and $\operatorname{Cov}(e)=(I-H)\sigma^2I(I-H)^\top=\sigma^2(I-H)$ by symmetry and idempotence. Similarly $\operatorname{Cov}(e,\hat y)=(I-H)\sigma^2H=\sigma^2(H-H^2)=0$.
(2) Since $H=H^2=H^\top H$, $h_{ii}=\sum_jh_{ij}^2\ge h_{ii}^2$, so $0\le h_{ii}\le1$; and $\sum h_{ii}=\operatorname{tr}H=p$. With an intercept, $H\mathbf 1=\mathbf 1$ ($\mathbf 1$ is in the column space). Let $J=\mathbf 1\mathbf 1^\top$; then $HJ=JH=J$ and $J^2=nJ$, so $\big(H-\tfrac1nJ\big)^2=H-\tfrac2nJ+\tfrac1nJ=H-\tfrac1nJ$. Thus $H-\frac1nJ$ is symmetric idempotent, and by the first argument its diagonal entries $h_{ii}-\frac1n$ are $\ge0$.
(3) The first statement is $\hat y=Hy$ written out. If $h_{ii}=1$, then $1=h_{ii}=h_{ii}^2+\sum_{j\ne i}h_{ij}^2$ forces $h_{ij}=0$ for $j\ne i$, so $\hat Y_i=Y_i$.
(4) Here $H-\frac1nJ$ is the projection onto the line spanned by the unit vector $u=(\mathbf x-\bar x\mathbf 1)/\sqrt{S_{xx}}$ (the column space of $X$ is spanned by the orthonormal pair $\mathbf 1/\sqrt n$, $u$), so $H=\frac1nJ+uu^\top$, which is the formula. ∎
So the residuals of high-leverage points have small variance — the line is pulled towards them. In multiple regression with an intercept the same argument gives $h_{ii}=\frac1n+(\mathbf x_i-\bar{\mathbf x})^\top S^{-1}(\mathbf x_i-\bar{\mathbf x})$, where $\mathbf x_i$ is the $i$-th vector of regressor values and $S$ their matrix of centred sums of squares and products: leverage is a (scaled) Mahalanobis distance from the centre of the design. Note that leverage depends only on the $x$'s, not on $y$.
The leverage of the $i$-th case is $h_{ii}$; its average value is $p/n$, and a case with $h_{ii}\gt2p/n$ is usually flagged as a high-leverage point. The (internally) studentised residual and the externally studentised residual are
$$r_i=\frac{e_i}{s\sqrt{1-h_{ii}}},\qquad t_i=\frac{e_i}{s_{(i)}\sqrt{1-h_{ii}}},$$where $s_{(i)}^2$ is the residual mean square of the fit with case $i$ deleted. Because the residuals have unequal variances $\sigma^2(1-h_{ii})$, the raw $e_i$ are not comparable with each other, while the $r_i$ all have (approximately) unit variance. A case with $|t_i|\gt2$ (or $3$ in large data sets) is a candidate outlier.
Let $\hat\beta_{(i)}$ be the least-squares estimate computed without case $i$. The deleted (predicted) residual is $e_{(i)}=Y_i-x_i^\top\hat\beta_{(i)}$, the error made in predicting case $i$ from the others, and $PRESS=\sum_ie_{(i)}^2$. Cook's distance measures how far the whole fit moves when case $i$ is deleted:
$$D_i=\frac{(\hat\beta_{(i)}-\hat\beta)^\top X^\top X(\hat\beta_{(i)}-\hat\beta)}{p\,s^2}$$ $$=\frac{\|X\hat\beta_{(i)}-X\hat\beta\|^2}{p\,s^2}.$$A case is called influential if deleting it changes the estimates or fitted values substantially; $D_i\gt1$ is the usual warning level (some analysts use $4/n$). The value $D_i=F_{p,n-p,0.5}\approx1$ would move $\hat\beta$ to the edge of the $50\%$ confidence ellipsoid for $\beta$.
Suppose $h_{ii}\lt1$. Then, without refitting,
- $\hat\beta-\hat\beta_{(i)}=\dfrac{(X^\top X)^{-1}x_i\,e_i}{1-h_{ii}}$;
- $e_{(i)}=\dfrac{e_i}{1-h_{ii}}$, so $PRESS=\sum_i\Big(\dfrac{e_i}{1-h_{ii}}\Big)^2$;
- $D_i=\dfrac{r_i^2}{p}\cdot\dfrac{h_{ii}}{1-h_{ii}}$ — Cook's distance is (studentised residual)$^2$ times a leverage factor;
- $(n-p-1)s_{(i)}^2=(n-p)s^2-\dfrac{e_i^2}{1-h_{ii}}$, hence $t_i=r_i\sqrt{\dfrac{n-p-1}{n-p-r_i^2}}$ and $r_i^2\lt n-p$;
- under (N), $t_i\sim t_{n-p-1}$.
Proof. Write $A=X^\top X$, $b=X^\top y$, $h=h_{ii}=x_i^\top A^{-1}x_i$. Deleting case $i$ replaces $A$ by $A-x_ix_i^\top$ and $b$ by $b-x_iY_i$. The Sherman–Morrison formula $(A-x_ix_i^\top)^{-1}=A^{-1}+\frac{A^{-1}x_ix_i^\top A^{-1}}{1-h}$ (check by multiplying out) gives
$$\hat\beta_{(i)}=\Big(A^{-1}+\frac{A^{-1}x_ix_i^\top A^{-1}}{1-h}\Big)(b-x_iY_i)$$ $$=\hat\beta-A^{-1}x_iY_i+\frac{A^{-1}x_i(\hat Y_i-hY_i)}{1-h}.$$Combining the last two terms over $1-h$: $-Y_i(1-h)+\hat Y_i-hY_i=\hat Y_i-Y_i=-e_i$, which proves (1). Then $e_{(i)}=Y_i-x_i^\top\hat\beta+x_i^\top(\hat\beta-\hat\beta_{(i)})=e_i+\frac{h\,e_i}{1-h}=\frac{e_i}{1-h}$, which is (2). For (3), by (1), $(\hat\beta_{(i)}-\hat\beta)^\top A(\hat\beta_{(i)}-\hat\beta)=\frac{e_i^2\,x_i^\top A^{-1}AA^{-1}x_i}{(1-h)^2}=\frac{e_i^2h}{(1-h)^2}$; dividing by $ps^2$ and using $r_i^2=\frac{e_i^2}{s^2(1-h)}$ gives (3).
(4) For every $j$, $Y_j-x_j^\top\hat\beta_{(i)}=e_j+x_j^\top(\hat\beta-\hat\beta_{(i)})=e_j+\frac{h_{ji}e_i}{1-h}$. Summing squares over all $j$ and using $\sum_jh_{ji}e_j=(He)_i=0$ and $\sum_jh_{ji}^2=h$:
$$\sum_{j=1}^n\big(Y_j-x_j^\top\hat\beta_{(i)}\big)^2=SSE+\frac{e_i^2h}{(1-h)^2}.$$The $j=i$ term is $e_{(i)}^2=\frac{e_i^2}{(1-h)^2}$, and the remaining terms form the residual sum of squares of the deleted fit, so $SSE_{(i)}=SSE+\frac{e_i^2h-e_i^2}{(1-h)^2}=SSE-\frac{e_i^2}{1-h}$. The deleted fit has $n-1$ cases and $p$ coefficients, so $(n-p-1)s_{(i)}^2=(n-p)s^2-s^2r_i^2$, i.e. $s^2(n-p-r_i^2)$; this is positive (when $n-p\ge2$, the generic case) so $r_i^2\lt n-p$, and $t_i=r_is/s_{(i)}$ gives the formula.
(5) The deleted fit uses only the other $n-1$ cases, so $\hat\beta_{(i)}$ and $s_{(i)}^2$ are independent of each other (Theorem 6.24(3)) and of $Y_i$. Hence $e_{(i)}=Y_i-x_i^\top\hat\beta_{(i)}$ is normal with mean $0$ and variance $\sigma^2\big(1+x_i^\top(A-x_ix_i^\top)^{-1}x_i\big)$, which by Sherman–Morrison is $\sigma^2\big(1+h+\frac{h^2}{1-h}\big)=\frac{\sigma^2}{1-h}$, independent of $(n-p-1)s_{(i)}^2/\sigma^2\sim\chi^2_{n-p-1}$. Therefore $\frac{e_{(i)}\sqrt{1-h}}{s_{(i)}}=\frac{e_i}{s_{(i)}\sqrt{1-h}}=t_i\sim t_{n-p-1}$. ∎
Formula (3) is the key to reading influence: a point is influential when it combines a sizeable studentised residual with a sizeable leverage. A large residual at low leverage moves the fit little; a high-leverage point that lies on the trend of the others moves it little; a high-leverage point off the trend moves it a lot. Note also that $r_i$ is not $t$-distributed (it is bounded by $\sqrt{n-p}$), while $t_i$ is exactly $t_{n-p-1}$ — which is why formal outlier tests use $t_i$, with a Bonferroni correction when all $n$ cases are screened.
- Residuals against fitted values ($e_i$ or $r_i$ against $\hat Y_i$). A curve suggests a missing term or a transformation of $x$; a funnel (spread growing with $\hat y$) suggests a variance-stabilising transformation of $y$ or weighted least squares. Plot against $\hat Y_i$, not against $Y_i$: residuals are uncorrelated with $\hat y$ but correlated with $y$ (Exercise III.5).
- Residuals against each regressor, against omitted variables, and in time order. Systematic patterns, or long runs of the same sign (the runs test of §5.3; the Durbin–Watson statistic for serial correlation), indicate model inadequacy or dependent errors.
- Normal probability (Q–Q) plot of the $r_i$: curvature indicates skewness, S-shapes heavy tails.
- Leverages $h_{ii}$ against $2p/n$; outliers via $|t_i|$; influence via $D_i$ against $1$ (and $4/n$).
- Multicollinearity: VIFs, and signs or sizes of coefficients that contradict the simple correlations.
- Act honestly. Check flagged cases for recording errors; refit with and without them and report both. Never delete a point only because it spoils the fit — it may be the most informative observation you have.
Transformations. Many non-linear relations become linear after a transformation, and the error structure changes with it. A multiplicative model $y=ae^{bx}\cdot e^{\varepsilon}$ becomes $\log y=\log a+bx+\varepsilon$; a power law $y=ax^b e^{\varepsilon}$ becomes $\log y=\log a+b\log x+\varepsilon$; $y=\frac{x}{a+bx}$ becomes $\frac1y=b+a\cdot\frac1x$. When the variance of $Y$ depends on its mean, $\operatorname{Var}(Y)\approx\phi(\mu)$, the delta method (§2.3) shows that $g(Y)$ has approximately constant variance when $g'(\mu)\propto\phi(\mu)^{-1/2}$: counts with $\operatorname{Var}\propto\mu$ call for $\sqrt y$, a constant coefficient of variation ($\operatorname{Var}\propto\mu^2$) for $\log y$, and binomial proportions for $\arcsin\sqrt y$. When the variances are known up to a constant, $\operatorname{Var}(\varepsilon_i)=\sigma^2/w_i$, weighted least squares minimises $\sum w_i(y_i-x_i^\top\beta)^2$, giving $\hat\beta_W=(X^\top WX)^{-1}X^\top Wy$ with $W=\operatorname{diag}(w_i)$; it is ordinary least squares applied to $\sqrt{w_i}\,y_i$ and $\sqrt{w_i}\,x_i$, and therefore BLUE (the generalised least-squares case of §6.1).
For a positive response, the Box–Cox transformation with parameter $\lambda$ is
$$y^{(\lambda)}=\begin{cases}\dfrac{y^\lambda-1}{\lambda},&\lambda\ne0,\\[4pt] \log y,&\lambda=0,\end{cases}$$which is continuous in $\lambda$ (since $\frac{y^\lambda-1}{\lambda}\to\log y$ as $\lambda\to0$). Assuming the normal linear model holds for $y^{(\lambda)}$ for some $\lambda$, the profile log-likelihood is $\ell(\lambda)=-\frac n2\log\frac{SSE(\lambda)}{n}+(\lambda-1)\sum\log y_i$, where $SSE(\lambda)$ is the residual sum of squares of the regression of $y^{(\lambda)}$ on $X$ and the last term is the log-Jacobian of $y\mapsto y^{(\lambda)}$. One chooses $\hat\lambda$ maximising $\ell$ (usually rounded to an interpretable value: $1$ none, $\frac12$ square root, $0$ log, $-1$ reciprocal); $\{\lambda:\ell(\lambda)\ge\ell(\hat\lambda)-\frac12\chi^2_{1,\alpha}\}$ is an approximate confidence interval (Wilks, §5.1).
After fitting on the log scale, $\exp(x_0^\top\hat\beta)$ estimates the median of $Y$ at $x_0$, not its mean (the mean is larger by the factor $e^{\sigma^2/2}$ under normal errors); prediction limits, being quantiles, simply transform back.
Logistic Regression
Suppose the response is binary: $Y=1$ (success, disease, default) or $Y=0$. Then $E[Y\mid x]=P(Y=1\mid x)=\pi(x)$, and a linear model $\pi(x)=x^\top\beta$ — the linear probability model — has three defects: fitted "probabilities" can leave $[0,1]$, the variance $\pi(1-\pi)$ is not constant, and the errors are far from normal. The standard remedy is to model a transformation of $\pi$ that ranges over the whole real line.
For $\pi\in(0,1)$ the odds are $\pi/(1-\pi)$ and the log-odds or logit is $\operatorname{logit}\pi=\log\frac{\pi}{1-\pi}$, a bijection $(0,1)\to\mathbb{R}$ with inverse the logistic function $\pi=\frac{e^\eta}{1+e^\eta}=\frac1{1+e^{-\eta}}$. In the logistic regression model, $Y_1,\dots,Y_N$ are independent,
$$Y_i\sim\operatorname{Bin}(m_i,\pi_i),\qquad \operatorname{logit}\pi_i=\eta_i=x_i^\top\beta,$$with known $m_i\ge1$ ($m_i=1$ for individual binary data; $m_i\gt1$ for grouped data with $Y_i$ successes in $m_i$ trials at the same $x_i$), known regressor vectors $x_i$ with first entry $1$, and $X$ (rows $x_i^\top$) of full column rank $p$. Interpretation: increasing $x_j$ by one unit, the others fixed, adds $\beta_j$ to the log-odds, i.e. multiplies the odds by $e^{\beta_j}$, the odds ratio for that regressor. The effect on the probability itself is $\partial\pi/\partial x_j=\beta_j\pi(1-\pi)$, largest ($\beta_j/4$) where $\pi=\frac12$. With one regressor, $\pi=\frac12$ at $x=-\beta_0/\beta_1$ (the median effective dose, ED50 or LD50, in dose–response studies).
Writing $\eta=X\beta$ and $\pi_i=\pi_i(\beta)$, the log-likelihood (up to a constant), score and Hessian are
$$\ell(\beta)=\sum_{i=1}^N\big[y_i\eta_i-m_i\log(1+e^{\eta_i})\big],$$ $$U(\beta)=\nabla\ell=X^\top(y-\mu),$$ $$\nabla^2\ell=-X^\top WX,$$where $\mu_i=m_i\pi_i$ and $W=\operatorname{diag}\big(m_i\pi_i(1-\pi_i)\big)$. The observed and expected Fisher information coincide: $I(\beta)=X^\top WX$. The function $\ell$ is strictly concave, so the MLE $\hat\beta$, if it exists, is unique and is the only solution of the likelihood equations $X^\top(y-\hat\mu)=0$. With an intercept these include $\sum_i m_i\hat\pi_i=\sum_iy_i$: the fitted number of successes equals the observed number.
Proof. The binomial log-probability is $y\log\pi+(m-y)\log(1-\pi)+\text{const}$, that is $y\log\frac{\pi}{1-\pi}+m\log(1-\pi)+\text{const}$, and $\log(1-\pi)=-\log(1+e^\eta)$; summing gives $\ell$. Since $\frac{d}{d\eta}\log(1+e^\eta)=\pi$ and $\frac{d\pi}{d\eta}=\pi(1-\pi)$, we get $\frac{\partial\ell}{\partial\eta_i}=y_i-m_i\pi_i$ and $\frac{\partial^2\ell}{\partial\eta_i^2}=-m_i\pi_i(1-\pi_i)$, and the chain rule with $\partial\eta_i/\partial\beta=x_i$ gives $U$ and $\nabla^2\ell=-\sum_im_i\pi_i(1-\pi_i)x_ix_i^\top=-X^\top WX$. This does not involve $y$, so the expected information $E[-\nabla^2\ell]$ equals the observed one. For $v\ne0$, $v^\top X^\top WXv=\sum_iw_i(x_i^\top v)^2\gt0$ because every $w_i\gt0$ and $Xv\ne0$ (full rank); so the Hessian is negative definite everywhere and $\ell$ is strictly concave. A strictly concave function has at most one maximiser, and any stationary point is its global maximiser. The first row of $X^\top$ is $\mathbf 1^\top$, giving $\sum(y_i-m_i\hat\pi_i)=0$. ∎
The Newton–Raphson step for maximising $\ell$ from the current value $\beta$ is
$$\beta^{\text{new}}=\beta+(X^\top WX)^{-1}X^\top(y-\mu)$$ $$=(X^\top WX)^{-1}X^\top Wz,$$with $W$, $\mu$ evaluated at $\beta$ and the working response $z_i=\eta_i+\dfrac{y_i-\mu_i}{m_i\pi_i(1-\pi_i)}$. So each step is a weighted least-squares regression of $z$ on $X$ with weights $w_i=m_i\pi_i(1-\pi_i)$ (IRLS). Because observed and expected information coincide, Newton–Raphson and Fisher scoring are the same algorithm here; when the MLE exists the iterates converge to it quadratically from any starting point close enough to it, and in practice from $\beta=0$.
Proof. Newton's method for a stationary point of $\ell$ is $\beta^{\text{new}}=\beta-(\nabla^2\ell)^{-1}\nabla\ell$, which by Theorem 6.29 is $\beta+(X^\top WX)^{-1}X^\top(y-\mu)$. Multiply out:
$$X^\top WX\beta+X^\top(y-\mu)$$ $$=X^\top W\big(X\beta+W^{-1}(y-\mu)\big)=X^\top Wz,$$so $\beta^{\text{new}}=(X^\top WX)^{-1}X^\top Wz$, which is the weighted least-squares solution. Quadratic convergence is the standard property of Newton's method at a root where the Jacobian (here $-X^\top WX$) is nonsingular and the function is twice continuously differentiable. ∎
For individual binary data ($m_i=1$), the MLE exists if and only if there is no separating direction: no $b\ne0$ with $x_i^\top b\ge0$ for every $i$ with $y_i=1$ and $x_i^\top b\le0$ for every $i$ with $y_i=0$. (Complete separation: the inequalities can be made strict; quasi-complete: only weakly.) With one regressor and an intercept, the MLE exists if and only if both outcomes occur and
$$\max\{x_i:y_i=0\}\gt\min\{x_i:y_i=1\}\quad\text{and}$$ $$\max\{x_i:y_i=1\}\gt\min\{x_i:y_i=0\},$$i.e. neither group's $x$-values lie (weakly) on one side of the other group's.
Proof. With $m_i=1$, the $i$-th term of $\ell$ is $-\log(1+e^{-\eta_i})$ if $y_i=1$ (increasing in $\eta_i$) and $-\log(1+e^{\eta_i})$ if $y_i=0$ (decreasing in $\eta_i$); every term is negative.
Separation ⇒ no MLE. If $b$ separates, then along $\beta+tb$ ($t\gt0$) each $\eta_i$ changes by $t\,x_i^\top b$, which is $\ge0$ when $y_i=1$ and $\le0$ when $y_i=0$; so every term is non-decreasing in $t$. Since $Xb\ne0$, some $x_i^\top b\ne0$ and that term is strictly increasing. Hence $\ell(\beta+tb)\gt\ell(\beta)$ for every $\beta$: no point is a maximiser, and Newton's iterates run off to infinity while $\ell\uparrow0$ (or to a finite supremum in the quasi-complete case).
No separation ⇒ MLE exists. For each unit vector $b$ some case violates the separation inequalities: $y_i=0$ with $x_i^\top b\gt0$, or $y_i=1$ with $x_i^\top b\lt0$. That term tends to $-\infty$ as $t\to\infty$ along $tb$, and the others are $\le0$, so $\ell(tb)\to-\infty$. Fix $b_0$ and choose $T_0$ with $\ell(T_0b_0)\lt\ell(0)-1$. By continuity, $\ell(T_0b)\lt\ell(0)-\frac12$ for all unit $b$ in a neighbourhood $N_0$ of $b_0$, and by concavity $\ell(tb)\le\ell(0)+\frac{t}{T_0}\big(\ell(T_0b)-\ell(0)\big)\lt\ell(0)$ for all $t\ge T_0$, $b\in N_0$. Finitely many such neighbourhoods cover the unit sphere (compactness); with $T$ the largest of their $T_0$'s, $\ell(\beta)\lt\ell(0)$ whenever $\|\beta\|\ge T$. So the maximum of the continuous $\ell$ over the compact ball $\|\beta\|\le T$ is a global maximum.
One regressor. For $b=(b_0,b_1)$: if $b_1\gt0$, separation means some threshold $c=-b_0/b_1$ has all $x_i\ge c$ for $y_i=1$ and all $x_i\le c$ for $y_i=0$, i.e. $\max\{x:y=0\}\le\min\{x:y=1\}$; $b_1\lt0$ gives the mirror image; and $b_1=0$, $b_0\ne0$ separates only if all $y_i$ are equal. ∎
Under the usual regularity conditions (the true $\beta$ fixed, $X^\top WX\to\infty$ in all directions with no regressor dominating), $\hat\beta$ exists with probability tending to $1$ and is approximately $N_p\big(\beta,(X^\top\hat WX)^{-1}\big)$ (§4.2). Hence, with $\operatorname{SE}(\hat\beta_j)$ the square root of the $j$-th diagonal entry of $(X^\top\hat WX)^{-1}$:
- Wald test of $\beta_j=0$: $z=\hat\beta_j/\operatorname{SE}(\hat\beta_j)\approx N(0,1)$; confidence interval for the odds ratio: $\exp\big(\hat\beta_j\pm z_{\alpha/2}\operatorname{SE}(\hat\beta_j)\big)$.
- Likelihood ratio test of $q$ linear restrictions: $G^2=2\big[\ell(\hat\beta)-\ell(\hat\beta_0)\big]\approx\chi^2_q$ under $H_0$ (Wilks' theorem, §5.1).
- For grouped data with all $m_i$ large, the deviance $D=2\big[\ell_{\text{sat}}-\ell(\hat\beta)\big]$, where $\ell_{\text{sat}}$ is the log-likelihood of the saturated model $\hat\pi_i=y_i/m_i$, is approximately $\chi^2_{N-p}$ when the model is correct, a goodness-of-fit test in the spirit of §5.2. Explicitly, $$\begin{aligned}D=2\sum_i\Big[&y_i\log\frac{y_i}{\hat\mu_i}\\&+(m_i-y_i)\log\frac{m_i-y_i}{m_i-\hat\mu_i}\Big].\end{aligned}$$
These are applications of the general MLE theory of §4.2 and the likelihood ratio theory of §5.1 to the concave likelihood of Theorem 6.29; we do not repeat the proofs. One case can be done exactly by hand.
Let $x\in\{0,1\}$, and suppose that among the $a+b$ cases with $x=1$ there are $a$ successes and among the $c+d$ cases with $x=0$ there are $c$ successes, with $a,b,c,d\gt0$. In the model $\operatorname{logit}\pi=\beta_0+\beta_1x$,
$$\hat\beta_0=\log\frac cd,\qquad \hat\beta_1=\log\frac{ad}{bc},$$ $$\operatorname{SE}(\hat\beta_1)=\sqrt{\frac1a+\frac1b+\frac1c+\frac1d}.$$So $e^{\hat\beta_1}$ is the sample odds ratio of the $2\times2$ table, and the Wald interval for it is the interval for the odds ratio met in §5.2.
Proof. Group the data by $x$. The likelihood equations of Theorem 6.29 are $\sum(y-\mu)=0$ and $\sum x(y-\mu)=0$; the second says the fitted number of successes in group $x=1$ is $a$, and then the first says that in group $x=0$ it is $c$. So $\hat\pi_1=\frac a{a+b}$, $\hat\pi_0=\frac c{c+d}$, and $\hat\beta_0=\operatorname{logit}\hat\pi_0=\log\frac cd$, $\hat\beta_0+\hat\beta_1=\log\frac ab$. Since $\ell$ is strictly concave, this solution is the MLE. At the MLE the weights are $w_1=(a+b)\hat\pi_1(1-\hat\pi_1)=\frac{ab}{a+b}$ and $w_0=\frac{cd}{c+d}$, and $X^\top\hat WX=\begin{pmatrix}w_0+w_1&w_1\\w_1&w_1\end{pmatrix}$, whose inverse has $(2,2)$ entry $\frac{w_0+w_1}{w_0w_1}=\frac1{w_0}+\frac1{w_1}$, and this is $\frac{c+d}{cd}+\frac{a+b}{ab}=\frac1a+\frac1b+\frac1c+\frac1d$. ∎
- Small residual does not mean harmless. In Example 5 the point $(15,25)$ has a residual of only $-0.27$ but Cook's distance $3.5$: its high leverage ($0.82$) pulls the line onto itself.
- Large residual does not mean influential. In simple regression an outlier at $x=\bar x$ has the minimum leverage $1/n$; it shifts the intercept and inflates $s$, but leaves the slope unchanged (Interactive 1, "outlier near the centre").
- Insignificant $t$'s, significant $F$. With collinear regressors (Example 3, $\operatorname{VIF}=29$) every individual $t$ test can fail while the overall $F$ test is overwhelming. Conversely, a significant overall $F$ does not single out any regressor.
- $R^2$ always rises when a regressor is added, even a column of random numbers; compare models by $\bar R^2$, the extra-sum-of-squares $F$ test, or prediction error (PRESS), never by $R^2$.
- Uncorrelated with $y$ is not useless. A suppressor variable with zero correlation with $y$ can raise $R^2$ from $0.5$ to $1$ (Example 4(c)).
- Without an intercept, $\sum e_i\ne0$ in general, $SST\ne SSR+SSE$, and $1-SSE/SST$ can be negative (Exercise II.3); software then reports an uncentred $R^2$ that is not comparable with the usual one.
- High $R^2$ does not mean a correct model: a curved relation can have $R^2=0.95$ and a residual plot shaped like a smile. Low $R^2$ does not mean no relation: with $n=1000$, $R^2=0.02$ gives $F\approx20$, a highly significant slope.
- Residuals are not the errors. They are correlated ($\operatorname{Cov}(e_i,e_j)=-\sigma^2h_{ij}$) with unequal variances $\sigma^2(1-h_{ii})$, and they are correlated with $y$ ($\operatorname{corr}(e,y)=\sqrt{1-R^2}$): plot them against $\hat y$, which is uncorrelated with them.
- Prediction intervals do not shrink to zero. More data pin down the line, not the next observation: the half-width tends to $z_{\alpha/2}\sigma$.
- Back-transforming a log model estimates the median, not the mean, of $y$.
- An odds ratio is not a risk ratio. In Example 6, $e^{\hat\beta_1}=3$ but $\hat\pi_1/\hat\pi_0=2.4$; "smokers are three times as likely" would be wrong.
- Logistic coefficients are on the log-odds scale. $\beta_j$ is not the change in probability per unit of $x_j$; that change, $\beta_j\pi(1-\pi)$, depends on where you are.
- Separation destroys the MLE. If the 0s and 1s are separated by a hyperplane, $\ell$ has no maximum; software reports a huge coefficient with a huger standard error and a tiny, meaningless Wald $z$ — precisely when the evidence of an effect is strongest. Use the likelihood ratio test, exact methods or penalised likelihood instead.
Exercises
- 1 For $n=10$ pairs, $\sum x=50$, $\sum y=120$, $\sum x^2=330$, $\sum xy=690$, $\sum y^2=1580$. Find $\hat\beta_1$, $\hat\beta_0$, $SSE$, $s^2$ and $R^2$.
- 2 For the data of part 1, under (N), test $H_0:\beta_1=0$ at the $5\%$ level and give a $95\%$ confidence interval for $\beta_1$.
- 3 Show that $\hat\beta_0$ and $\hat\beta_1$ are uncorrelated if and only if $\bar x=0$, and that under (N) they are then independent. Why do analysts often centre $x$ before fitting?
- 4 In the no-intercept model $Y_i=\beta x_i+\varepsilon_i$ (GM), show that $\hat\beta=\sum x_iY_i/\sum x_i^2$, $\operatorname{Var}(\hat\beta)=\sigma^2/\sum x_i^2$ and $E[SSE]=(n-1)\sigma^2$. For $x=(1,2,3)$, $y=(3,2,1)$ show that $\sum e_i\ne0$.
- 5 With $n=27$ pairs, the sample correlation is $r=0.40$. Test $H_0:\beta_1=0$ at the $5\%$ level. What is the smallest $|r|$ that would be significant?
- 6 Show that the confidence and prediction intervals of Theorem 6.22 are narrowest at $x_0=\bar x$. If $x_1,\dots,x_n$ is a design with $S_{xx}/n\to c\gt0$ and $\bar x\to\bar x_\infty$, find the limits of the two half-widths as $n\to\infty$.
- 7 (Applied) For the revision data of Example 1, give $95\%$ intervals for the mean score of students who revise $10$ hours and for one such student's score.
- 8 (Design) You may choose $n$ (even) values $x_i\in[-1,1]$. Which choice minimises $\operatorname{Var}(\hat\beta_1)$, and what is the minimum? Why might a statistician still not use it?
- 9 (Calibration) In Example 1 a new student scores $70$. Estimate the number of hours revised by inverting the fitted line. Why is this not the same as regressing $x$ on $y$?
- 1 A model $y=\beta_0+\beta_1x_1+\beta_2x_2+\varepsilon$ is fitted to $n=6$ cases with centred regressors, giving $X^\top X=\begin{pmatrix}6&0&0\\0&4&2\\0&2&4\end{pmatrix}$, $X^\top y=(30,10,14)^\top$ and $\sum y_i^2=212$. Find $\hat\beta$, $SSE$ and $s^2$.
- 2 For part 1 find $R^2$, $\bar R^2$, the $t$ statistics of the slopes and the overall $F$ statistic. What do you conclude at the $5\%$ level?
- 3 For $x=(1,2,3)$, $y=(3,2,1)$ and the no-intercept model of Exercise I.4, compute $1-SSE/SST$. What goes wrong?
- 4 With $n=20$, a model with $3$ regressors has $R^2=0.60$; adding $2$ more raises $R^2$ to $0.64$. Compute $\bar R^2$ for both models and the extra-sum-of-squares $F$. Is the addition worthwhile?
- 5 In a model with two regressors whose correlation is $0.95$, find the VIF and the factor by which the standard errors of the slopes exceed those of an otherwise identical design with uncorrelated regressors.
- 6 Show that if the centred regressors are mutually orthogonal, each $\hat\beta_j$ equals the slope of the simple regression of $y$ on $x_j$, and deleting a regressor does not change the other estimates.
- 7 (Frisch–Waugh) For the data of Example 3, regress $x_2$ on $x_1$ and $y$ on $x_1$, and regress the second set of residuals on the first. Verify that the slope is $\hat\beta_2=1$ and that $\|\tilde x_2\|^2=S_{22}(1-r_{12}^2)$.
- 8 For $x=1,2,\dots,10$, find the correlation between $x$ and $x^2$ and the VIF in the quadratic model $y=\beta_0+\beta_1x+\beta_2x^2+\varepsilon$. Show that with $u=x-5.5$ the regressors $u$ and $u^2$ are uncorrelated.
- 1 For the simple regression design $x=1,2,3,4,10$, compute all leverages. Which exceed $2p/n$? Check that they sum to $p$.
- 2 Prove that if the row $x_i^\top$ occurs $m$ times among the rows of $X$, then $h_{ii}\le1/m$. Deduce that a single point can have leverage $1$ only if it is unreplicated.
- 3 In a regression with $n=30$, $p=4$ and $s=1.5$, a case has $e_i=3.0$ and $h_{ii}=0.25$. Compute $r_i$, $t_i$ and $D_i$. Is it an outlier at the $5\%$ level (ignoring multiplicity)? Is it influential?
- 4 For $x=(0,1,2)$, $y=(0,2,1)$, compute the leverages, the residuals and PRESS from Theorem 6.28(2); check one deleted residual by refitting.
- 5 In a model with an intercept, show that the sample correlation between $e$ and $y$ is $\sqrt{1-R^2}$, while that between $e$ and $\hat y$ is $0$. What does this mean for residual plots?
- 6 The data $x=1,2,4,8$, $y=2.0,5.7,16.0,45.3$ appear to follow $y=ax^b$. Linearise, and estimate $a$ and $b$ by least squares.
- 7 Show that $\frac{y^\lambda-1}{\lambda}\to\log y$ as $\lambda\to0$, and explain why the Box–Cox profile log-likelihood contains the term $(\lambda-1)\sum\log y_i$.
- 8 In $Y_i=\beta x_i+\varepsilon_i$ with $x_i\gt0$ and $\operatorname{Var}(\varepsilon_i)=\sigma^2x_i$ (uncorrelated), show that the weighted least-squares estimator is $\tilde\beta=\sum Y_i/\sum x_i$ (the ratio estimator of §8.2), that it is BLUE, and compare its variance with that of $\sum x_iY_i/\sum x_i^2$.
Type paired values (commas or spaces). The lab fits $y=\beta_0+\beta_1x$ and lists, for each case, the leverage $h_{ii}$, residual $e_i$, studentised residuals $r_i$ and $t_i$, and Cook's distance $D_i$, flagging $h_{ii}\gt2p/n$, $|t_i|\gt2$ and $D_i\gt1$. Use it for Examples 1 and 5 and Exercises I.1, III.1 and III.4.
- 1 Convert $\pi=0.2$ to odds and log-odds, and odds of $3$ to a probability.
- 2 A fitted model for a disease is $\operatorname{logit}\hat\pi=-3+0.05x$ ($x$ = age in years). Find $\hat\pi$ at ages $40$ and $60$, the odds ratio for a $10$-year age difference, and the age at which $\hat\pi=\frac12$.
- 3 For $\pi(x)=1/\big(1+e^{-(\beta_0+\beta_1x)}\big)$ with $\beta_1\gt0$ and $x_{50}=-\beta_0/\beta_1$, show that $\pi(x_{50}+u)+\pi(x_{50}-u)=1$ for all $u$, and that the slope $\pi'(x)$ is largest at $x_{50}$, where it equals $\beta_1/4$.
- 4 Fit the linear probability model by least squares to $x=0,1,2,3,4,5$, $y=0,0,0,1,1,1$. Show that the fitted values leave $[0,1]$. Explain why $\operatorname{Var}(\varepsilon_i)$ cannot be constant in this model. What happens if you fit a logistic regression to these data?
- 5 For binary data $x=(-1,0,1,2)$, $y=(0,1,0,1)$, carry out one Newton–Raphson step from $\beta=(0,0)$.
- 6 Show that for $x=(1,2,3,4)$, $y=(0,0,1,1)$ the logistic MLE does not exist, whereas for $y=(0,1,0,1)$ it does. Relate the second data set to part 5 and write down its MLE, given that the MLE in part 5 is $(-0.4541,\ 0.9082)$.
- 7 A model gives $\operatorname{logit}\hat\pi=-4+0.05\,\text{age}+0.9\,\text{smoker}$. Find the odds ratio for smokers against non-smokers of the same age, the odds ratio for $10$ years of age, $\hat\pi$ for a $50$-year-old smoker, and the age at which a non-smoker has the same estimated risk as a $40$-year-old smoker.
- 8 (Case–control sampling) Suppose $P(Y=1\mid x)$ follows $\operatorname{logit}\pi=\beta_0+\beta_1x$ in the population, but cases ($Y=1$) are sampled with probability $\tau_1$ and controls with probability $\tau_0$, independently of $x$. Writing $S$ for the event that a unit is sampled, show that $\operatorname{logit}P(Y=1\mid x,S)$ equals $\beta_0+\log(\tau_1/\tau_0)+\beta_1x$. Why does this make logistic regression the natural tool for case–control studies?
- 9 A logistic fit gives $\hat\beta_1=0.693$ with $\operatorname{SE}=0.25$, and dropping $x_1$ lowers the maximised log-likelihood by $4.2$. Give a $95\%$ interval for the odds ratio $e^{\beta_1}$ and the $p$-value of the likelihood ratio test.
E1. CSIR Part B GATE ST In a simple linear regression of $y$ on $x$ based on $n=12$ observations, $S_{xx}=40$, $S_{xy}=60$ and $S_{yy}=100$. The value of the $F$ statistic for testing $H_0:\beta_1=0$ is (A) $9$; (B) $9.49$; (C) $90$; (D) $0.9$.
E2. CSIR Part B In a linear regression with $p=3$ coefficients fitted to $n=20$ cases, case $i$ has leverage $h_{ii}=0.5$ and internally studentised residual $r_i=2$. Its Cook's distance is (A) $0.67$; (B) $1.33$; (C) $2.67$; (D) $4$.
E3. CSIR Part B A logistic regression of a binary response on $x$ gives $\operatorname{logit}\hat\pi=-2+0.4x$. Which statement is correct? (A) At $x=5$ the estimated probability of success is $0.4$. (B) Increasing $x$ by $1$ multiplies the estimated odds of success by $e^{0.4}\approx1.49$. (C) Increasing $x$ by $1$ increases the estimated probability of success by $0.4$. (D) The estimated probability of success exceeds $\frac12$ for every $x\gt4$.
E4. CSIR Part C A linear regression of $y$ on $x_1,\dots,x_k$ with an intercept is fitted by least squares to $n\gt k+1$ observations, with residuals $e_i$ and fitted values $\hat y_i$, and $SSE\gt0$. Which of the following are necessarily true? (One or more options may be correct.) (A) $\sum e_i=0$. (B) $\sum e_iy_i=0$. (C) $\sum e_i\hat y_i=0$. (D) The sample correlation between $(e_i)$ and $(y_i)$ is $0$.
E5. CSIR Part C A linear regression with an intercept and $k$ regressors is fitted to $n\gt k+2$ observations. A further regressor $z$ is added, the design matrix still having full column rank. Which of the following are necessarily true? (A) $R^2$ does not decrease. (B) The adjusted $R^2$ does not decrease. (C) The adjusted $R^2$ increases if and only if the $t$ statistic of $z$ in the enlarged model exceeds $1$ in absolute value. (D) The residual mean square $s^2$ does not increase.
E6. CSIR Part C Binary responses $y_1,\dots,y_n$ are related to a real regressor $x$ by the logistic model $\operatorname{logit}\pi_i=\beta_0+\beta_1x_i$, fitted by maximum likelihood. Which of the following are true? (One or more options may be correct.) (A) If $y_i=1$ exactly for those $i$ with $x_i\gt3$, and both outcomes occur, the MLE of $(\beta_0,\beta_1)$ does not exist. (B) Whenever the MLE exists, Newton–Raphson started at $\beta=(0,0)$ reaches it in one step. (C) Whenever the MLE exists, $\sum_i\hat\pi_i=\sum_iy_i$. (D) Whenever the MLE exists, $e^{\hat\beta_1}$ estimates $P(Y=1\mid x+1)/P(Y=1\mid x)$.
Suggested plan — 3 lectures × 60 min.
- Simple regression as a statistical model. Definition 6.11, Theorems 6.19–6.21 (the orthonormal-basis proof is worth doing in full: it is the template for every $\chi^2$ and independence argument in linear models). Example 1 on the board; Theorem 6.22 and Example 2, then project Interactive 1 with both bands on and drag P along the trend to the far right, to show the bands widening away from $\bar x$. Close with Theorem 6.23 and the ANOVA table.
- Multiple regression. Recap Theorem 6.24 briefly (it was proved in §6.1), then spend the time on Theorems 6.25 and 6.26. Do Example 3 slowly — let the class predict the $t$ tests from the simple regressions before revealing them — and Example 4 as a "prove or disprove" discussion. Ten minutes on the hat matrix and Theorem 6.27.
- Diagnostics and logistic regression. Theorem 6.28 (derive Sherman–Morrison on the board), Example 5, then Interactive 1 in its three preset modes. Second half: Definition 6.19, Theorems 6.29–6.31, Example 6 (link to the odds ratio of §5.2) and Example 7 with Interactive 2, ending on the separated data set.
Discussion questions. (1) A point has leverage $0.9$ and residual $0.01$. Is it a good point or a bad point? What extra information would you need? (2) Two regressors are each highly significant alone and both insignificant together. What would you tell a client who asks "which one matters?" (3) A clinical model reports an odds ratio of $1.5$ for a common outcome (baseline risk $40\%$). A newspaper writes "50% more likely". Is that right?
Misconceptions to watch for. Treating residuals as independent errors with equal variance; judging influence from the size of the residual; using $R^2$ to compare models with different numbers of regressors or without an intercept; believing that insignificant $t$ statistics justify dropping all those variables together; reading logistic coefficients as changes in probability; confusing odds ratios with risk ratios; trusting a huge logistic coefficient with a tiny Wald $z$ (separation).
Suggested homework. I(1, 2, 4, 5, 7), II(1, 2, 4, 6, 8), III(1, 3, 4, 5, 8), IV(2, 4, 5, 6, 8); Exam corner E2, E4, E5.
Research question. How precisely does a dose–response experiment determine the median lethal dose LD50, and which of the three standard confidence intervals for it — delta method, Fieller, profile likelihood — can be trusted when the groups are small?
Why it matters. LD50 and ED50 estimates underpin the registration of pesticides and veterinary drugs, the potency comparison of vaccines and the toxicity classification of chemicals in regulatory frameworks around the world; ethical pressure to use fewer animals makes small-sample behaviour a live question in toxicology.
What to do.
- Take the insecticide data of Interactive 2: coded log-concentrations $x=0,1,\dots,5$, $20$ insects at each, with $2,3,8,11,17,18$ deaths. Write your own IRLS routine (Theorem 6.30) and confirm the iterates shown in the widget.
- Estimate $\text{LD50}=-\beta_0/\beta_1$ and its standard error by the delta method (§2.3), using the gradient $(-1/\beta_1,\ \beta_0/\beta_1^2)$ and $(X^\top\hat WX)^{-1}$.
- Compute Fieller's interval: the set of $\theta$ with $(\hat\beta_0+\theta\hat\beta_1)^2\le z_{0.025}^2\operatorname{Var}(\hat\beta_0+\theta\hat\beta_1)$, a quadratic inequality in $\theta$. Then compute the profile-likelihood interval by refitting with $\eta=\beta_1(x-\theta)$ for a grid of $\theta$.
- Simulate $2000$ experiments from the fitted model with $20$ insects per dose, and again with $5$; record how often each interval covers the true LD50 and how often Fieller's interval is unbounded.
Tools: Python (numpy, scipy) or R; the check lab and Interactive 2 for orientation.
Hand in: A two-page note with the three intervals for the real data, a coverage table for the two group sizes, and a recommendation for a laboratory that can afford only $30$ insects.
Going further. Compare the logit with the probit link $\Phi^{-1}(\pi)=\beta_0+\beta_1x$ on the same data, and read about optimal designs for estimating an ED50 (where should the doses be placed?).
🧪 Self-check quiz
Q1. A regression with an intercept and $4$ regressors is fitted to $n=40$ cases. The average leverage $\frac1n\sum h_{ii}$ is
Q2. At the same $x_0$, the prediction interval for a new $Y_0$ is wider than the confidence interval for $E[Y_0]$ because
Q3. In a model with two regressors whose sample correlation is $0.8$, the variance inflation factor of each slope is
Q4. A logistic model has $\operatorname{logit}\pi=-1+0.7x$. Increasing $x$ by one unit
Q5. Adding one regressor to a linear model with an intercept increases the adjusted $R^2$ exactly when