Chapter 6 · Linear Models and Regression

6.2 Analysis of Variance and Covariance

Comparing several means at once: the one-way analysis of variance with its partition of sums of squares and $F$-test, contrasts and multiple comparisons, two-way layouts with and without interaction, fixed, random and mixed effects with expected mean squares and variance components, and the analysis of covariance with one covariate.

An agronomist compares four varieties of wheat, a pharmacologist five doses of a drug, an engineer three suppliers of a component. Each wants to know whether the population means differ, and if so, which ones and by how much. Testing every pair with a two-sample $t$-test is tempting but wrong: with $6$ pairs, each tested at $5\%$, the chance of at least one false "discovery" is bounded only by $6\times0.05=0.30$ (Bonferroni's inequality), and would be $1-0.95^6=0.26$ if the tests were independent; moreover the separate tests waste information, because each uses only two groups to estimate the error variance. R. A. Fisher's answer, developed for the field trials at Rothamsted in the 1920s, was the analysis of variance (ANOVA): split the total variability of the data into a part explained by the groups and a part due to error, and compare the two with one $F$-test.

In the language of §6.1, every ANOVA is a linear model whose design matrix is made of indicator columns, the $F$-test is the test of a linear hypothesis, and the partitions of sums of squares are instances of Cochran's theorem. This section works out the standard layouts in detail: the one-way classification with contrasts and multiple comparisons; two-way classifications with one observation per cell and with replication, where interaction appears; random and mixed effects models, where the levels of a factor are a sample from a population and the questions concern variance components; and the analysis of covariance (ANCOVA), which combines ANOVA with regression on a concomitant variable. CSIR NET questions ask for degrees of freedom, $F$ values and expected mean squares (Part B) and for the correct denominator of an $F$-test, the estimability of effects and the interpretation of interaction (Part C). The designs of Chapter 8 (CRD, RBD, Latin squares, factorials) use exactly these tools. Upper points are written $F_{\nu_1,\nu_2,\alpha}$ as in §4.3.

One-Way Analysis of Variance

📘 Definition 6.6 — The one-way fixed effects model

Observations $y_{ij}$, $i=1,\dots,k$ (groups or treatments), $j=1,\dots,n_i$, follow $$y_{ij}=\mu_i+\varepsilon_{ij}=\mu+\alpha_i+\varepsilon_{ij},\qquad\varepsilon_{ij}\ \text{iid}\ N(0,\sigma^2),$$ with $N=\sum n_i$ observations in all. The group means $\mu_i$ are unknown constants (fixed effects); writing $\mu_i=\mu+\alpha_i$ with the side condition $\sum n_i\alpha_i=0$ makes $\mu=\bar\mu=\sum n_i\mu_i/N$ the weighted average mean and $\alpha_i=\mu_i-\bar\mu$ the effect of treatment $i$. Write $\bar y_i=\frac1{n_i}\sum_jy_{ij}$, $\bar y=\frac1N\sum_{ij}y_{ij}$, and $$\text{SST}=\sum_{ij}(y_{ij}-\bar y)^2,\quad\text{SSB}=\sum_in_i(\bar y_i-\bar y)^2,\quad\text{SSW}=\sum_{ij}(y_{ij}-\bar y_i)^2$$ for the total, between-groups (treatment) and within-groups (error) sums of squares.

By §6.1 the model has rank $k$; the estimable functions are the $\mu_i$ and their combinations, in particular every contrast $\sum c_i\alpha_i$ with $\sum c_i=0$; the BLUE of $\mu_i$ is $\bar y_i$; the fitted values are the group means; $\text{RSS}=\text{SSW}$ on $N-k$ degrees of freedom.

📐 Theorem 6.11 — The one-way ANOVA
  1. (Partition.) $\text{SST}=\text{SSB}+\text{SSW}$.
  2. (Expected mean squares, no normality needed.) With $\text{MSB}=\text{SSB}/(k-1)$ and $\text{MSW}=\text{SSW}/(N-k)$, $$E[\text{MSW}]=\sigma^2,\qquad E[\text{MSB}]=\sigma^2+\frac{\sum_in_i(\mu_i-\bar\mu)^2}{k-1}.$$
  3. (Distributions.) Under normality, $\text{SSW}/\sigma^2\sim\chi^2_{N-k}$, independent of $(\bar y_1,\dots,\bar y_k)$ and hence of SSB; $\text{SSB}/\sigma^2$ is noncentral $\chi^2_{k-1}(\delta)$ with $\delta=\sum n_i(\mu_i-\bar\mu)^2/\sigma^2$.
  4. (The $F$-test.) $H_0:\mu_1=\cdots=\mu_k$ is rejected at level $\alpha$ when $$F=\frac{\text{MSB}}{\text{MSW}}\gt F_{k-1,N-k,\alpha};$$ under $H_0$, $F\sim F_{k-1,N-k}$. This is the $F$-test of a general linear hypothesis (§6.1) and the likelihood ratio test.

Proof. (1) $y_{ij}-\bar y=(y_{ij}-\bar y_i)+(\bar y_i-\bar y)$. Squaring and summing, the cross term is $2\sum_i(\bar y_i-\bar y)\sum_j(y_{ij}-\bar y_i)=0$, because the inner sum is zero for each $i$.

(2) $\sum_j(y_{ij}-\bar y_i)^2=(n_i-1)S_i^2$, and $E[S_i^2]=\sigma^2$ (§4.1), so $E[\text{SSW}]=(N-k)\sigma^2$. For SSB use $\text{SSB}=\sum n_i\bar y_i^2-N\bar y^2$, with $E[\bar y_i^2]=\mu_i^2+\sigma^2/n_i$ and $E[\bar y^2]=\bar\mu^2+\sigma^2/N$: $$E[\text{SSB}]=\sum n_i\mu_i^2+k\sigma^2-N\bar\mu^2-\sigma^2=(k-1)\sigma^2+\sum n_i(\mu_i-\bar\mu)^2.$$

(3) For each group, $S_i^2$ is independent of $\bar y_i$ and $(n_i-1)S_i^2/\sigma^2\sim\chi^2_{n_i-1}$ (the fundamental theorem for normal samples, §4.1); the groups are independent, so SSW$/\sigma^2$ is a sum of independent $\chi^2$ variables with $\sum(n_i-1)=N-k$ degrees of freedom, independent of all the $\bar y_i$. For SSB put $z_i=\sqrt{n_i}\,\bar y_i\sim N(\sqrt{n_i}\mu_i,\sigma^2)$, independent, and $s=(\sqrt{n_1},\dots,\sqrt{n_k})^{\mathsf T}/\sqrt N$, a unit vector. Then $\sum n_i\bar y_i^2=\|z\|^2$ and $N\bar y^2=(s^{\mathsf T}z)^2$, so $\text{SSB}=\|z\|^2-(s^{\mathsf T}z)^2=\|Pz\|^2$, where $P=I-ss^{\mathsf T}$ is the projection onto $s^\perp$, of rank $k-1$. By the rotation argument used for the normal linear model in §6.1, $\|Pz\|^2/\sigma^2$ is $\chi^2_{k-1}$ with noncentrality $\|P\,E z\|^2/\sigma^2=\sum n_i(\mu_i-\bar\mu)^2/\sigma^2$.

(4) Under $H_0$, $\delta=0$ and $F$ is a ratio of independent $\chi^2$ variables divided by their degrees of freedom. In the language of the general $F$-test of §6.1: the full model has $\text{RSS}=\text{SSW}$ with $n-r=N-k$; the reduced model $E[y_{ij}]=\mu$ has $\text{RSS}_0=\text{SST}$; the hypothesis consists of $q=k-1$ independent contrasts; and $\text{RSS}_0-\text{RSS}=\text{SSB}$. ∎

The results are set out in an ANOVA table. For hand computation use the correction factor $\text{CF}=G^2/N$, where $G=\sum_{ij}y_{ij}$ is the grand total, and the group totals $T_i$: $$\text{SST}=\sum_{ij}y_{ij}^2-\text{CF},\qquad\text{SSB}=\sum_i\frac{T_i^2}{n_i}-\text{CF}.$$

SourcedfSSMS$E[\text{MS}]$$F$
Between groups$k-1$SSB$\frac{\text{SSB}}{k-1}$$\sigma^2+\frac{\sum n_i\alpha_i^2}{k-1}$$\frac{\text{MSB}}{\text{MSW}}$
Within groups$N-k$SSW$\frac{\text{SSW}}{N-k}$$\sigma^2$
Total$N-1$SST
✏️ Example 1 — Three feeds
Weight gains (kg) of young animals on three feeds: feed A: $20,23,25,22$; feed B: $27,25,29,30,24$; feed C: $22,19,21,24,20,20$. Construct the ANOVA table and test at $5\%$ whether the feeds differ in mean gain.
Totals $T_A=90$ ($n=4$, mean $22.5$), $T_B=135$ ($n=5$, mean $27$), $T_C=126$ ($n=6$, mean $21$); $G=351$, $N=15$, $\text{CF}=351^2/15=8213.4$, $\sum y^2=8371$.
Sums of squares $\text{SST}=8371-8213.4=157.6$. $\text{SSB}=\frac{90^2}4+\frac{135^2}5+\frac{126^2}6-8213.4=2025+3645+2646-8213.4=102.6$. $\text{SSW}=157.6-102.6=55.0$. (Check: the within-group sums of squares are $13+26+16=55$.)
Table Between: df $2$, MS $51.30$. Within: df $12$, MS $4.583$. $F=51.30/4.583=11.19$.
Decision $F_{2,12,0.05}=3.885$ (and $F_{2,12,0.01}=6.927$), so $H_0$ is rejected at $5\%$ and at $1\%$ ($p=0.0018$). The feeds differ; Example 2 asks which.

Contrasts and multiple comparisons

📘 Definition 6.7 — Contrasts and orthogonal contrasts

A contrast among the group means is $\psi=\sum_ic_i\mu_i$ with $\sum_ic_i=0$ (not all $c_i$ zero). Two contrasts with coefficient vectors $c$ and $d$ are orthogonal if $\sum_ic_id_i/n_i=0$ (for equal group sizes: $\sum c_id_i=0$). Pairwise differences $\mu_i-\mu_j$ are contrasts.

📐 Theorem 6.12 — Inference on contrasts; partition of SSB
  1. The BLUE of $\psi=\sum c_i\mu_i$ is $\hat\psi=\sum c_i\bar y_i$, with $\operatorname{Var}(\hat\psi)=\sigma^2\sum_ic_i^2/n_i$. Under normality, $$\frac{\hat\psi-\psi}{\sqrt{\text{MSW}\sum c_i^2/n_i}}\sim t_{N-k}.$$
  2. The single-degree-of-freedom sum of squares of a contrast is $\text{SS}(\hat\psi)=\hat\psi^2\big/\sum_i(c_i^2/n_i)$, and $\text{SS}(\hat\psi)/\text{MSW}=t^2$ for $H_0:\psi=0$.
  3. If $\psi_1,\dots,\psi_{k-1}$ are mutually orthogonal contrasts, then $$\text{SSB}=\text{SS}(\hat\psi_1)+\cdots+\text{SS}(\hat\psi_{k-1}),$$ and under normality these sums of squares are independent.

Proof. (1) $\hat\psi$ is linear and unbiased, and it is a function of the BLUEs $\bar y_i$ of the estimable $\mu_i$, so it is the BLUE (the Gauss–Markov theorem, §6.1). The variance follows from the independence of the $\bar y_i$, and the $t$-law from the confidence intervals of the normal linear model (§6.1).

(2), (3) Use $z_i=\sqrt{n_i}\bar y_i$ and the unit vector $s$ from the proof of Theorem 6.11. With $w_i=c_i/\sqrt{n_i}$ we have $\hat\psi=w^{\mathsf T}z$, $\sum c_i^2/n_i=\|w\|^2$ and $\text{SS}(\hat\psi)=(w^{\mathsf T}z)^2/\|w\|^2$, the squared length of the projection of $z$ on the direction of $w$. The condition $\sum c_i=0$ says $w^{\mathsf T}s\sqrt N=\sum c_i=0$, so $w\perp s$; orthogonality of two contrasts says $w^{\mathsf T}w'=\sum c_id_i/n_i=0$. Thus $k-1$ mutually orthogonal contrasts give an orthogonal basis $w_1,\dots,w_{k-1}$ of $s^\perp$, and $\text{SSB}=\|Pz\|^2=\sum_l(w_l^{\mathsf T}z)^2/\|w_l\|^2$. Coordinates of the normal vector $z$ (covariance $\sigma^2I$) along orthogonal directions are independent. ∎

Testing many contrasts at level $\alpha$ each inflates the chance of some false rejection. Three classical remedies give simultaneous confidence statements.

📐 Theorem 6.13 — Multiple comparisons (Bonferroni, Scheffé, Tukey)

In the normal one-way model, let $\hat\sigma^2=\text{MSW}$ on $\nu=N-k$ degrees of freedom.

  1. (Bonferroni.) For $m$ pre-specified contrasts, the intervals $\hat\psi_l\pm t_{\nu,\alpha/(2m)}\,\hat\sigma\sqrt{\sum c_{li}^2/n_i}$ hold simultaneously with probability at least $1-\alpha$.
  2. (Scheffé.) With probability exactly $1-\alpha$, simultaneously for all contrasts $c$, $$\left|\hat\psi_c-\psi_c\right|\le\sqrt{(k-1)F_{k-1,\nu,\alpha}}\;\hat\sigma\sqrt{\textstyle\sum c_i^2/n_i}.$$ Moreover, the $F$-test of Theorem 6.11 rejects $H_0$ if and only if some contrast $\hat\psi_c$ falls outside its Scheffé interval around $0$.
  3. (Tukey.) If $n_1=\cdots=n_k=n$, then with probability $1-\alpha$, simultaneously for all pairs, $|(\bar y_i-\bar y_j)-(\mu_i-\mu_j)|\le q_{k,\nu,\alpha}\,\hat\sigma/\sqrt n$, where $q_{k,\nu,\alpha}$ is the upper point of the studentized range $\left(\max_iZ_i-\min_iZ_i\right)/\sqrt{U/\nu}$ ($Z_i$ iid $N(0,1)$, $U\sim\chi^2_\nu$ independent). For unequal $n_i$ the Tukey–Kramer intervals replace $\hat\sigma/\sqrt n$ by $\hat\sigma\sqrt{\frac12\left(\frac1{n_i}+\frac1{n_j}\right)}$ (conservative).

Proof. (1) Boole's inequality: the probability that at least one of $m$ intervals fails is at most $m\cdot\alpha/m$.

(2) With $z_i=\sqrt{n_i}(\bar y_i-\mu_i)$, a vector of iid $N(0,\sigma^2)$ coordinates, and $w=(c_i/\sqrt{n_i})$, we have $\hat\psi_c-\psi_c=w^{\mathsf T}z$ and $w\perp s$. By the Cauchy–Schwarz inequality, $$\sup_{w\perp s,\ w\ne0}\frac{(w^{\mathsf T}z)^2}{\|w\|^2}=\|Pz\|^2,$$ attained at $w=Pz$. Now $\|Pz\|^2/\sigma^2\sim\chi^2_{k-1}$ independently of $\hat\sigma^2$, so $\|Pz\|^2/((k-1)\hat\sigma^2)\sim F_{k-1,\nu}$, and the event "all intervals hold" is exactly $\{\|Pz\|^2\le(k-1)\hat\sigma^2F_{k-1,\nu,\alpha}\}$, of probability $1-\alpha$. Under $H_0$ (all $\mu_i$ equal) $Pz$ computed with $\mu_i=\bar\mu$ gives $\|Pz\|^2=\text{SSB}$, so the supremum of $\text{SS}(\hat\psi_c)$ over contrasts is SSB, and "some contrast exceeds the Scheffé bound" is exactly "$\text{SSB}/((k-1)\text{MSW})\gt F_{k-1,\nu,\alpha}$".

(3) With equal $n$, $(\bar y_i-\mu_i)\sqrt n/\sigma$ are iid $N(0,1)$, independent of $\hat\sigma$, and $\max_{i,j}|(\bar y_i-\mu_i)-(\bar y_j-\mu_j)|\sqrt n/\hat\sigma$ is the studentized range. ∎

For pairwise comparisons Tukey's intervals are the shortest of the three; Bonferroni is good for a few planned contrasts; Scheffé is the price of complete freedom to look at any contrast, including ones suggested by the data. Fisher's least significant difference (unadjusted $t$-tests, used only after a significant $F$) controls the familywise error rate only when $k=3$.

✏️ Example 2 — Which feeds differ?
For the data of Example 1 ($\text{MSW}=4.583$ on $12$ df), (a) compare all three pairs of feeds with Bonferroni-adjusted tests at familywise level $5\%$ and compare the multipliers of the Bonferroni, Scheffé and Tukey–Kramer methods; (b) estimate the contrast "B against the average of A and C", find its sum of squares and the share of SSB that it explains.
Pairs $\bar y_B-\bar y_A=4.5$, $\operatorname{SE}=\sqrt{4.583(\frac15+\frac14)}=1.436$, $t=3.13$. $\bar y_B-\bar y_C=6.0$, $\operatorname{SE}=\sqrt{4.583(\frac15+\frac16)}=1.296$, $t=4.63$. $\bar y_A-\bar y_C=1.5$, $\operatorname{SE}=1.382$, $t=1.09$.
Multipliers Bonferroni with $m=3$: $t_{12,0.05/6}=2.779$. Scheffé: $\sqrt{2F_{2,12,0.05}}=\sqrt{2\times3.885}=2.788$. Tukey–Kramer: $q_{3,12,0.05}/\sqrt2=3.773/\sqrt2=2.668$. (Unadjusted: $t_{12,0.025}=2.179$.)
Decisions With any of the three adjusted multipliers, B differs from A ($3.13$) and from C ($4.63$), while A and C are not distinguishable ($1.09$). Feed B is best.
(b) Contrast $c=(-\frac12,1,-\frac12)$: $\hat\psi=27-\frac{22.5+21}{2}=5.25$, $\sum c_i^2/n_i=\frac{0.25}4+\frac15+\frac{0.25}6=0.3042$, $\operatorname{SE}=\sqrt{4.583\times0.3042}=1.181$, $t=4.45$. $\text{SS}(\hat\psi)=5.25^2/0.3042=90.62$, which is $88\%$ of $\text{SSB}=102.6$. The orthogonal complementary contrast (with these unequal $n_i$, coefficients proportional to $(34,5,-39)$) carries the remaining $11.98$.
Interactive 1 — Between and Within: Watching the One-Way $F$ Statistic

Each tick draws a fresh data set (seeded generator): three groups of size $n$ from normal populations with means $-\delta,0,\delta$ and $\sigma=1$. Short coloured bars mark the group means, the dashed line the grand mean. The blue vertical segments are the deviations $\bar y_i-\bar y$ that make up SSB; the faint red segments are the deviations $y_{ij}-\bar y_i$ that make up SSW. The bar on the right splits SST into the two parts. Set $\delta=0$: $H_0$ is true, $F$ hovers around $1$ and about $5\%$ of data sets are rejected. Increase $\delta$ or $n$ and watch SSB take over.

0.606
SSB (df)—
SSW (df)—
F and p-value—
data sets rejected at 5%—

Two-Way Classifications and Interaction

Often the units can be grouped by two factors: varieties and fields, machines and operators, doses and age groups. Including the second factor removes its variability from the error (the idea of blocking, §8.3) and lets us ask whether the factors act additively.

📘 Definition 6.8 — Two-way models: additive and with interaction

Factor A has levels $i=1,\dots,a$ and factor B levels $j=1,\dots,b$; errors are iid $N(0,\sigma^2)$.

  1. The additive model (one observation per cell): $y_{ij}=\mu+\alpha_i+\beta_j+\varepsilon_{ij}$, with $\sum_i\alpha_i=\sum_j\beta_j=0$.
  2. The model with interaction ($n\ge2$ observations per cell): $y_{ijk}=\mu+\alpha_i+\beta_j+\gamma_{ij}+\varepsilon_{ijk}$, $k=1,\dots,n$, with $\sum_i\alpha_i=\sum_j\beta_j=0$ and $\sum_i\gamma_{ij}=0$ for each $j$, $\sum_j\gamma_{ij}=0$ for each $i$.

With cell means $\mu_{ij}=E[y_{ijk}]$, the interaction is $\gamma_{ij}=\mu_{ij}-\bar\mu_{i\cdot}-\bar\mu_{\cdot j}+\bar\mu_{\cdot\cdot}$. There is no interaction iff $\mu_{ij}-\mu_{i'j}$ does not depend on $j$, i.e. the profiles $j\mapsto\mu_{ij}$ of the levels of A are parallel.

📐 Theorem 6.14 — Two-way ANOVA with one observation per cell

In the additive model with $a\times b$ observations, write $\bar y_{i\cdot},\bar y_{\cdot j},\bar y_{\cdot\cdot}$ for row, column and grand means, and $$\text{SSA}=b\sum_i(\bar y_{i\cdot}-\bar y_{\cdot\cdot})^2,\quad\text{SSB}=a\sum_j(\bar y_{\cdot j}-\bar y_{\cdot\cdot})^2,$$ $$\text{SSE}=\sum_{ij}(y_{ij}-\bar y_{i\cdot}-\bar y_{\cdot j}+\bar y_{\cdot\cdot})^2.$$ Then $\text{SST}=\text{SSA}+\text{SSB}+\text{SSE}$ with degrees of freedom $ab-1=(a-1)+(b-1)+(a-1)(b-1)$; $$E[\text{MSA}]=\sigma^2+\frac{b\sum\alpha_i^2}{a-1},\quad E[\text{MSB}]=\sigma^2+\frac{a\sum\beta_j^2}{b-1},\quad E[\text{MSE}]=\sigma^2;$$ the three sums of squares are independent, and $H_0:\alpha_i\equiv0$ is rejected when $\text{MSA}/\text{MSE}\gt F_{a-1,(a-1)(b-1),\alpha}$ (similarly for B).

Proof. Write $r_i=\bar y_{i\cdot}-\bar y_{\cdot\cdot}$, $c_j=\bar y_{\cdot j}-\bar y_{\cdot\cdot}$ and $$e_{ij}=y_{ij}-\bar y_{i\cdot}-\bar y_{\cdot j}+\bar y_{\cdot\cdot},$$ so that $y_{ij}-\bar y_{\cdot\cdot}=r_i+c_j+e_{ij}$. The three arrays are orthogonal as vectors in $\mathbb R^{ab}$. For instance, $\sum_je_{ij}=b\bar y_{i\cdot}-b\bar y_{i\cdot}-b\bar y_{\cdot\cdot}+b\bar y_{\cdot\cdot}=0$ for each $i$, so $\sum_{ij}r_ie_{ij}=\sum_ir_i\sum_je_{ij}=0$; similarly $\sum_{ij}c_je_{ij}=0$; and the row and column arrays are orthogonal because $\sum_i(\bar y_{i\cdot}-\bar y_{\cdot\cdot})=0$. Squaring gives the partition. The arrays lie in subspaces of dimensions $a-1$, $b-1$ and $(a-1)(b-1)$ (the row array is determined by $a$ numbers summing to zero, and so on), which together with the constant vectors fill $\mathbb R^{ab}$; Cochran's theorem (§6.1) gives independent $\chi^2$ laws. For expectations, substitute the model: $\bar y_{i\cdot}-\bar y_{\cdot\cdot}=\alpha_i+\bar\varepsilon_{i\cdot}-\bar\varepsilon_{\cdot\cdot}$, so $E[\text{SSA}]=b\sum\alpha_i^2+b(a-1)\sigma^2/b$; and the third array equals $\varepsilon_{ij}-\bar\varepsilon_{i\cdot}-\bar\varepsilon_{\cdot j}+\bar\varepsilon_{\cdot\cdot}$ (the parameters cancel), whose expected squared length is $(a-1)(b-1)\sigma^2$ by the unbiased estimation of $\sigma^2$ in §6.1 (it is the residual of a model of rank $a+b-1$). ∎

With one observation per cell an interaction cannot be separated from error: if $\gamma_{ij}\ne0$, then $E[\text{MSE}]=\sigma^2+\sum\gamma_{ij}^2/((a-1)(b-1))$, which inflates the denominator and makes the $F$-tests conservative. Tukey's one-degree-of-freedom test for non-additivity tests the special form $\gamma_{ij}=\lambda\alpha_i\beta_j$. The layout is the randomised block design of §8.3, with A as treatments and B as blocks.

✏️ Example 3 — Three fertilisers on four fields
Each of three fertilisers (rows) is applied to one plot in each of four fields (columns). Yields: fertiliser 1: $12,15,14,11$; fertiliser 2: $16,18,17,13$; fertiliser 3: $14,19,15,12$. Analyse with the additive two-way model, and compare with a one-way analysis that ignores the fields.
Means Row means $13,\ 16,\ 15$; column means $14,\ 17.33,\ 15.33,\ 12$; grand mean $14.667$; $\sum y^2=2650$, $\text{CF}=176^2/12=2581.33$.
Sums of squares $\text{SST}=68.67$; $\text{SSA}=4[(−1.667)^2+1.333^2+0.333^2]=18.67$; $\text{SSB}=3[(−0.667)^2+2.667^2+0.667^2+(−2.667)^2]=45.33$; $\text{SSE}=68.67-18.67-45.33=4.67$ on $6$ df, $\text{MSE}=0.778$.
Tests Fertilisers: $F=9.333/0.778=12.0\gt F_{2,6,0.05}=5.14$ ($p=0.008$). Fields: $F=15.11/0.778=19.4\gt F_{3,6,0.05}=4.76$ ($p=0.002$). Both factors matter.
Ignoring fields A one-way analysis puts the field variation into the error: $\text{SSW}=4.67+45.33=50.0$ on $9$ df, $\text{MSW}=5.56$, and $F=9.333/5.56=1.68\lt F_{2,9,0.05}=4.26$ ($p=0.24$). The fertiliser effect is invisible. Blocking (removing a known source of variation) is what made it detectable.
📐 Theorem 6.15 — Two-way ANOVA with $n$ observations per cell

In the model with interaction (Definition 6.8(2)), with cell means $\bar y_{ij\cdot}$, $$\begin{aligned}\text{SSA}&=bn\textstyle\sum_i(\bar y_{i\cdot\cdot}-\bar y_{\cdot\cdot\cdot})^2,\qquad \text{SSB}=an\textstyle\sum_j(\bar y_{\cdot j\cdot}-\bar y_{\cdot\cdot\cdot})^2,\\ \text{SSAB}&=n\textstyle\sum_{ij}(\bar y_{ij\cdot}-\bar y_{i\cdot\cdot}-\bar y_{\cdot j\cdot}+\bar y_{\cdot\cdot\cdot})^2,\qquad\text{SSE}=\textstyle\sum_{ijk}(y_{ijk}-\bar y_{ij\cdot})^2,\end{aligned}$$ and $\text{SST}=\text{SSA}+\text{SSB}+\text{SSAB}+\text{SSE}$ with degrees of freedom $$abn-1=(a-1)+(b-1)+(a-1)(b-1)+ab(n-1).$$ The four sums of squares are independent, and $$\begin{aligned}E[\text{MSA}]&=\sigma^2+\tfrac{bn\sum\alpha_i^2}{a-1},\qquad E[\text{MSB}]=\sigma^2+\tfrac{an\sum\beta_j^2}{b-1},\\ E[\text{MSAB}]&=\sigma^2+\tfrac{n\sum\gamma_{ij}^2}{(a-1)(b-1)},\qquad E[\text{MSE}]=\sigma^2.\end{aligned}$$ Each effect is tested by its mean square divided by MSE, with $ab(n-1)$ denominator degrees of freedom.

Proof (sketch). Write $y_{ijk}-\bar y_{\cdot\cdot\cdot}$ as the sum of the row, column and interaction arrays of the cell means (Theorem 6.14 applied to the $a\times b$ table of cell means, each entry repeated $n$ times) and the within-cell deviations $y_{ijk}-\bar y_{ij\cdot}$. The within-cell array is orthogonal to anything constant within cells, which gives the four-way partition; the dimensions are as stated and add up to $abn-1$, so Cochran's theorem applies. For the expectations, substitute the model: the row array has entries $\alpha_i+(\bar\varepsilon_{i\cdot\cdot}-\bar\varepsilon_{\cdot\cdot\cdot})$, with the interaction terms cancelling because of the side conditions, and similarly for the others; $\text{SSE}/\sigma^2\sim\chi^2_{ab(n-1)}$ because each cell is a normal sample of size $n$. ∎

🧭 Problem-solving strategy — reading a two-way analysis
  1. Check the degrees of freedom first. They must add to $N-1$; with one observation per cell there is no separate error line, and the "residual" is the interaction line.
  2. Test the interaction first. If it is significant, the main effects are averages over the levels of the other factor and may be misleading; compare the levels of A separately at each level of B (simple effects), and draw the interaction plot.
  3. If the interaction is negligible, interpret the main effects and compare their levels with the methods of Theorem 6.13, using MSE and the appropriate number of observations per level ($bn$ for a level of A).
  4. Unbalanced data (unequal cell counts) break the orthogonality: the sum of squares for A depends on whether B is already in the model (Exercise II.8). Then use the general $F$-test of §6.1 for each hypothesis.
✏️ Example 4 — A significant interaction hides a main effect
Two catalysts (A) are tried at three temperatures (B), with two runs per combination. Yields: catalyst 1: $(10,12)$, $(14,16)$, $(18,20)$; catalyst 2: $(13,15)$, $(15,17)$, $(13,15)$ at temperatures $1,2,3$. Analyse the data and interpret.
Cell means Catalyst 1: $11,15,19$; catalyst 2: $14,16,14$. Row means $15,\ 14.667$; column means $12.5,\ 15.5,\ 16.5$; grand mean $14.833$.
Sums of squares $\text{SSA}=6[(0.167)^2+(−0.167)^2]=0.33$; $\text{SSB}=4[(−2.333)^2+0.667^2+1.667^2]=34.67$; the interaction residuals $\bar y_{ij}-\bar y_{i\cdot}-\bar y_{\cdot j}+\bar y$ are $\mp1.667,\ \mp0.667,\ \pm2.333$ (upper sign for catalyst 1), so $\text{SSAB}=2\cdot2(1.667^2+0.667^2+2.333^2)=34.67$; each cell contributes $2$ to SSE, so $\text{SSE}=12$ on $6$ df, $\text{MSE}=2$. Check: $\text{SST}=81.67$. ✓
Tests A: $F=0.333/2=0.17$ (not significant). B: $F=17.33/2=8.67\gt F_{2,6,0.05}=5.14$ ($p=0.017$). AB: $F=8.67$ on $(2,6)$, also $p=0.017$.
Interpretation It would be wrong to conclude that the catalysts do not matter. Catalyst 1 gains steadily with temperature while catalyst 2 does not: catalyst 2 is better by $3$ at temperature 1 and catalyst 1 is better by $5$ at temperature 3. The two effects cancel in the row averages. The recommendation is a combination: catalyst 1 at temperature 3.
Interactive 2 — Interaction Plots: Parallel Profiles or Not

A $2\times3$ layout with $n=4$ observations per cell and $\sigma=1$. The sliders set the true main effect of A (gap between the blue and red profiles), the trend of B across its three levels, and the interaction (how much the slopes of the two profiles differ). Dashed lines are the true cell means $\mu_{ij}$; dots and solid lines are the sample cell means of a fresh data set (seeded generator), with the individual observations faint. With the interaction at $0$ the true profiles are parallel and the $F$-test for AB rejects only about $5\%$ of the time; make the interaction large and the lines cross, while the $F$ for A can become small even though A clearly matters at each level of B.

1.01.50.0
F for A (p)—
F for B (p)—
F for AB (p)—
MSE (18 df)—

Fixed, Random and Mixed Effects

In Example 1 the three feeds are the only feeds of interest: their effects are fixed constants, and conclusions concern these feeds. Suppose instead that five batches of raw material are drawn at random from a supplier's production, and several specimens are tested from each. Nobody cares about these particular batches; the question is how much batches vary in general. The batch effects are then random variables, and the parameter of interest is their variance.

📘 Definition 6.9 — Random and mixed effects models; variance components

The one-way random effects model is $$y_{ij}=\mu+a_i+\varepsilon_{ij},\qquad i=1,\dots,k,\ j=1,\dots,n,$$ where $a_i\sim N(0,\sigma_a^2)$ and $\varepsilon_{ij}\sim N(0,\sigma^2)$, all mutually independent. The parameters are $\mu$ and the variance components $\sigma_a^2$, $\sigma^2$; the intraclass correlation is $\rho=\sigma_a^2/(\sigma_a^2+\sigma^2)$. In a two-way layout, A and B may both be fixed, both random (with interaction $c_{ij}\sim N(0,\sigma_{ab}^2)$), or one of each: a mixed model. A factor is treated as random when its levels are a random sample from a larger population of levels to which the conclusions should apply.

📐 Theorem 6.16 — The balanced one-way random effects model

With $\text{SSA}=n\sum_i(\bar y_{i\cdot}-\bar y_{\cdot\cdot})^2$ on $k-1$ df and $\text{SSE}=\sum_{ij}(y_{ij}-\bar y_{i\cdot})^2$ on $k(n-1)$ df:

  1. $\operatorname{Var}(y_{ij})=\sigma_a^2+\sigma^2$, $\operatorname{Corr}(y_{ij},y_{ij'})=\rho$ for $j\ne j'$, and observations in different groups are independent;
  2. $\text{SSE}/\sigma^2\sim\chi^2_{k(n-1)}$ and $\text{SSA}/(\sigma^2+n\sigma_a^2)\sim\chi^2_{k-1}$, independently; hence $E[\text{MSE}]=\sigma^2$ and $E[\text{MSA}]=\sigma^2+n\sigma_a^2$;
  3. $F=\text{MSA}/\text{MSE}$ satisfies $F/(1+n\theta)\sim F_{k-1,k(n-1)}$ with $\theta=\sigma_a^2/\sigma^2$. The test of $H_0:\sigma_a^2=0$ rejects when $F\gt F_{k-1,k(n-1),\alpha}$, and its power is $P\left(F_{k-1,k(n-1)}\gt F_{k-1,k(n-1),\alpha}/(1+n\theta)\right)$;
  4. the ANOVA estimators $\hat\sigma^2=\text{MSE}$ and $\hat\sigma_a^2=(\text{MSA}-\text{MSE})/n$ are unbiased, and $\hat\sigma_a^2\lt0$ with positive probability;
  5. with $F_U=F_{k-1,k(n-1),\alpha/2}$ and $F_L=F_{k-1,k(n-1),1-\alpha/2}$, $$\left[\frac1n\left(\frac{F}{F_U}-1\right),\ \frac1n\left(\frac{F}{F_L}-1\right)\right]$$ is a $100(1-\alpha)\%$ confidence interval for $\theta=\sigma_a^2/\sigma^2$, and $\theta\mapsto\theta/(1+\theta)$ turns it into one for $\rho$.

Proof. (1) $\operatorname{Cov}(y_{ij},y_{ij'})=\operatorname{Var}(a_i)=\sigma_a^2$ for $j\ne j'$, while different groups share no random term.

(2) Within group $i$, $y_{ij}-\bar y_{i\cdot}=\varepsilon_{ij}-\bar\varepsilon_{i\cdot}$: the random effect cancels. So $\text{SSE}/\sigma^2\sim\chi^2_{k(n-1)}$ exactly as in the fixed model, and it is independent of the $\bar\varepsilon_{i\cdot}$ (§4.1) and of the $a_i$. The group means $\bar y_{i\cdot}=\mu+a_i+\bar\varepsilon_{i\cdot}$ are iid $N(\mu,\sigma_a^2+\sigma^2/n)$, so $\sum_i(\bar y_{i\cdot}-\bar y_{\cdot\cdot})^2\big/(\sigma_a^2+\sigma^2/n)\sim\chi^2_{k-1}$, i.e. $\text{SSA}/(\sigma^2+n\sigma_a^2)\sim\chi^2_{k-1}$, a function of the $\bar y_{i\cdot}$ and hence independent of SSE.

(3) $\dfrac{\text{MSA}/(\sigma^2+n\sigma_a^2)}{\text{MSE}/\sigma^2}=\dfrac{F}{1+n\theta}\sim F_{k-1,k(n-1)}$, and $P(F\gt c)=P\left(F/(1+n\theta)\gt c/(1+n\theta)\right)$.

(4) Unbiasedness follows from (2). $\hat\sigma_a^2\lt0$ iff $F\lt1$, which has probability $P(F_{k-1,k(n-1)}\lt1/(1+n\theta))\gt0$.

(5) $P\left(F_L\le F/(1+n\theta)\le F_U\right)=1-\alpha$, and solving the two inequalities for $\theta$ gives the interval. ∎

The test is the same as in the fixed model; what changes is the hypothesis ($\sigma_a^2=0$ instead of equal means), the interpretation, and the fact that the power has an exact central $F$ form. The table of expected mean squares is the key to every random or mixed analysis: to test a term, divide its mean square by the mean square whose expectation equals that of the term's mean square when the term is absent.

📐 Theorem 6.17 — Expected mean squares in the balanced two-way layout

For the $a\times b$ layout with $n$ observations per cell and interaction, the expected mean squares are as follows (mixed: A fixed with $\sum\alpha_i=0$, B and AB random, unrestricted model; $Q_A=\sum\alpha_i^2/(a-1)$, $Q_B=\sum\beta_j^2/(b-1)$, $Q_{AB}=\sum\gamma_{ij}^2/((a-1)(b-1))$).

SourcedfFixedRandomMixed (A fixed)
A$a-1$$\sigma^2+bnQ_A$$\sigma^2+n\sigma_{ab}^2+bn\sigma_a^2$$\sigma^2+n\sigma_{ab}^2+bnQ_A$
B$b-1$$\sigma^2+anQ_B$$\sigma^2+n\sigma_{ab}^2+an\sigma_b^2$$\sigma^2+n\sigma_{ab}^2+an\sigma_b^2$
AB$(a-1)(b-1)$$\sigma^2+nQ_{AB}$$\sigma^2+n\sigma_{ab}^2$$\sigma^2+n\sigma_{ab}^2$
Error$ab(n-1)$$\sigma^2$$\sigma^2$$\sigma^2$

Hence: in the fixed model every effect is tested against MSE; in the random model A and B are tested against MSAB and AB against MSE; in the mixed model the fixed factor A is tested against MSAB, and AB against MSE. Under normality and balance each such ratio has an exact $F$ distribution under its null hypothesis.

Proof (the random model; the others are similar). With $y_{ijk}=\mu+a_i+b_j+c_{ij}+\varepsilon_{ijk}$, $$\bar y_{i\cdot\cdot}-\bar y_{\cdot\cdot\cdot}=(a_i-\bar a)+(\bar c_{i\cdot}-\bar c_{\cdot\cdot})+(\bar\varepsilon_{i\cdot\cdot}-\bar\varepsilon_{\cdot\cdot\cdot});$$ the $b_j$ cancel because every row mean contains the same $\bar b$. The three parts are independent; $\bar c_{i\cdot}$ (an average of $b$ interaction terms) has variance $\sigma_{ab}^2/b$ and $\bar\varepsilon_{i\cdot\cdot}$ has variance $\sigma^2/(bn)$, and for iid variables $E\sum_i(u_i-\bar u)^2=(a-1)\operatorname{Var}(u_i)$. Hence $$E[\text{SSA}]=bn(a-1)\left[\sigma_a^2+\frac{\sigma_{ab}^2}{b}+\frac{\sigma^2}{bn}\right]=(a-1)\left[\sigma^2+n\sigma_{ab}^2+bn\sigma_a^2\right].$$ For the interaction line, $\bar y_{ij\cdot}-\bar y_{i\cdot\cdot}-\bar y_{\cdot j\cdot}+\bar y_{\cdot\cdot\cdot}$ contains only the $c$ and $\varepsilon$ terms, in the same double-centred form as the residual of Theorem 6.14, giving $(a-1)(b-1)(\sigma_{ab}^2+\sigma^2/n)$ times $n$. In the mixed model replace $a_i-\bar a$ by the constant $\alpha_i$ (with $\bar\alpha=0$). Exact $F$ laws: in the balanced case the four sums of squares are independent and each, divided by its expected mean square times its df, is $\chi^2$, by the same orthogonal decomposition as in Theorem 6.15. ∎

Some texts use a restricted mixed model, which imposes $\sum_ic_{ij}=0$ for each $j$. Then $E[\text{MSB}]=\sigma^2+an\sigma_b^2$ (no $\sigma_{ab}^2$ term, with the interaction variance defined slightly differently), and B is tested against MSE. Both versions test the fixed factor against MSAB. State which version you use; examination problems generally intend the unrestricted one unless they say otherwise. Variance components are estimated by equating mean squares to their expectations: in the random two-way model, $\hat\sigma_{ab}^2=(\text{MSAB}-\text{MSE})/n$, $\hat\sigma_a^2=(\text{MSA}-\text{MSAB})/(bn)$, $\hat\sigma_b^2=(\text{MSB}-\text{MSAB})/(an)$.

✏️ Example 5 — Variation between batches
Four specimens from each of $k=5$ randomly chosen batches of a polymer are tested for tensile strength. The analysis gives $\text{SSA}=120$ and $\text{SSE}=90$. Test whether batches vary, estimate the variance components and the intraclass correlation, and give $95\%$ confidence intervals for $\theta=\sigma_a^2/\sigma^2$, for $\rho$ and for $\sigma^2$.
Table $\text{MSA}=120/4=30$, $\text{MSE}=90/15=6$, $F=5.0$ on $(4,15)$. $F_{4,15,0.05}=3.056$: reject $H_0:\sigma_a^2=0$ ($p=0.009$).
Components $\hat\sigma^2=6$, $\hat\sigma_a^2=(30-6)/4=6$, $\hat\rho=6/(6+6)=0.5$: half of the variance of a single measurement is due to batches.
Interval for $\theta$ $F_{4,15,0.025}=3.804$, $F_{4,15,0.975}=0.1155$. Then $\frac14\left(\frac{5}{3.804}-1\right)=0.079$ and $\frac14\left(\frac5{0.1155}-1\right)=10.57$; so $0.079\le\theta\le10.57$ and $0.073\le\rho\le0.914$. With only five batches the data say little about the size of the batch variance.
Interval for $\sigma^2$ $\text{SSE}/\sigma^2\sim\chi^2_{15}$: $\left[\frac{90}{27.49},\frac{90}{6.262}\right]=(3.27,\ 14.37)$, much narrower because it rests on $15$ degrees of freedom.
✏️ Example 6 — The right denominator in a mixed model
Three machines (fixed) are run by four operators chosen at random from the workforce, with two parts per machine–operator combination. The mean squares are $\text{MSA}=40$ (machines), $\text{MSB}=18$ (operators), $\text{MSAB}=8$, $\text{MSE}=2$. Decide whether each statement is correct, and carry out the correct tests at $5\%$. (a) Machines should be tested by $\text{MSA}/\text{MSE}=20$ on $(2,12)$ df. (b) The interaction is tested by $\text{MSAB}/\text{MSE}$ on $(6,12)$ df. (c) In the unrestricted model operators are tested by $\text{MSB}/\text{MSAB}$.
(a) Incorrect In the mixed model $E[\text{MSA}]=\sigma^2+n\sigma_{ab}^2+bnQ_A$. Without machine effects this is $\sigma^2+n\sigma_{ab}^2=E[\text{MSAB}]$, not $\sigma^2$. The correct test is $F=40/8=5.0$ on $(2,6)$ df; $F_{2,6,0.05}=5.14$, so machines are not significant at $5\%$ ($p=0.053$). The wrong ratio $20$ on $(2,12)$ would give $p=0.0002$: the operator-to-operator inconsistency of the machine differences must be part of the yardstick, because the conclusion is meant to hold for the whole workforce.
(b) Correct $F=8/2=4.0$ on $(6,12)$; $F_{6,12,0.05}=3.00$: significant ($p=0.020$). $\hat\sigma_{ab}^2=(8-2)/2=3$.
(c) Correct $E[\text{MSB}]=\sigma^2+n\sigma_{ab}^2+an\sigma_b^2$, so $F=18/8=2.25$ on $(3,6)$; $F_{3,6,0.05}=4.76$: not significant ($p=0.18$); $\hat\sigma_b^2=(18-8)/6=1.67$. (In the restricted model one would use $18/2=9$ on $(3,12)$, $p=0.002$, a different conclusion; the choice of model matters.)

Analysis of Covariance

In a teaching experiment, students taught by different methods also differ in their prior knowledge; in an agricultural trial, plots differ in their initial number of plants; in a clinical trial, patients differ in their baseline blood pressure. If a covariate $x$ measured before treatment is linearly related to the response, adjusting for it removes part of the error variance and corrects for chance imbalances between groups. The analysis of covariance combines the one-way model with a common regression slope.

📘 Definition 6.10 — The one-way ANCOVA model

$$y_{ij}=\mu+\alpha_i+\beta(x_{ij}-\bar x_{\cdot\cdot})+\varepsilon_{ij},\qquad\varepsilon_{ij}\ \text{iid}\ N(0,\sigma^2),$$ $i=1,\dots,k$, $j=1,\dots,n_i$, $N=\sum n_i$, where $x_{ij}$ is a known covariate not influenced by the treatments. The slope $\beta$ is common to all groups (parallel regression lines). Write the within-group (error) sums of squares and products $$E_{xx}=\sum_{ij}(x_{ij}-\bar x_{i\cdot})^2,\quad E_{xy}=\sum_{ij}(x_{ij}-\bar x_{i\cdot})(y_{ij}-\bar y_{i\cdot}),\quad E_{yy}=\sum_{ij}(y_{ij}-\bar y_{i\cdot})^2,$$ and $T_{xx},T_{xy},T_{yy}$ for the same sums about the grand means.

📐 Theorem 6.18 — Analysis of covariance with one covariate

Assume $E_{xx}\gt0$. Then:

  1. the least squares estimates are $\hat\beta=E_{xy}/E_{xx}$ (the pooled within-group slope) and $\widehat{\mu+\alpha_i}=\bar y_{i\cdot}-\hat\beta(\bar x_{i\cdot}-\bar x_{\cdot\cdot})$, the adjusted treatment means;
  2. the error sum of squares is $\text{SSE}=E_{yy}-E_{xy}^2/E_{xx}$ on $N-k-1$ df, and $\hat\sigma^2=\text{SSE}/(N-k-1)$;
  3. under $H_0:\alpha_1=\cdots=\alpha_k$ the model is a single regression line, with $\text{SSE}_0=T_{yy}-T_{xy}^2/T_{xx}$, and $$F=\frac{(\text{SSE}_0-\text{SSE})/(k-1)}{\text{SSE}/(N-k-1)}\sim F_{k-1,N-k-1};$$
  4. $H_0:\beta=0$ is tested by $F=\dfrac{E_{xy}^2/E_{xx}}{\hat\sigma^2}\sim F_{1,N-k-1}$;
  5. the difference of two adjusted means has variance $$\sigma^2\left(\frac1{n_i}+\frac1{n_j}+\frac{(\bar x_{i\cdot}-\bar x_{j\cdot})^2}{E_{xx}}\right).$$

Proof. (1), (2) Put $m_i=\mu+\alpha_i-\beta\bar x_{\cdot\cdot}$. For fixed $\beta$, $\sum_{ij}(y_{ij}-m_i-\beta x_{ij})^2$ is minimised over $m_i$ by $m_i=\bar y_{i\cdot}-\beta\bar x_{i\cdot}$ (a sample mean minimises the sum of squared deviations). Substituting, $$S(\beta)=\sum_{ij}\left[(y_{ij}-\bar y_{i\cdot})-\beta(x_{ij}-\bar x_{i\cdot})\right]^2=E_{yy}-2\beta E_{xy}+\beta^2E_{xx},$$ a quadratic minimised at $\hat\beta=E_{xy}/E_{xx}$ with minimum $E_{yy}-E_{xy}^2/E_{xx}$. Then $\widehat{\mu+\alpha_i}=\hat m_i+\hat\beta\bar x_{\cdot\cdot}=\bar y_{i\cdot}-\hat\beta(\bar x_{i\cdot}-\bar x_{\cdot\cdot})$. The model has rank $k+1$ (the $k$ group indicators and $x$, which is not in their span because $E_{xx}\gt0$), so the error df are $N-k-1$.

(3) The same computation with one common intercept gives $\text{SSE}_0=T_{yy}-T_{xy}^2/T_{xx}$; the hypothesis consists of $k-1$ independent estimable contrasts, so the general $F$-test of §6.1 applies with $q=k-1$, $n-r=N-k-1$. (4) is the same $F$-test with $\text{RSS}_0=E_{yy}$ (the one-way model) and $\text{RSS}_0-\text{RSS}=E_{xy}^2/E_{xx}$.

(5) $\hat\beta=\sum_{ij}(x_{ij}-\bar x_{i\cdot})y_{ij}/E_{xx}$ has variance $\sigma^2/E_{xx}$, and $$\operatorname{Cov}(\bar y_{i\cdot},\hat\beta)=\frac{\sigma^2}{n_iE_{xx}}\sum_j(x_{ij}-\bar x_{i\cdot})=0.$$ The difference of adjusted means is $\bar y_{i\cdot}-\bar y_{j\cdot}-\hat\beta(\bar x_{i\cdot}-\bar x_{j\cdot})$, a sum of three uncorrelated terms with variances $\sigma^2/n_i$, $\sigma^2/n_j$ and $(\bar x_{i\cdot}-\bar x_{j\cdot})^2\sigma^2/E_{xx}$. ∎

The error sum of squares can be written $E_{yy}(1-r_w^2)$, where $r_w=E_{xy}/\sqrt{E_{xx}E_{yy}}$ is the within-group correlation: a covariate that is strongly correlated with the response within groups removes most of the error variance, at the cost of one degree of freedom. The adjustment also moves the treatment means towards what they would have been at a common covariate value $\bar x_{\cdot\cdot}$. Two assumptions need checking: that the slopes are equal (fit separate slopes and compare by an $F$-test, Exercise IV.5), and that the treatments do not affect the covariate (otherwise adjusting removes part of the treatment effect).

✏️ Example 7 — Teaching methods adjusted for a pre-test
Twelve students are assigned at random to three teaching methods, four each. Pre-test scores $x$ and post-test scores $y$ are: method 1: $x=(10,12,15,13)$, $y=(25,26,31,29)$; method 2: $x=(14,16,12,18)$, $y=(32,37,30,39)$; method 3: $x=(8,11,13,12)$, $y=(21,23,28,24)$. Carry out the analysis of covariance and compare it with a one-way ANOVA of $y$ alone.
Means $\bar x_{i\cdot}=12.5,\ 15,\ 11$ ($\bar x=12.833$); $\bar y_{i\cdot}=27.75,\ 34.5,\ 24$ ($\bar y=28.75$).
Sums of squares and products Within groups: $E_{xx}=13+20+14=47$, $E_{xy}=16.5+32+17=65.5$, $E_{yy}=22.75+53+26=101.75$. Total: $T_{xx}=79.67$, $T_{xy}=151.5$, $T_{yy}=328.25$.
Fit $\hat\beta=65.5/47=1.394$. $\text{SSE}=101.75-65.5^2/47=101.75-91.28=10.47$ on $12-3-1=8$ df, $\hat\sigma^2=1.309$. Under $H_0$: $\text{SSE}_0=328.25-151.5^2/79.67=40.15$.
Test $F=\frac{(40.15-10.47)/2}{1.309}=11.34\gt F_{2,8,0.05}=4.46$ ($p=0.005$): the methods differ after adjustment. The slope is highly significant: $F=91.28/1.309=69.8$ on $(1,8)$.
Adjusted means $27.75-1.394(12.5-12.833)=28.21$; $34.5-1.394(15-12.833)=31.48$; $24-1.394(11-12.833)=26.55$. Method 2 still leads, but by less than the raw means suggest (its students started higher). For methods 2 and 1, $\operatorname{SE}=\sqrt{1.309\left(\frac14+\frac14+\frac{2.5^2}{47}\right)}=0.910$ and $t=3.27/0.910=3.59$ on $8$ df.
Comparison One-way ANOVA of $y$: $\text{MSW}=101.75/9=11.31$, $F=(226.5/2)/11.31=10.02$ on $(2,9)$. The covariate reduced the error mean square from $11.31$ to $1.31$ (the within-group correlation is $r_w=0.947$, and $1-r_w^2=0.103$), which sharpens every comparison.
⚠️ Counterexamples and pitfalls
  1. Many $t$-tests are not one ANOVA. With $k=5$ groups there are $10$ pairs; testing each at $5\%$ gives a familywise error rate far above $5\%$ (Bonferroni's inequality bounds it only by $10\times0.05=0.50$). Use the $F$-test and an adjusted multiple-comparison method.
  2. A significant $F$ does not say which means differ. It guarantees that some contrast is Scheffé-significant (Theorem 6.13(2)), but that contrast need not be a pairwise difference; and a non-significant $F$ does not prove the means equal.
  3. No main effect does not mean no effect. In Example 4, catalyst has $F=0.17$, yet the catalyst changes the yield by $3$ to $5$ units at the extreme temperatures. With a large interaction, main effects are averages of opposite effects.
  4. One observation per cell: interaction and error are confounded. The "error" line then estimates $\sigma^2+$ interaction; a test for interaction needs replication or a model for its form (Tukey).
  5. Wrong denominators in random and mixed models. The fixed factor of a mixed model, and both main effects of a random model, are tested against MSAB, not MSE (Example 6); using MSE overstates significance.
  6. Negative variance-component estimates. $\hat\sigma_a^2=(\text{MSA}-\text{MSE})/n$ is negative whenever $F\lt1$; this is not an arithmetic error. Report $0$ (and note that the truncated estimator is biased) or use a likelihood-based method.
  7. Unequal variances with unequal group sizes. The $F$-test is fairly robust to non-normality and, with equal $n_i$, to unequal variances; but if the larger groups have the smaller variances it rejects far too often. Check the residuals.
  8. Unbalanced two-way data. SSA, SSB and SSAB no longer add up to the model sum of squares, and the sum of squares for A depends on the order of fitting (Exercise II.8).
  9. ANCOVA can lower $F$. Adjustment does not always increase the treatment $F$: if the groups differ in $y$ only because they differ in $x$, the adjusted means are close and the treatment effect vanishes (Exam corner E6).
  10. Covariates affected by treatment, or unequal slopes. Adjusting for a variable measured after treatment can remove the treatment effect; with non-parallel lines there is no single "adjusted difference", and it depends on the value of $x$.
  11. Fixed or random is a choice about the inference, not about the data. The same numbers can be analysed both ways; the random model generalises to unseen levels and pays for it with wider intervals.

Exercises

I
One-way ANOVA, contrasts and multiple comparisons.
  • 1 Prove that $\text{SST}=\text{SSB}+\text{SSW}$ in the one-way layout.
  • 2 Groups A: $8,10,12$; B: $12,14,13,15$; C: $9,7,8$. Construct the ANOVA table and test equality of means at $5\%$.
  • 3 Derive $E[\text{MSB}]=\sigma^2+\sum n_i(\mu_i-\bar\mu)^2/(k-1)$ without assuming normality.
  • 4 Show that for $k=2$ the ANOVA $F$ equals the square of the pooled two-sample $t$ statistic.
  • 5 With $k=4$ groups of $n=5$ and means $10,12,14,18$, show that the contrasts $(1,-1,0,0)$, $(1,1,-2,0)$, $(1,1,1,-3)$ are mutually orthogonal, compute their sums of squares, and check that they add up to SSB.
  • 6 An experiment has $k=6$ groups. How many pairwise comparisons are there, and at what level must each be tested for a Bonferroni familywise level of $5\%$?
  • 7 Prove that the one-way $F$-test rejects at level $\alpha$ if and only if at least one contrast $\hat\psi$ satisfies $\text{SS}(\hat\psi)\gt(k-1)\,\text{MSW}\,F_{k-1,N-k,\alpha}$.
  • 8 With $k=4$, $n=6$ per group, $\sigma=1$ and true means $(0,0,0,1)$, find the noncentrality parameter of the $F$-test and its power at $5\%$ (software for the noncentral $F$).
  • 9 Show that the side conditions $\sum n_i\alpha_i=0$ and $\alpha_k=0$ give different estimates of $\alpha_1$, but the same estimate of every contrast.
1 Write $y_{ij}-\bar y=(y_{ij}-\bar y_i)+(\bar y_i-\bar y)$; the cross term is $2\sum_i(\bar y_i-\bar y)\sum_j(y_{ij}-\bar y_i)=0$.
2 Means $10,\ 13.5,\ 8$; grand mean $10.8$; $\text{SSB}=3(0.64)+4(7.29)+3(7.84)=54.6$; $\text{SSW}=8+5+2=15$ on $7$ df. $F=(54.6/2)/(15/7)=27.3/2.143=12.74\gt F_{2,7,0.05}=4.74$ ($p=0.005$): reject.
3 $\text{SSB}=\sum n_i\bar y_i^2-N\bar y^2$ with $E\bar y_i^2=\mu_i^2+\sigma^2/n_i$, $E\bar y^2=\bar\mu^2+\sigma^2/N$; so $E[\text{SSB}]=(k-1)\sigma^2+\sum n_i\mu_i^2-N\bar\mu^2=(k-1)\sigma^2+\sum n_i(\mu_i-\bar\mu)^2$.
4 For $k=2$, $\bar y=(n_1\bar y_1+n_2\bar y_2)/N$ gives $\text{SSB}=\frac{n_1n_2}{N}(\bar y_1-\bar y_2)^2$ and $\text{MSW}=S_p^2$, so $F=\frac{(\bar y_1-\bar y_2)^2}{S_p^2(1/n_1+1/n_2)}=t^2$.
5 Dot products: $1-1=0$; $1-1+0=0$; $1+1-2=0$. $\text{SS}=\frac{(\sum c_i\bar y_i)^2}{\sum c_i^2/5}$: $\frac{(-2)^2}{0.4}=10$, $\frac{(-6)^2}{1.2}=30$, $\frac{(-18)^2}{2.4}=135$. $\text{SSB}=5\sum(\bar y_i-13.5)^2=5\times35=175=10+30+135$. ✓
6 $\binom62=15$ pairs, each at $0.05/15=0.00333$.
7 By the Cauchy–Schwarz argument of Theorem 6.13(2) (with $\mu_i$ replaced by a common value, which cancels in contrasts), $\max_c\text{SS}(\hat\psi_c)=\text{SSB}$, attained by $c_i\propto n_i(\bar y_i-\bar y)$. So some contrast exceeds $(k-1)\text{MSW}F_{k-1,N-k,\alpha}$ iff $\text{SSB}$ does, iff $F\gt F_{k-1,N-k,\alpha}$.
8 $\bar\mu=0.25$, $\sum(\mu_i-\bar\mu)^2=3(0.0625)+0.5625=0.75$, $\delta=6\times0.75=4.5$. Critical value $F_{3,20,0.05}=3.10$; power $P(F_{3,20}(4.5)\gt3.10)=0.335$.
9 The fitted group means are $\bar y_i$ in both cases. With $\sum n_i\alpha_i=0$: $\hat\mu=\bar y$, $\hat\alpha_i=\bar y_i-\bar y$. With $\alpha_k=0$: $\hat\mu=\bar y_k$, $\hat\alpha_i=\bar y_i-\bar y_k$. So $\hat\alpha_1$ differs, but $\sum c_i\hat\alpha_i=\sum c_i\bar y_i$ when $\sum c_i=0$ in both cases.
II
Two-way classifications.
  • 1 Analyse the $3\times3$ table (rows A, columns B, one observation per cell) with rows $5,7,6$; $8,9,10$; $5,6,4$ by the additive model.
  • 2 Verify that the row array $(\bar y_{i\cdot}-\bar y_{\cdot\cdot})$ and the column array $(\bar y_{\cdot j}-\bar y_{\cdot\cdot})$ of Theorem 6.14 are orthogonal.
  • 3 Show that if the additive model is fitted but interaction is present, $E[\text{MSE}]=\sigma^2+\sum\gamma_{ij}^2/((a-1)(b-1))$. What does this do to the $F$-tests for main effects?
  • 4 In a $3\times4$ layout with $n=3$ per cell, $\text{SSA}=24$, $\text{SSB}=45$, $\text{SSAB}=18$, $\text{SST}=159$. Complete the ANOVA table (fixed effects) and test all effects at $5\%$.
  • 5 For Example 4, estimate the interaction effects $\gamma_{ij}$ and draw the interaction plot. Which simple effect of catalyst is largest?
  • 6 Show that in a $2\times2$ layout with $n$ observations per cell, $\text{SSAB}=\frac n4\left(\bar y_{11}-\bar y_{12}-\bar y_{21}+\bar y_{22}\right)^2$.
  • 7 Explain why the hypothesis of no interaction cannot be tested with one observation per cell without further assumptions, and state the form of interaction assumed by Tukey's one-degree-of-freedom test.
  • 8 (Unbalanced data.) Cells $(1,1)$: $2,4$; $(1,2)$: $6$; $(2,1)$: $3$; $(2,2)$: $7,9$. Fitting additive models, compute the sum of squares for A fitted first, $\text{SS}(A)$, and after B, $\text{SS}(A\mid B)$.
1 Row means $6,9,5$; column means $6,\ 7.33,\ 6.67$; grand mean $6.667$. $\text{SSA}=26$, $\text{SSB}=2.67$, $\text{SST}=32$, $\text{SSE}=3.33$ on $4$ df ($\text{MSE}=0.833$). $F_A=13/0.833=15.6\gt F_{2,4,0.05}=6.94$ ($p=0.013$); $F_B=1.6$ ($p=0.31$). Rows differ, columns do not.
2 $\sum_{ij}(\bar y_{i\cdot}-\bar y)(\bar y_{\cdot j}-\bar y)=\left[\sum_i(\bar y_{i\cdot}-\bar y)\right]\left[\sum_j(\bar y_{\cdot j}-\bar y)\right]=0\times0$.
3 With interaction, the residual array is $\gamma_{ij}+(\varepsilon_{ij}-\bar\varepsilon_{i\cdot}-\bar\varepsilon_{\cdot j}+\bar\varepsilon)$ (the side conditions make $\gamma$ double-centred), and the cross term has mean $0$: $E[\text{SSE}]=\sum\gamma_{ij}^2+(a-1)(b-1)\sigma^2$. The denominator is inflated, so the tests for main effects lose power and are conservative.
4 $\text{SSE}=159-24-45-18=72$. df: $2,3,6,24$ (total $35$). MS: $12,\ 15,\ 3,\ 3$. $F_{AB}=1.0\lt F_{6,24,0.05}=2.51$: no interaction. $F_A=4.0\gt F_{2,24,0.05}=3.40$ ($p=0.032$); $F_B=5.0\gt F_{3,24,0.05}=3.01$ ($p=0.008$).
5 $\hat\gamma_{1j}=-1.667,\ -0.667,\ 2.333$ and $\hat\gamma_{2j}=1.667,\ 0.667,\ -2.333$. The profiles cross between temperatures 2 and 3. Simple effects of catalyst (1 minus 2): $-3,\ -1,\ +5$; the largest is $+5$ at temperature 3.
6 With $a=b=2$, each interaction residual equals $\pm\frac14(\bar y_{11}-\bar y_{12}-\bar y_{21}+\bar y_{22})$; there are $4$ cells and each is counted $n$ times: $\text{SSAB}=4n\cdot\frac1{16}(\cdots)^2$.
7 The interaction model has $ab$ cell means for $ab$ observations: it fits perfectly, leaving $0$ df for error. Tukey assumes $\gamma_{ij}=\lambda\alpha_i\beta_j$, which costs one parameter, and tests $\lambda=0$ with $1$ and $(a-1)(b-1)-1$ df.
8 Cell means $3,6,3,8$. $\text{SS}(A)=8.17$ (row means $4$ and $6.33$, from $3$ observations each), but $\text{SS}(A\mid B)=1.33$; likewise $\text{SS}(B)=28.17$ and $\text{SS}(B\mid A)=21.33$. With unequal cell counts the factors are correlated, and the order of fitting matters.
III
Random and mixed effects.
  • 1 In the one-way random model, show that $\operatorname{Corr}(y_{ij},y_{ij'})=\sigma_a^2/(\sigma_a^2+\sigma^2)$ for $j\ne j'$ and that the covariance matrix of the $n$ observations in a group is $\sigma^2I+\sigma_a^2J$.
  • 2 Prove $E[\text{MSA}]=\sigma^2+n\sigma_a^2$ directly from $\bar y_{i\cdot}=\mu+a_i+\bar\varepsilon_{i\cdot}$.
  • 3 With $k=6$ groups of $n=3$, $\text{MSA}=14$ and $\text{MSE}=4$: test $\sigma_a^2=0$ at $5\%$ and estimate $\sigma_a^2$ and the intraclass correlation.
  • 4 If $k=5$, $n=4$ and $\sigma_a^2=\sigma^2$, find the probability that the ANOVA estimator $\hat\sigma_a^2$ is negative.
  • 5 For $k=5$, $n=4$, find the power of the $5\%$ test of $\sigma_a^2=0$ when $\sigma_a^2/\sigma^2=0.5$.
  • 6 In the two-way random model derive $E[\text{MSB}]=\sigma^2+n\sigma_{ab}^2+an\sigma_b^2$.
  • 7 In a mixed model with A fixed ($a=4$), B random ($b=5$) and $n=2$, give the degrees of freedom of every line and the $F$ ratio (with its df) for each test.
  • 8 Under normality, show that $\operatorname{Var}(\hat\sigma_a^2)=\frac{2}{n^2}\left[\frac{(\sigma^2+n\sigma_a^2)^2}{k-1}+\frac{\sigma^4}{k(n-1)}\right]$ in the balanced one-way random model.
1 $\operatorname{Cov}(y_{ij},y_{ij'})=\operatorname{Var}(a_i)=\sigma_a^2$, $\operatorname{Var}(y_{ij})=\sigma_a^2+\sigma^2$. In matrix form the covariance is $\sigma_a^2$ everywhere plus $\sigma^2$ on the diagonal: $\sigma^2I+\sigma_a^2J$.
2 The $\bar y_{i\cdot}$ are iid with variance $\sigma_a^2+\sigma^2/n$, so $E\sum_i(\bar y_{i\cdot}-\bar y_{\cdot\cdot})^2=(k-1)(\sigma_a^2+\sigma^2/n)$; multiply by $n/(k-1)$.
3 $F=14/4=3.5$ on $(5,12)$; $F_{5,12,0.05}=3.11$: reject ($p=0.035$). $\hat\sigma_a^2=(14-4)/3=3.33$, $\hat\rho=3.33/7.33=0.455$.
4 $\hat\sigma_a^2\lt0\iff F\lt1\iff F/(1+n\theta)\lt1/5$ with $\theta=1$, $n=4$: $P(F_{4,15}\lt0.2)=0.066$.
5 $1+n\theta=3$; power $=P(F_{4,15}\gt3.056/3)=P(F_{4,15}\gt1.019)=0.43$.
6 $\bar y_{\cdot j\cdot}-\bar y_{\cdot\cdot\cdot}=(b_j-\bar b)+(\bar c_{\cdot j}-\bar c_{\cdot\cdot})+(\bar\varepsilon_{\cdot j\cdot}-\bar\varepsilon_{\cdot\cdot\cdot})$, with variances $\sigma_b^2$, $\sigma_{ab}^2/a$, $\sigma^2/(an)$ for the iid terms. So $E[\text{SSB}]=an(b-1)\left(\sigma_b^2+\sigma_{ab}^2/a+\sigma^2/(an)\right)$; divide by $b-1$.
7 df: A $3$, B $4$, AB $12$, error $ab(n-1)=20$, total $39$. A: $\text{MSA}/\text{MSAB}$ on $(3,12)$; B (unrestricted): $\text{MSB}/\text{MSAB}$ on $(4,12)$ (restricted: $\text{MSB}/\text{MSE}$ on $(4,20)$); AB: $\text{MSAB}/\text{MSE}$ on $(12,20)$.
8 MSA and MSE are independent, with $\operatorname{Var}(\text{MSA})=2(\sigma^2+n\sigma_a^2)^2/(k-1)$ and $\operatorname{Var}(\text{MSE})=2\sigma^4/(k(n-1))$ (a $\chi^2_\nu$ variable has variance $2\nu$). Divide the sum by $n^2$.
IV
Analysis of covariance.
  • 1 Derive $\hat\beta=E_{xy}/E_{xx}$ in the ANCOVA model by first minimising over the group intercepts.
  • 2 Show that $\bar y_{i\cdot}$ and $\hat\beta$ are uncorrelated, and deduce the variance of an adjusted mean.
  • 3 Two groups of three: group 1 $x=(2,4,6)$, $y=(5,8,10)$; group 2 $x=(5,7,9)$, $y=(9,12,14)$. Compute the adjusted difference of means and test it at $5\%$. Compare with the unadjusted analysis.
  • 4 Show that if all group means $\bar x_{i\cdot}$ of the covariate are equal, the adjusted means equal the raw means and $\text{SSE}_0-\text{SSE}$ equals the one-way treatment sum of squares of $y$. What does the covariate still achieve?
  • 5 (Parallelism.) For the data of Example 7, fit a separate slope in each group and test at $5\%$ whether the slopes are equal.
  • 6 A trial of a diet uses weight loss as response and "calories eaten during the trial" as a covariate. Explain why this adjustment is misleading.
  • 7 Show that $\text{SSE}=E_{yy}(1-r_w^2)$ with $r_w=E_{xy}/\sqrt{E_{xx}E_{yy}}$, and compute $r_w$ for Example 7.
1 For fixed $\beta$ the intercept of group $i$ is $\bar y_{i\cdot}-\beta\bar x_{i\cdot}$; substituting leaves $\sum[(y_{ij}-\bar y_{i\cdot})-\beta(x_{ij}-\bar x_{i\cdot})]^2=E_{yy}-2\beta E_{xy}+\beta^2E_{xx}$, minimised at $E_{xy}/E_{xx}$.
2 $\hat\beta=\sum_{ij}(x_{ij}-\bar x_{i\cdot})y_{ij}/E_{xx}$, so $\operatorname{Cov}(\bar y_{i\cdot},\hat\beta)=\frac{\sigma^2}{n_iE_{xx}}\sum_j(x_{ij}-\bar x_{i\cdot})=0$. Hence $\operatorname{Var}(\bar y_{i\cdot}-\hat\beta(\bar x_{i\cdot}-\bar x))=\sigma^2\left[\frac1{n_i}+\frac{(\bar x_{i\cdot}-\bar x)^2}{E_{xx}}\right]$.
3 $\bar x=4,7$; $\bar y=7.667,\ 11.667$; $E_{xx}=16$, $E_{xy}=20$, $E_{yy}=25.33$; $\hat\beta=1.25$; $\text{SSE}=0.333$ on $3$ df. Adjusted means $9.54$ and $9.79$: adjusted difference $0.25$ (raw difference $4.0$). $\operatorname{SE}=\sqrt{0.111\left(\frac13+\frac13+\frac{9}{16}\right)}=0.370$, $t=0.68$, $F=0.46\lt F_{1,3,0.05}=10.13$: no treatment effect once $x$ is accounted for. Unadjusted: $F=3.79$ on $(1,4)$, $p=0.12$. The raw difference was almost entirely due to the covariate.
4 If $\bar x_{i\cdot}=\bar x$ for all $i$, then $T_{xx}=E_{xx}$ and $T_{xy}=E_{xy}$ (the between-group parts vanish), so $\text{SSE}_0-\text{SSE}=T_{yy}-E_{yy}$, the one-way SSB of $y$; the adjusted means are $\bar y_{i\cdot}$. The covariate still reduces the error sum of squares from $E_{yy}$ to $E_{yy}-E_{xy}^2/E_{xx}$, so the test gains power.
5 Separate slopes: $16.5/13=1.269$, $32/20=1.6$, $17/14=1.214$; $\text{SSE}_{\text{sep}}=\sum(E_{yy,i}-E_{xy,i}^2/E_{xx,i})=8.96$ on $12-6=6$ df. $F=\frac{(10.47-8.96)/2}{8.96/6}=0.50\lt F_{2,6,0.05}=5.14$: parallel lines are acceptable.
6 Calories eaten are influenced by the diet itself. Adjusting compares people "at equal calorie intake", which removes the very mechanism through which the diet works; the adjusted effect is not the effect of assigning the diet.
7 $E_{yy}-E_{xy}^2/E_{xx}=E_{yy}\left(1-\frac{E_{xy}^2}{E_{xx}E_{yy}}\right)$. For Example 7: $r_w=65.5/\sqrt{47\times101.75}=0.947$, and $101.75(1-0.897)=10.47$. ✓
🎯 Exam corner CSIR Part BCSIR Part CGATE ST

E1. CSIR Part B GATE ST In a one-way ANOVA with $4$ groups of $6$ observations each, the between-groups sum of squares is $36$ and the total sum of squares is $96$. The value of the $F$ statistic is (A) $1.8$; (B) $0.6$; (C) $4.0$; (D) $2.4$.

Idea $F=\frac{\text{SSB}/(k-1)}{\text{SSW}/(N-k)}$ with $\text{SSW}=\text{SST}-\text{SSB}$.
Compute $k-1=3$, $N-k=20$, $\text{SSW}=60$: $F=\frac{12}{3}=4.0$ (above $F_{3,20,0.05}=3.10$). Option (B) is $\text{SSB}/\text{SSW}$ without degrees of freedom; (A) uses $N-1$ in the denominator.
Answer (C).

E2. CSIR Part B In a two-way fixed effects layout with factor A at $3$ levels, factor B at $4$ levels and $2$ observations per cell, the $F$ statistic for testing the interaction has degrees of freedom (A) $(6,23)$; (B) $(6,12)$; (C) $(5,12)$; (D) $(6,18)$.

Idea Theorem 6.15: interaction $(a-1)(b-1)$, error $ab(n-1)$.
Compute $(a-1)(b-1)=6$ and $ab(n-1)=12$; check $2+3+6+12=23=N-1$.
Answer (B).

E3. CSIR Part B GATE ST In a balanced one-way random effects model with $5$ groups of $4$ observations, $\text{MSA}=26$ and $\text{MSE}=6$. The ANOVA estimate of the intraclass correlation $\sigma_a^2/(\sigma_a^2+\sigma^2)$ is closest to (A) $0.77$; (B) $0.23$; (C) $0.83$; (D) $0.45$.

Idea $E[\text{MSA}]=\sigma^2+n\sigma_a^2$, $E[\text{MSE}]=\sigma^2$ (Theorem 6.16).
Compute $\hat\sigma_a^2=(26-6)/4=5$, $\hat\sigma^2=6$, $\hat\rho=5/11=0.4545$. The distractors: $20/26$ forgets to divide by $n$; $6/26$ is $\text{MSE}/\text{MSA}$; $5/6$ is $\hat\sigma_a^2/\hat\sigma^2$.
Answer (D).

E4. CSIR Part C In the normal one-way fixed effects model with $k\ge2$ groups, which of the following are true? (A) MSW is unbiased for $\sigma^2$ whether or not the group means are equal. (B) $E[\text{MSB}]\ge E[\text{MSW}]$, with equality if and only if all group means are equal. (C) SSB and SSW are independent only when the group means are equal. (D) For $k=2$, the $F$ statistic is the square of the pooled two-sample $t$ statistic.

(A) True $E[\text{SSW}]=\sum(n_i-1)\sigma^2$ in every case: within-group deviations do not involve the $\mu_i$.
(B) True $E[\text{MSB}]-E[\text{MSW}]=\sum n_i(\mu_i-\bar\mu)^2/(k-1)\ge0$, zero iff all $\mu_i=\bar\mu$.
(C) False SSW is independent of $(\bar y_1,\dots,\bar y_k)$ for all values of the means (§4.1, applied group by group), and SSB is a function of the $\bar y_i$. Only the central $\chi^2$ law of SSB needs $H_0$.
(D) True Exercise I.4.
Answer (A), (B) and (D).

E5. CSIR Part C Consider a balanced two-way mixed model with A fixed at $a$ levels, B random at $b$ levels, a random interaction (unrestricted model) and $n\ge2$ observations per cell. Which of the following are true? (A) The hypothesis of no A effects is tested by MSA/MSAB on $(a-1,(a-1)(b-1))$ degrees of freedom. (B) The hypothesis $\sigma_{ab}^2=0$ is tested by MSAB/MSE. (C) With $n=1$ observation per cell, the hypothesis of no A effects could not be tested. (D) $E[\text{MSE}]=\sigma^2$.

(A) True $E[\text{MSA}]=\sigma^2+n\sigma_{ab}^2+bnQ_A$ and $E[\text{MSAB}]=\sigma^2+n\sigma_{ab}^2$ (Theorem 6.17).
(B) True $E[\text{MSAB}]=\sigma^2+n\sigma_{ab}^2$ reduces to $E[\text{MSE}]=\sigma^2$ exactly when $\sigma_{ab}^2=0$.
(C) False The test of A uses MSAB, which exists with $n=1$ (it has $(a-1)(b-1)$ df). What cannot be tested with $n=1$ is the interaction itself, since there is no MSE.
(D) True Within-cell deviations involve only the errors.
Answer (A), (B) and (D).

E6. CSIR Part C In a one-way analysis of covariance with $k$ treatments, $N$ experimental units and one covariate $x$ (common slope), which of the following are true? (A) The error sum of squares has $N-k-1$ degrees of freedom. (B) The adjusted treatment means are $\bar y_{i\cdot}-\hat\beta(\bar x_{i\cdot}-\bar x_{\cdot\cdot})$, where $\hat\beta$ is the pooled within-treatment regression coefficient. (C) If all treatments have the same covariate mean, the adjusted and unadjusted treatment means coincide. (D) The $F$ statistic for treatments in the ANCOVA is always at least as large as the one-way ANOVA $F$ statistic computed from $y$ alone.

(A) True The model has $k$ treatment means and one slope: rank $k+1$.
(B) True Theorem 6.18(1), with $\hat\beta=E_{xy}/E_{xx}$.
(C) True The adjustment term $\hat\beta(\bar x_{i\cdot}-\bar x_{\cdot\cdot})$ is zero for every $i$.
(D) False Counterexample: group 1 $x=(1,2,3)$, $y=(1.1,2.0,2.9)$; group 2 $x=(5,6,7)$, $y=(5.0,6.1,6.9)$. One-way ANOVA of $y$ gives $F=27.9$ on $(1,4)$; ANCOVA gives $F=3.31$ on $(1,3)$. The groups differ in $y$ mainly because they differ in $x$ (Exercise IV.3 is another instance).
Answer (A), (B) and (C).
🧑‍🏫 For the teacher

Suggested plan — 3 lectures × 60 min.

  1. One-way ANOVA. Start with the multiplicity problem of many $t$-tests. Definition 6.6 and Theorem 6.11 with full proofs of (1)–(2) and the projection proof of (3); Example 1 with the correction-factor shortcut. Project Interactive 1, first with $\delta=0$ (the $F$ values hover around $1$, about $5\%$ rejected), then with $\delta=0.6$. Contrasts (Theorem 6.12), the three multiple-comparison methods (Theorem 6.13) and Example 2.
  2. Two-way layouts. Theorem 6.14 and Example 3, stressing the gain from blocking (the same fertiliser data are significant with blocks and invisible without). Definition 6.8(2), Theorem 6.15, Example 4, then Interactive 2: raise the interaction slider until the profiles cross and watch $F$ for A fall. The strategy box on how to read a two-way table.
  3. Random effects and ANCOVA. Definition 6.9, Theorem 6.16 with proof, Example 5 (point out how wide the interval for $\theta$ is). Theorem 6.17 as a table with the rule for denominators; Example 6 as a class discussion. ANCOVA: Theorem 6.18 via the two-stage minimisation, Example 7. Finish with E5 and E6.

Discussion questions. (1) A company tests four suppliers chosen because they are the only ones available. Should "supplier" be fixed or random? What if they were drawn from a list of fifty? (2) Why can a factor have a large effect at every level of another factor and still have a main-effect $F$ near zero? (3) In Example 7, would you report the raw or the adjusted difference between methods 2 and 1, and why?

Misconceptions to watch for. Using MSE as the denominator for every test in random and mixed models; interpreting main effects in the presence of strong interaction; believing that a significant $F$ identifies the differing pair; counting error degrees of freedom as $N-k$ in ANCOVA; adjusting for covariates measured after treatment; reporting a negative variance component without comment.

Suggested homework. I(2, 4, 5, 7), II(1, 4, 6, 8), III(3, 4, 6, 7), IV(3, 4, 5); Exam corner E1, E5, E6.

🔬 Research–Teaching NexusResearch-tutored · optional, about 60 minutes

Research question. How robust is the one-way ANOVA $F$-test? Estimate by simulation its actual type I error rate when the errors are skewed or heavy-tailed, and when the group variances are unequal, for balanced and unbalanced designs. When should Welch's ANOVA or a rank test (Kruskal–Wallis) be used instead?

Why it matters. ANOVA is applied every day in agriculture, medicine, psychology and engineering, often to data that are not normal and have unequal spreads. The exact theory of Theorem 6.11 assumes normality and equal variances; whether a published "significant difference" can be trusted depends on how badly the test behaves when these fail. Simulation studies of this kind are a standard part of methodological research and of the guidelines statistical societies give to applied scientists.

What to do.

  1. Write a simulation that draws $k=3$ groups with equal means from a chosen population, computes the $F$ statistic and records whether $F\gt F_{k-1,N-k,0.05}$. Use $10\,000$ replicates per setting.
  2. Error laws: normal, exponential (skewed), $t_3$ (heavy-tailed). Designs: $n_i=(10,10,10)$ and $(5,10,20)$. Variances: equal, and standard deviations in the ratio $1:2:4$ (assign the large variance once to the small group, once to the large group).
  3. Tabulate the estimated sizes with their Monte Carlo standard errors ($\approx\sqrt{0.05\times0.95/10\,000}=0.002$).
  4. Repeat with Welch's test and the Kruskal–Wallis test, and write recommendations.

Tools: R (aov, oneway.test, kruskal.test) or Python (scipy.stats.f_oneway, scipy.stats.kruskal); Interactive 1 shows the null behaviour for normal data.

Hand in: A table of estimated sizes for each test and setting, a short explanation of the pattern, and a one-paragraph recommendation for practitioners.

Going further. Read about Box's (1954) approximations for the effect of unequal variances, and about permutation tests, which give exact size under exchangeability without assuming normality.

Instructor note: with equal variances the $F$-test keeps its size (about $0.04$–$0.05$) under all three error laws, balanced or not; with equal $n_i=10$ and standard deviations $1:2:4$ the size rises only to about $0.06$–$0.09$. In the unbalanced design, putting the largest variance in the smallest group makes the size rise to roughly $0.23$–$0.27$, and putting it in the largest group makes the test conservative (size about $0.01$–$0.03$). Welch's test stays close to $0.05$ throughout; Kruskal–Wallis tests a different hypothesis (equality of distributions) and is also affected by unequal spreads.

🧪 Self-check quiz

0/5 correct

Q1. A one-way ANOVA compares $k$ groups with $N$ observations in total. The $F$ statistic has degrees of freedom

Q2. In the one-way fixed effects model, $E[\text{MSB}]$ equals

Q3. In a balanced two-way random effects model with interaction, the main effect of A is tested by

Q4. A balanced one-way random effects analysis with $n=5$ observations per group gives $\text{MSA}=19$ and $\text{MSE}=4$. The ANOVA estimate of $\sigma_a^2$ is

Q5. In a one-way analysis of covariance with $k=4$ treatments, $N=24$ units and one covariate, the error degrees of freedom are