6.2 Analysis of Variance and Covariance
Comparing several means at once: the one-way analysis of variance with its partition of sums of squares and $F$-test, contrasts and multiple comparisons, two-way layouts with and without interaction, fixed, random and mixed effects with expected mean squares and variance components, and the analysis of covariance with one covariate.
An agronomist compares four varieties of wheat, a pharmacologist five doses of a drug, an engineer three suppliers of a component. Each wants to know whether the population means differ, and if so, which ones and by how much. Testing every pair with a two-sample $t$-test is tempting but wrong: with $6$ pairs, each tested at $5\%$, the chance of at least one false "discovery" is bounded only by $6\times0.05=0.30$ (Bonferroni's inequality), and would be $1-0.95^6=0.26$ if the tests were independent; moreover the separate tests waste information, because each uses only two groups to estimate the error variance. R. A. Fisher's answer, developed for the field trials at Rothamsted in the 1920s, was the analysis of variance (ANOVA): split the total variability of the data into a part explained by the groups and a part due to error, and compare the two with one $F$-test.
In the language of §6.1, every ANOVA is a linear model whose design matrix is made of indicator columns, the $F$-test is the test of a linear hypothesis, and the partitions of sums of squares are instances of Cochran's theorem. This section works out the standard layouts in detail: the one-way classification with contrasts and multiple comparisons; two-way classifications with one observation per cell and with replication, where interaction appears; random and mixed effects models, where the levels of a factor are a sample from a population and the questions concern variance components; and the analysis of covariance (ANCOVA), which combines ANOVA with regression on a concomitant variable. CSIR NET questions ask for degrees of freedom, $F$ values and expected mean squares (Part B) and for the correct denominator of an $F$-test, the estimability of effects and the interpretation of interaction (Part C). The designs of Chapter 8 (CRD, RBD, Latin squares, factorials) use exactly these tools. Upper points are written $F_{\nu_1,\nu_2,\alpha}$ as in §4.3.
One-Way Analysis of Variance
Observations $y_{ij}$, $i=1,\dots,k$ (groups or treatments), $j=1,\dots,n_i$, follow $$y_{ij}=\mu_i+\varepsilon_{ij}=\mu+\alpha_i+\varepsilon_{ij},\qquad\varepsilon_{ij}\ \text{iid}\ N(0,\sigma^2),$$ with $N=\sum n_i$ observations in all. The group means $\mu_i$ are unknown constants (fixed effects); writing $\mu_i=\mu+\alpha_i$ with the side condition $\sum n_i\alpha_i=0$ makes $\mu=\bar\mu=\sum n_i\mu_i/N$ the weighted average mean and $\alpha_i=\mu_i-\bar\mu$ the effect of treatment $i$. Write $\bar y_i=\frac1{n_i}\sum_jy_{ij}$, $\bar y=\frac1N\sum_{ij}y_{ij}$, and $$\text{SST}=\sum_{ij}(y_{ij}-\bar y)^2,\quad\text{SSB}=\sum_in_i(\bar y_i-\bar y)^2,\quad\text{SSW}=\sum_{ij}(y_{ij}-\bar y_i)^2$$ for the total, between-groups (treatment) and within-groups (error) sums of squares.
By §6.1 the model has rank $k$; the estimable functions are the $\mu_i$ and their combinations, in particular every contrast $\sum c_i\alpha_i$ with $\sum c_i=0$; the BLUE of $\mu_i$ is $\bar y_i$; the fitted values are the group means; $\text{RSS}=\text{SSW}$ on $N-k$ degrees of freedom.
- (Partition.) $\text{SST}=\text{SSB}+\text{SSW}$.
- (Expected mean squares, no normality needed.) With $\text{MSB}=\text{SSB}/(k-1)$ and $\text{MSW}=\text{SSW}/(N-k)$, $$E[\text{MSW}]=\sigma^2,\qquad E[\text{MSB}]=\sigma^2+\frac{\sum_in_i(\mu_i-\bar\mu)^2}{k-1}.$$
- (Distributions.) Under normality, $\text{SSW}/\sigma^2\sim\chi^2_{N-k}$, independent of $(\bar y_1,\dots,\bar y_k)$ and hence of SSB; $\text{SSB}/\sigma^2$ is noncentral $\chi^2_{k-1}(\delta)$ with $\delta=\sum n_i(\mu_i-\bar\mu)^2/\sigma^2$.
- (The $F$-test.) $H_0:\mu_1=\cdots=\mu_k$ is rejected at level $\alpha$ when $$F=\frac{\text{MSB}}{\text{MSW}}\gt F_{k-1,N-k,\alpha};$$ under $H_0$, $F\sim F_{k-1,N-k}$. This is the $F$-test of a general linear hypothesis (§6.1) and the likelihood ratio test.
Proof. (1) $y_{ij}-\bar y=(y_{ij}-\bar y_i)+(\bar y_i-\bar y)$. Squaring and summing, the cross term is $2\sum_i(\bar y_i-\bar y)\sum_j(y_{ij}-\bar y_i)=0$, because the inner sum is zero for each $i$.
(2) $\sum_j(y_{ij}-\bar y_i)^2=(n_i-1)S_i^2$, and $E[S_i^2]=\sigma^2$ (§4.1), so $E[\text{SSW}]=(N-k)\sigma^2$. For SSB use $\text{SSB}=\sum n_i\bar y_i^2-N\bar y^2$, with $E[\bar y_i^2]=\mu_i^2+\sigma^2/n_i$ and $E[\bar y^2]=\bar\mu^2+\sigma^2/N$: $$E[\text{SSB}]=\sum n_i\mu_i^2+k\sigma^2-N\bar\mu^2-\sigma^2=(k-1)\sigma^2+\sum n_i(\mu_i-\bar\mu)^2.$$
(3) For each group, $S_i^2$ is independent of $\bar y_i$ and $(n_i-1)S_i^2/\sigma^2\sim\chi^2_{n_i-1}$ (the fundamental theorem for normal samples, §4.1); the groups are independent, so SSW$/\sigma^2$ is a sum of independent $\chi^2$ variables with $\sum(n_i-1)=N-k$ degrees of freedom, independent of all the $\bar y_i$. For SSB put $z_i=\sqrt{n_i}\,\bar y_i\sim N(\sqrt{n_i}\mu_i,\sigma^2)$, independent, and $s=(\sqrt{n_1},\dots,\sqrt{n_k})^{\mathsf T}/\sqrt N$, a unit vector. Then $\sum n_i\bar y_i^2=\|z\|^2$ and $N\bar y^2=(s^{\mathsf T}z)^2$, so $\text{SSB}=\|z\|^2-(s^{\mathsf T}z)^2=\|Pz\|^2$, where $P=I-ss^{\mathsf T}$ is the projection onto $s^\perp$, of rank $k-1$. By the rotation argument used for the normal linear model in §6.1, $\|Pz\|^2/\sigma^2$ is $\chi^2_{k-1}$ with noncentrality $\|P\,E z\|^2/\sigma^2=\sum n_i(\mu_i-\bar\mu)^2/\sigma^2$.
(4) Under $H_0$, $\delta=0$ and $F$ is a ratio of independent $\chi^2$ variables divided by their degrees of freedom. In the language of the general $F$-test of §6.1: the full model has $\text{RSS}=\text{SSW}$ with $n-r=N-k$; the reduced model $E[y_{ij}]=\mu$ has $\text{RSS}_0=\text{SST}$; the hypothesis consists of $q=k-1$ independent contrasts; and $\text{RSS}_0-\text{RSS}=\text{SSB}$. ∎
The results are set out in an ANOVA table. For hand computation use the correction factor $\text{CF}=G^2/N$, where $G=\sum_{ij}y_{ij}$ is the grand total, and the group totals $T_i$: $$\text{SST}=\sum_{ij}y_{ij}^2-\text{CF},\qquad\text{SSB}=\sum_i\frac{T_i^2}{n_i}-\text{CF}.$$
| Source | df | SS | MS | $E[\text{MS}]$ | $F$ |
|---|---|---|---|---|---|
| Between groups | $k-1$ | SSB | $\frac{\text{SSB}}{k-1}$ | $\sigma^2+\frac{\sum n_i\alpha_i^2}{k-1}$ | $\frac{\text{MSB}}{\text{MSW}}$ |
| Within groups | $N-k$ | SSW | $\frac{\text{SSW}}{N-k}$ | $\sigma^2$ | |
| Total | $N-1$ | SST |
Contrasts and multiple comparisons
A contrast among the group means is $\psi=\sum_ic_i\mu_i$ with $\sum_ic_i=0$ (not all $c_i$ zero). Two contrasts with coefficient vectors $c$ and $d$ are orthogonal if $\sum_ic_id_i/n_i=0$ (for equal group sizes: $\sum c_id_i=0$). Pairwise differences $\mu_i-\mu_j$ are contrasts.
- The BLUE of $\psi=\sum c_i\mu_i$ is $\hat\psi=\sum c_i\bar y_i$, with $\operatorname{Var}(\hat\psi)=\sigma^2\sum_ic_i^2/n_i$. Under normality, $$\frac{\hat\psi-\psi}{\sqrt{\text{MSW}\sum c_i^2/n_i}}\sim t_{N-k}.$$
- The single-degree-of-freedom sum of squares of a contrast is $\text{SS}(\hat\psi)=\hat\psi^2\big/\sum_i(c_i^2/n_i)$, and $\text{SS}(\hat\psi)/\text{MSW}=t^2$ for $H_0:\psi=0$.
- If $\psi_1,\dots,\psi_{k-1}$ are mutually orthogonal contrasts, then $$\text{SSB}=\text{SS}(\hat\psi_1)+\cdots+\text{SS}(\hat\psi_{k-1}),$$ and under normality these sums of squares are independent.
Proof. (1) $\hat\psi$ is linear and unbiased, and it is a function of the BLUEs $\bar y_i$ of the estimable $\mu_i$, so it is the BLUE (the Gauss–Markov theorem, §6.1). The variance follows from the independence of the $\bar y_i$, and the $t$-law from the confidence intervals of the normal linear model (§6.1).
(2), (3) Use $z_i=\sqrt{n_i}\bar y_i$ and the unit vector $s$ from the proof of Theorem 6.11. With $w_i=c_i/\sqrt{n_i}$ we have $\hat\psi=w^{\mathsf T}z$, $\sum c_i^2/n_i=\|w\|^2$ and $\text{SS}(\hat\psi)=(w^{\mathsf T}z)^2/\|w\|^2$, the squared length of the projection of $z$ on the direction of $w$. The condition $\sum c_i=0$ says $w^{\mathsf T}s\sqrt N=\sum c_i=0$, so $w\perp s$; orthogonality of two contrasts says $w^{\mathsf T}w'=\sum c_id_i/n_i=0$. Thus $k-1$ mutually orthogonal contrasts give an orthogonal basis $w_1,\dots,w_{k-1}$ of $s^\perp$, and $\text{SSB}=\|Pz\|^2=\sum_l(w_l^{\mathsf T}z)^2/\|w_l\|^2$. Coordinates of the normal vector $z$ (covariance $\sigma^2I$) along orthogonal directions are independent. ∎
Testing many contrasts at level $\alpha$ each inflates the chance of some false rejection. Three classical remedies give simultaneous confidence statements.
In the normal one-way model, let $\hat\sigma^2=\text{MSW}$ on $\nu=N-k$ degrees of freedom.
- (Bonferroni.) For $m$ pre-specified contrasts, the intervals $\hat\psi_l\pm t_{\nu,\alpha/(2m)}\,\hat\sigma\sqrt{\sum c_{li}^2/n_i}$ hold simultaneously with probability at least $1-\alpha$.
- (Scheffé.) With probability exactly $1-\alpha$, simultaneously for all contrasts $c$, $$\left|\hat\psi_c-\psi_c\right|\le\sqrt{(k-1)F_{k-1,\nu,\alpha}}\;\hat\sigma\sqrt{\textstyle\sum c_i^2/n_i}.$$ Moreover, the $F$-test of Theorem 6.11 rejects $H_0$ if and only if some contrast $\hat\psi_c$ falls outside its Scheffé interval around $0$.
- (Tukey.) If $n_1=\cdots=n_k=n$, then with probability $1-\alpha$, simultaneously for all pairs, $|(\bar y_i-\bar y_j)-(\mu_i-\mu_j)|\le q_{k,\nu,\alpha}\,\hat\sigma/\sqrt n$, where $q_{k,\nu,\alpha}$ is the upper point of the studentized range $\left(\max_iZ_i-\min_iZ_i\right)/\sqrt{U/\nu}$ ($Z_i$ iid $N(0,1)$, $U\sim\chi^2_\nu$ independent). For unequal $n_i$ the Tukey–Kramer intervals replace $\hat\sigma/\sqrt n$ by $\hat\sigma\sqrt{\frac12\left(\frac1{n_i}+\frac1{n_j}\right)}$ (conservative).
Proof. (1) Boole's inequality: the probability that at least one of $m$ intervals fails is at most $m\cdot\alpha/m$.
(2) With $z_i=\sqrt{n_i}(\bar y_i-\mu_i)$, a vector of iid $N(0,\sigma^2)$ coordinates, and $w=(c_i/\sqrt{n_i})$, we have $\hat\psi_c-\psi_c=w^{\mathsf T}z$ and $w\perp s$. By the Cauchy–Schwarz inequality, $$\sup_{w\perp s,\ w\ne0}\frac{(w^{\mathsf T}z)^2}{\|w\|^2}=\|Pz\|^2,$$ attained at $w=Pz$. Now $\|Pz\|^2/\sigma^2\sim\chi^2_{k-1}$ independently of $\hat\sigma^2$, so $\|Pz\|^2/((k-1)\hat\sigma^2)\sim F_{k-1,\nu}$, and the event "all intervals hold" is exactly $\{\|Pz\|^2\le(k-1)\hat\sigma^2F_{k-1,\nu,\alpha}\}$, of probability $1-\alpha$. Under $H_0$ (all $\mu_i$ equal) $Pz$ computed with $\mu_i=\bar\mu$ gives $\|Pz\|^2=\text{SSB}$, so the supremum of $\text{SS}(\hat\psi_c)$ over contrasts is SSB, and "some contrast exceeds the Scheffé bound" is exactly "$\text{SSB}/((k-1)\text{MSW})\gt F_{k-1,\nu,\alpha}$".
(3) With equal $n$, $(\bar y_i-\mu_i)\sqrt n/\sigma$ are iid $N(0,1)$, independent of $\hat\sigma$, and $\max_{i,j}|(\bar y_i-\mu_i)-(\bar y_j-\mu_j)|\sqrt n/\hat\sigma$ is the studentized range. ∎
For pairwise comparisons Tukey's intervals are the shortest of the three; Bonferroni is good for a few planned contrasts; Scheffé is the price of complete freedom to look at any contrast, including ones suggested by the data. Fisher's least significant difference (unadjusted $t$-tests, used only after a significant $F$) controls the familywise error rate only when $k=3$.
Two-Way Classifications and Interaction
Often the units can be grouped by two factors: varieties and fields, machines and operators, doses and age groups. Including the second factor removes its variability from the error (the idea of blocking, §8.3) and lets us ask whether the factors act additively.
Factor A has levels $i=1,\dots,a$ and factor B levels $j=1,\dots,b$; errors are iid $N(0,\sigma^2)$.
- The additive model (one observation per cell): $y_{ij}=\mu+\alpha_i+\beta_j+\varepsilon_{ij}$, with $\sum_i\alpha_i=\sum_j\beta_j=0$.
- The model with interaction ($n\ge2$ observations per cell): $y_{ijk}=\mu+\alpha_i+\beta_j+\gamma_{ij}+\varepsilon_{ijk}$, $k=1,\dots,n$, with $\sum_i\alpha_i=\sum_j\beta_j=0$ and $\sum_i\gamma_{ij}=0$ for each $j$, $\sum_j\gamma_{ij}=0$ for each $i$.
With cell means $\mu_{ij}=E[y_{ijk}]$, the interaction is $\gamma_{ij}=\mu_{ij}-\bar\mu_{i\cdot}-\bar\mu_{\cdot j}+\bar\mu_{\cdot\cdot}$. There is no interaction iff $\mu_{ij}-\mu_{i'j}$ does not depend on $j$, i.e. the profiles $j\mapsto\mu_{ij}$ of the levels of A are parallel.
In the additive model with $a\times b$ observations, write $\bar y_{i\cdot},\bar y_{\cdot j},\bar y_{\cdot\cdot}$ for row, column and grand means, and $$\text{SSA}=b\sum_i(\bar y_{i\cdot}-\bar y_{\cdot\cdot})^2,\quad\text{SSB}=a\sum_j(\bar y_{\cdot j}-\bar y_{\cdot\cdot})^2,$$ $$\text{SSE}=\sum_{ij}(y_{ij}-\bar y_{i\cdot}-\bar y_{\cdot j}+\bar y_{\cdot\cdot})^2.$$ Then $\text{SST}=\text{SSA}+\text{SSB}+\text{SSE}$ with degrees of freedom $ab-1=(a-1)+(b-1)+(a-1)(b-1)$; $$E[\text{MSA}]=\sigma^2+\frac{b\sum\alpha_i^2}{a-1},\quad E[\text{MSB}]=\sigma^2+\frac{a\sum\beta_j^2}{b-1},\quad E[\text{MSE}]=\sigma^2;$$ the three sums of squares are independent, and $H_0:\alpha_i\equiv0$ is rejected when $\text{MSA}/\text{MSE}\gt F_{a-1,(a-1)(b-1),\alpha}$ (similarly for B).
Proof. Write $r_i=\bar y_{i\cdot}-\bar y_{\cdot\cdot}$, $c_j=\bar y_{\cdot j}-\bar y_{\cdot\cdot}$ and $$e_{ij}=y_{ij}-\bar y_{i\cdot}-\bar y_{\cdot j}+\bar y_{\cdot\cdot},$$ so that $y_{ij}-\bar y_{\cdot\cdot}=r_i+c_j+e_{ij}$. The three arrays are orthogonal as vectors in $\mathbb R^{ab}$. For instance, $\sum_je_{ij}=b\bar y_{i\cdot}-b\bar y_{i\cdot}-b\bar y_{\cdot\cdot}+b\bar y_{\cdot\cdot}=0$ for each $i$, so $\sum_{ij}r_ie_{ij}=\sum_ir_i\sum_je_{ij}=0$; similarly $\sum_{ij}c_je_{ij}=0$; and the row and column arrays are orthogonal because $\sum_i(\bar y_{i\cdot}-\bar y_{\cdot\cdot})=0$. Squaring gives the partition. The arrays lie in subspaces of dimensions $a-1$, $b-1$ and $(a-1)(b-1)$ (the row array is determined by $a$ numbers summing to zero, and so on), which together with the constant vectors fill $\mathbb R^{ab}$; Cochran's theorem (§6.1) gives independent $\chi^2$ laws. For expectations, substitute the model: $\bar y_{i\cdot}-\bar y_{\cdot\cdot}=\alpha_i+\bar\varepsilon_{i\cdot}-\bar\varepsilon_{\cdot\cdot}$, so $E[\text{SSA}]=b\sum\alpha_i^2+b(a-1)\sigma^2/b$; and the third array equals $\varepsilon_{ij}-\bar\varepsilon_{i\cdot}-\bar\varepsilon_{\cdot j}+\bar\varepsilon_{\cdot\cdot}$ (the parameters cancel), whose expected squared length is $(a-1)(b-1)\sigma^2$ by the unbiased estimation of $\sigma^2$ in §6.1 (it is the residual of a model of rank $a+b-1$). ∎
With one observation per cell an interaction cannot be separated from error: if $\gamma_{ij}\ne0$, then $E[\text{MSE}]=\sigma^2+\sum\gamma_{ij}^2/((a-1)(b-1))$, which inflates the denominator and makes the $F$-tests conservative. Tukey's one-degree-of-freedom test for non-additivity tests the special form $\gamma_{ij}=\lambda\alpha_i\beta_j$. The layout is the randomised block design of §8.3, with A as treatments and B as blocks.
In the model with interaction (Definition 6.8(2)), with cell means $\bar y_{ij\cdot}$, $$\begin{aligned}\text{SSA}&=bn\textstyle\sum_i(\bar y_{i\cdot\cdot}-\bar y_{\cdot\cdot\cdot})^2,\qquad \text{SSB}=an\textstyle\sum_j(\bar y_{\cdot j\cdot}-\bar y_{\cdot\cdot\cdot})^2,\\ \text{SSAB}&=n\textstyle\sum_{ij}(\bar y_{ij\cdot}-\bar y_{i\cdot\cdot}-\bar y_{\cdot j\cdot}+\bar y_{\cdot\cdot\cdot})^2,\qquad\text{SSE}=\textstyle\sum_{ijk}(y_{ijk}-\bar y_{ij\cdot})^2,\end{aligned}$$ and $\text{SST}=\text{SSA}+\text{SSB}+\text{SSAB}+\text{SSE}$ with degrees of freedom $$abn-1=(a-1)+(b-1)+(a-1)(b-1)+ab(n-1).$$ The four sums of squares are independent, and $$\begin{aligned}E[\text{MSA}]&=\sigma^2+\tfrac{bn\sum\alpha_i^2}{a-1},\qquad E[\text{MSB}]=\sigma^2+\tfrac{an\sum\beta_j^2}{b-1},\\ E[\text{MSAB}]&=\sigma^2+\tfrac{n\sum\gamma_{ij}^2}{(a-1)(b-1)},\qquad E[\text{MSE}]=\sigma^2.\end{aligned}$$ Each effect is tested by its mean square divided by MSE, with $ab(n-1)$ denominator degrees of freedom.
Proof (sketch). Write $y_{ijk}-\bar y_{\cdot\cdot\cdot}$ as the sum of the row, column and interaction arrays of the cell means (Theorem 6.14 applied to the $a\times b$ table of cell means, each entry repeated $n$ times) and the within-cell deviations $y_{ijk}-\bar y_{ij\cdot}$. The within-cell array is orthogonal to anything constant within cells, which gives the four-way partition; the dimensions are as stated and add up to $abn-1$, so Cochran's theorem applies. For the expectations, substitute the model: the row array has entries $\alpha_i+(\bar\varepsilon_{i\cdot\cdot}-\bar\varepsilon_{\cdot\cdot\cdot})$, with the interaction terms cancelling because of the side conditions, and similarly for the others; $\text{SSE}/\sigma^2\sim\chi^2_{ab(n-1)}$ because each cell is a normal sample of size $n$. ∎
- Check the degrees of freedom first. They must add to $N-1$; with one observation per cell there is no separate error line, and the "residual" is the interaction line.
- Test the interaction first. If it is significant, the main effects are averages over the levels of the other factor and may be misleading; compare the levels of A separately at each level of B (simple effects), and draw the interaction plot.
- If the interaction is negligible, interpret the main effects and compare their levels with the methods of Theorem 6.13, using MSE and the appropriate number of observations per level ($bn$ for a level of A).
- Unbalanced data (unequal cell counts) break the orthogonality: the sum of squares for A depends on whether B is already in the model (Exercise II.8). Then use the general $F$-test of §6.1 for each hypothesis.
Fixed, Random and Mixed Effects
In Example 1 the three feeds are the only feeds of interest: their effects are fixed constants, and conclusions concern these feeds. Suppose instead that five batches of raw material are drawn at random from a supplier's production, and several specimens are tested from each. Nobody cares about these particular batches; the question is how much batches vary in general. The batch effects are then random variables, and the parameter of interest is their variance.
The one-way random effects model is $$y_{ij}=\mu+a_i+\varepsilon_{ij},\qquad i=1,\dots,k,\ j=1,\dots,n,$$ where $a_i\sim N(0,\sigma_a^2)$ and $\varepsilon_{ij}\sim N(0,\sigma^2)$, all mutually independent. The parameters are $\mu$ and the variance components $\sigma_a^2$, $\sigma^2$; the intraclass correlation is $\rho=\sigma_a^2/(\sigma_a^2+\sigma^2)$. In a two-way layout, A and B may both be fixed, both random (with interaction $c_{ij}\sim N(0,\sigma_{ab}^2)$), or one of each: a mixed model. A factor is treated as random when its levels are a random sample from a larger population of levels to which the conclusions should apply.
With $\text{SSA}=n\sum_i(\bar y_{i\cdot}-\bar y_{\cdot\cdot})^2$ on $k-1$ df and $\text{SSE}=\sum_{ij}(y_{ij}-\bar y_{i\cdot})^2$ on $k(n-1)$ df:
- $\operatorname{Var}(y_{ij})=\sigma_a^2+\sigma^2$, $\operatorname{Corr}(y_{ij},y_{ij'})=\rho$ for $j\ne j'$, and observations in different groups are independent;
- $\text{SSE}/\sigma^2\sim\chi^2_{k(n-1)}$ and $\text{SSA}/(\sigma^2+n\sigma_a^2)\sim\chi^2_{k-1}$, independently; hence $E[\text{MSE}]=\sigma^2$ and $E[\text{MSA}]=\sigma^2+n\sigma_a^2$;
- $F=\text{MSA}/\text{MSE}$ satisfies $F/(1+n\theta)\sim F_{k-1,k(n-1)}$ with $\theta=\sigma_a^2/\sigma^2$. The test of $H_0:\sigma_a^2=0$ rejects when $F\gt F_{k-1,k(n-1),\alpha}$, and its power is $P\left(F_{k-1,k(n-1)}\gt F_{k-1,k(n-1),\alpha}/(1+n\theta)\right)$;
- the ANOVA estimators $\hat\sigma^2=\text{MSE}$ and $\hat\sigma_a^2=(\text{MSA}-\text{MSE})/n$ are unbiased, and $\hat\sigma_a^2\lt0$ with positive probability;
- with $F_U=F_{k-1,k(n-1),\alpha/2}$ and $F_L=F_{k-1,k(n-1),1-\alpha/2}$, $$\left[\frac1n\left(\frac{F}{F_U}-1\right),\ \frac1n\left(\frac{F}{F_L}-1\right)\right]$$ is a $100(1-\alpha)\%$ confidence interval for $\theta=\sigma_a^2/\sigma^2$, and $\theta\mapsto\theta/(1+\theta)$ turns it into one for $\rho$.
Proof. (1) $\operatorname{Cov}(y_{ij},y_{ij'})=\operatorname{Var}(a_i)=\sigma_a^2$ for $j\ne j'$, while different groups share no random term.
(2) Within group $i$, $y_{ij}-\bar y_{i\cdot}=\varepsilon_{ij}-\bar\varepsilon_{i\cdot}$: the random effect cancels. So $\text{SSE}/\sigma^2\sim\chi^2_{k(n-1)}$ exactly as in the fixed model, and it is independent of the $\bar\varepsilon_{i\cdot}$ (§4.1) and of the $a_i$. The group means $\bar y_{i\cdot}=\mu+a_i+\bar\varepsilon_{i\cdot}$ are iid $N(\mu,\sigma_a^2+\sigma^2/n)$, so $\sum_i(\bar y_{i\cdot}-\bar y_{\cdot\cdot})^2\big/(\sigma_a^2+\sigma^2/n)\sim\chi^2_{k-1}$, i.e. $\text{SSA}/(\sigma^2+n\sigma_a^2)\sim\chi^2_{k-1}$, a function of the $\bar y_{i\cdot}$ and hence independent of SSE.
(3) $\dfrac{\text{MSA}/(\sigma^2+n\sigma_a^2)}{\text{MSE}/\sigma^2}=\dfrac{F}{1+n\theta}\sim F_{k-1,k(n-1)}$, and $P(F\gt c)=P\left(F/(1+n\theta)\gt c/(1+n\theta)\right)$.
(4) Unbiasedness follows from (2). $\hat\sigma_a^2\lt0$ iff $F\lt1$, which has probability $P(F_{k-1,k(n-1)}\lt1/(1+n\theta))\gt0$.
(5) $P\left(F_L\le F/(1+n\theta)\le F_U\right)=1-\alpha$, and solving the two inequalities for $\theta$ gives the interval. ∎
The test is the same as in the fixed model; what changes is the hypothesis ($\sigma_a^2=0$ instead of equal means), the interpretation, and the fact that the power has an exact central $F$ form. The table of expected mean squares is the key to every random or mixed analysis: to test a term, divide its mean square by the mean square whose expectation equals that of the term's mean square when the term is absent.
For the $a\times b$ layout with $n$ observations per cell and interaction, the expected mean squares are as follows (mixed: A fixed with $\sum\alpha_i=0$, B and AB random, unrestricted model; $Q_A=\sum\alpha_i^2/(a-1)$, $Q_B=\sum\beta_j^2/(b-1)$, $Q_{AB}=\sum\gamma_{ij}^2/((a-1)(b-1))$).
| Source | df | Fixed | Random | Mixed (A fixed) |
|---|---|---|---|---|
| A | $a-1$ | $\sigma^2+bnQ_A$ | $\sigma^2+n\sigma_{ab}^2+bn\sigma_a^2$ | $\sigma^2+n\sigma_{ab}^2+bnQ_A$ |
| B | $b-1$ | $\sigma^2+anQ_B$ | $\sigma^2+n\sigma_{ab}^2+an\sigma_b^2$ | $\sigma^2+n\sigma_{ab}^2+an\sigma_b^2$ |
| AB | $(a-1)(b-1)$ | $\sigma^2+nQ_{AB}$ | $\sigma^2+n\sigma_{ab}^2$ | $\sigma^2+n\sigma_{ab}^2$ |
| Error | $ab(n-1)$ | $\sigma^2$ | $\sigma^2$ | $\sigma^2$ |
Hence: in the fixed model every effect is tested against MSE; in the random model A and B are tested against MSAB and AB against MSE; in the mixed model the fixed factor A is tested against MSAB, and AB against MSE. Under normality and balance each such ratio has an exact $F$ distribution under its null hypothesis.
Proof (the random model; the others are similar). With $y_{ijk}=\mu+a_i+b_j+c_{ij}+\varepsilon_{ijk}$, $$\bar y_{i\cdot\cdot}-\bar y_{\cdot\cdot\cdot}=(a_i-\bar a)+(\bar c_{i\cdot}-\bar c_{\cdot\cdot})+(\bar\varepsilon_{i\cdot\cdot}-\bar\varepsilon_{\cdot\cdot\cdot});$$ the $b_j$ cancel because every row mean contains the same $\bar b$. The three parts are independent; $\bar c_{i\cdot}$ (an average of $b$ interaction terms) has variance $\sigma_{ab}^2/b$ and $\bar\varepsilon_{i\cdot\cdot}$ has variance $\sigma^2/(bn)$, and for iid variables $E\sum_i(u_i-\bar u)^2=(a-1)\operatorname{Var}(u_i)$. Hence $$E[\text{SSA}]=bn(a-1)\left[\sigma_a^2+\frac{\sigma_{ab}^2}{b}+\frac{\sigma^2}{bn}\right]=(a-1)\left[\sigma^2+n\sigma_{ab}^2+bn\sigma_a^2\right].$$ For the interaction line, $\bar y_{ij\cdot}-\bar y_{i\cdot\cdot}-\bar y_{\cdot j\cdot}+\bar y_{\cdot\cdot\cdot}$ contains only the $c$ and $\varepsilon$ terms, in the same double-centred form as the residual of Theorem 6.14, giving $(a-1)(b-1)(\sigma_{ab}^2+\sigma^2/n)$ times $n$. In the mixed model replace $a_i-\bar a$ by the constant $\alpha_i$ (with $\bar\alpha=0$). Exact $F$ laws: in the balanced case the four sums of squares are independent and each, divided by its expected mean square times its df, is $\chi^2$, by the same orthogonal decomposition as in Theorem 6.15. ∎
Some texts use a restricted mixed model, which imposes $\sum_ic_{ij}=0$ for each $j$. Then $E[\text{MSB}]=\sigma^2+an\sigma_b^2$ (no $\sigma_{ab}^2$ term, with the interaction variance defined slightly differently), and B is tested against MSE. Both versions test the fixed factor against MSAB. State which version you use; examination problems generally intend the unrestricted one unless they say otherwise. Variance components are estimated by equating mean squares to their expectations: in the random two-way model, $\hat\sigma_{ab}^2=(\text{MSAB}-\text{MSE})/n$, $\hat\sigma_a^2=(\text{MSA}-\text{MSAB})/(bn)$, $\hat\sigma_b^2=(\text{MSB}-\text{MSAB})/(an)$.
Analysis of Covariance
In a teaching experiment, students taught by different methods also differ in their prior knowledge; in an agricultural trial, plots differ in their initial number of plants; in a clinical trial, patients differ in their baseline blood pressure. If a covariate $x$ measured before treatment is linearly related to the response, adjusting for it removes part of the error variance and corrects for chance imbalances between groups. The analysis of covariance combines the one-way model with a common regression slope.
$$y_{ij}=\mu+\alpha_i+\beta(x_{ij}-\bar x_{\cdot\cdot})+\varepsilon_{ij},\qquad\varepsilon_{ij}\ \text{iid}\ N(0,\sigma^2),$$ $i=1,\dots,k$, $j=1,\dots,n_i$, $N=\sum n_i$, where $x_{ij}$ is a known covariate not influenced by the treatments. The slope $\beta$ is common to all groups (parallel regression lines). Write the within-group (error) sums of squares and products $$E_{xx}=\sum_{ij}(x_{ij}-\bar x_{i\cdot})^2,\quad E_{xy}=\sum_{ij}(x_{ij}-\bar x_{i\cdot})(y_{ij}-\bar y_{i\cdot}),\quad E_{yy}=\sum_{ij}(y_{ij}-\bar y_{i\cdot})^2,$$ and $T_{xx},T_{xy},T_{yy}$ for the same sums about the grand means.
Assume $E_{xx}\gt0$. Then:
- the least squares estimates are $\hat\beta=E_{xy}/E_{xx}$ (the pooled within-group slope) and $\widehat{\mu+\alpha_i}=\bar y_{i\cdot}-\hat\beta(\bar x_{i\cdot}-\bar x_{\cdot\cdot})$, the adjusted treatment means;
- the error sum of squares is $\text{SSE}=E_{yy}-E_{xy}^2/E_{xx}$ on $N-k-1$ df, and $\hat\sigma^2=\text{SSE}/(N-k-1)$;
- under $H_0:\alpha_1=\cdots=\alpha_k$ the model is a single regression line, with $\text{SSE}_0=T_{yy}-T_{xy}^2/T_{xx}$, and $$F=\frac{(\text{SSE}_0-\text{SSE})/(k-1)}{\text{SSE}/(N-k-1)}\sim F_{k-1,N-k-1};$$
- $H_0:\beta=0$ is tested by $F=\dfrac{E_{xy}^2/E_{xx}}{\hat\sigma^2}\sim F_{1,N-k-1}$;
- the difference of two adjusted means has variance $$\sigma^2\left(\frac1{n_i}+\frac1{n_j}+\frac{(\bar x_{i\cdot}-\bar x_{j\cdot})^2}{E_{xx}}\right).$$
Proof. (1), (2) Put $m_i=\mu+\alpha_i-\beta\bar x_{\cdot\cdot}$. For fixed $\beta$, $\sum_{ij}(y_{ij}-m_i-\beta x_{ij})^2$ is minimised over $m_i$ by $m_i=\bar y_{i\cdot}-\beta\bar x_{i\cdot}$ (a sample mean minimises the sum of squared deviations). Substituting, $$S(\beta)=\sum_{ij}\left[(y_{ij}-\bar y_{i\cdot})-\beta(x_{ij}-\bar x_{i\cdot})\right]^2=E_{yy}-2\beta E_{xy}+\beta^2E_{xx},$$ a quadratic minimised at $\hat\beta=E_{xy}/E_{xx}$ with minimum $E_{yy}-E_{xy}^2/E_{xx}$. Then $\widehat{\mu+\alpha_i}=\hat m_i+\hat\beta\bar x_{\cdot\cdot}=\bar y_{i\cdot}-\hat\beta(\bar x_{i\cdot}-\bar x_{\cdot\cdot})$. The model has rank $k+1$ (the $k$ group indicators and $x$, which is not in their span because $E_{xx}\gt0$), so the error df are $N-k-1$.
(3) The same computation with one common intercept gives $\text{SSE}_0=T_{yy}-T_{xy}^2/T_{xx}$; the hypothesis consists of $k-1$ independent estimable contrasts, so the general $F$-test of §6.1 applies with $q=k-1$, $n-r=N-k-1$. (4) is the same $F$-test with $\text{RSS}_0=E_{yy}$ (the one-way model) and $\text{RSS}_0-\text{RSS}=E_{xy}^2/E_{xx}$.
(5) $\hat\beta=\sum_{ij}(x_{ij}-\bar x_{i\cdot})y_{ij}/E_{xx}$ has variance $\sigma^2/E_{xx}$, and $$\operatorname{Cov}(\bar y_{i\cdot},\hat\beta)=\frac{\sigma^2}{n_iE_{xx}}\sum_j(x_{ij}-\bar x_{i\cdot})=0.$$ The difference of adjusted means is $\bar y_{i\cdot}-\bar y_{j\cdot}-\hat\beta(\bar x_{i\cdot}-\bar x_{j\cdot})$, a sum of three uncorrelated terms with variances $\sigma^2/n_i$, $\sigma^2/n_j$ and $(\bar x_{i\cdot}-\bar x_{j\cdot})^2\sigma^2/E_{xx}$. ∎
The error sum of squares can be written $E_{yy}(1-r_w^2)$, where $r_w=E_{xy}/\sqrt{E_{xx}E_{yy}}$ is the within-group correlation: a covariate that is strongly correlated with the response within groups removes most of the error variance, at the cost of one degree of freedom. The adjustment also moves the treatment means towards what they would have been at a common covariate value $\bar x_{\cdot\cdot}$. Two assumptions need checking: that the slopes are equal (fit separate slopes and compare by an $F$-test, Exercise IV.5), and that the treatments do not affect the covariate (otherwise adjusting removes part of the treatment effect).
- Many $t$-tests are not one ANOVA. With $k=5$ groups there are $10$ pairs; testing each at $5\%$ gives a familywise error rate far above $5\%$ (Bonferroni's inequality bounds it only by $10\times0.05=0.50$). Use the $F$-test and an adjusted multiple-comparison method.
- A significant $F$ does not say which means differ. It guarantees that some contrast is Scheffé-significant (Theorem 6.13(2)), but that contrast need not be a pairwise difference; and a non-significant $F$ does not prove the means equal.
- No main effect does not mean no effect. In Example 4, catalyst has $F=0.17$, yet the catalyst changes the yield by $3$ to $5$ units at the extreme temperatures. With a large interaction, main effects are averages of opposite effects.
- One observation per cell: interaction and error are confounded. The "error" line then estimates $\sigma^2+$ interaction; a test for interaction needs replication or a model for its form (Tukey).
- Wrong denominators in random and mixed models. The fixed factor of a mixed model, and both main effects of a random model, are tested against MSAB, not MSE (Example 6); using MSE overstates significance.
- Negative variance-component estimates. $\hat\sigma_a^2=(\text{MSA}-\text{MSE})/n$ is negative whenever $F\lt1$; this is not an arithmetic error. Report $0$ (and note that the truncated estimator is biased) or use a likelihood-based method.
- Unequal variances with unequal group sizes. The $F$-test is fairly robust to non-normality and, with equal $n_i$, to unequal variances; but if the larger groups have the smaller variances it rejects far too often. Check the residuals.
- Unbalanced two-way data. SSA, SSB and SSAB no longer add up to the model sum of squares, and the sum of squares for A depends on the order of fitting (Exercise II.8).
- ANCOVA can lower $F$. Adjustment does not always increase the treatment $F$: if the groups differ in $y$ only because they differ in $x$, the adjusted means are close and the treatment effect vanishes (Exam corner E6).
- Covariates affected by treatment, or unequal slopes. Adjusting for a variable measured after treatment can remove the treatment effect; with non-parallel lines there is no single "adjusted difference", and it depends on the value of $x$.
- Fixed or random is a choice about the inference, not about the data. The same numbers can be analysed both ways; the random model generalises to unseen levels and pays for it with wider intervals.
Exercises
- 1 Prove that $\text{SST}=\text{SSB}+\text{SSW}$ in the one-way layout.
- 2 Groups A: $8,10,12$; B: $12,14,13,15$; C: $9,7,8$. Construct the ANOVA table and test equality of means at $5\%$.
- 3 Derive $E[\text{MSB}]=\sigma^2+\sum n_i(\mu_i-\bar\mu)^2/(k-1)$ without assuming normality.
- 4 Show that for $k=2$ the ANOVA $F$ equals the square of the pooled two-sample $t$ statistic.
- 5 With $k=4$ groups of $n=5$ and means $10,12,14,18$, show that the contrasts $(1,-1,0,0)$, $(1,1,-2,0)$, $(1,1,1,-3)$ are mutually orthogonal, compute their sums of squares, and check that they add up to SSB.
- 6 An experiment has $k=6$ groups. How many pairwise comparisons are there, and at what level must each be tested for a Bonferroni familywise level of $5\%$?
- 7 Prove that the one-way $F$-test rejects at level $\alpha$ if and only if at least one contrast $\hat\psi$ satisfies $\text{SS}(\hat\psi)\gt(k-1)\,\text{MSW}\,F_{k-1,N-k,\alpha}$.
- 8 With $k=4$, $n=6$ per group, $\sigma=1$ and true means $(0,0,0,1)$, find the noncentrality parameter of the $F$-test and its power at $5\%$ (software for the noncentral $F$).
- 9 Show that the side conditions $\sum n_i\alpha_i=0$ and $\alpha_k=0$ give different estimates of $\alpha_1$, but the same estimate of every contrast.
- 1 Analyse the $3\times3$ table (rows A, columns B, one observation per cell) with rows $5,7,6$; $8,9,10$; $5,6,4$ by the additive model.
- 2 Verify that the row array $(\bar y_{i\cdot}-\bar y_{\cdot\cdot})$ and the column array $(\bar y_{\cdot j}-\bar y_{\cdot\cdot})$ of Theorem 6.14 are orthogonal.
- 3 Show that if the additive model is fitted but interaction is present, $E[\text{MSE}]=\sigma^2+\sum\gamma_{ij}^2/((a-1)(b-1))$. What does this do to the $F$-tests for main effects?
- 4 In a $3\times4$ layout with $n=3$ per cell, $\text{SSA}=24$, $\text{SSB}=45$, $\text{SSAB}=18$, $\text{SST}=159$. Complete the ANOVA table (fixed effects) and test all effects at $5\%$.
- 5 For Example 4, estimate the interaction effects $\gamma_{ij}$ and draw the interaction plot. Which simple effect of catalyst is largest?
- 6 Show that in a $2\times2$ layout with $n$ observations per cell, $\text{SSAB}=\frac n4\left(\bar y_{11}-\bar y_{12}-\bar y_{21}+\bar y_{22}\right)^2$.
- 7 Explain why the hypothesis of no interaction cannot be tested with one observation per cell without further assumptions, and state the form of interaction assumed by Tukey's one-degree-of-freedom test.
- 8 (Unbalanced data.) Cells $(1,1)$: $2,4$; $(1,2)$: $6$; $(2,1)$: $3$; $(2,2)$: $7,9$. Fitting additive models, compute the sum of squares for A fitted first, $\text{SS}(A)$, and after B, $\text{SS}(A\mid B)$.
- 1 In the one-way random model, show that $\operatorname{Corr}(y_{ij},y_{ij'})=\sigma_a^2/(\sigma_a^2+\sigma^2)$ for $j\ne j'$ and that the covariance matrix of the $n$ observations in a group is $\sigma^2I+\sigma_a^2J$.
- 2 Prove $E[\text{MSA}]=\sigma^2+n\sigma_a^2$ directly from $\bar y_{i\cdot}=\mu+a_i+\bar\varepsilon_{i\cdot}$.
- 3 With $k=6$ groups of $n=3$, $\text{MSA}=14$ and $\text{MSE}=4$: test $\sigma_a^2=0$ at $5\%$ and estimate $\sigma_a^2$ and the intraclass correlation.
- 4 If $k=5$, $n=4$ and $\sigma_a^2=\sigma^2$, find the probability that the ANOVA estimator $\hat\sigma_a^2$ is negative.
- 5 For $k=5$, $n=4$, find the power of the $5\%$ test of $\sigma_a^2=0$ when $\sigma_a^2/\sigma^2=0.5$.
- 6 In the two-way random model derive $E[\text{MSB}]=\sigma^2+n\sigma_{ab}^2+an\sigma_b^2$.
- 7 In a mixed model with A fixed ($a=4$), B random ($b=5$) and $n=2$, give the degrees of freedom of every line and the $F$ ratio (with its df) for each test.
- 8 Under normality, show that $\operatorname{Var}(\hat\sigma_a^2)=\frac{2}{n^2}\left[\frac{(\sigma^2+n\sigma_a^2)^2}{k-1}+\frac{\sigma^4}{k(n-1)}\right]$ in the balanced one-way random model.
- 1 Derive $\hat\beta=E_{xy}/E_{xx}$ in the ANCOVA model by first minimising over the group intercepts.
- 2 Show that $\bar y_{i\cdot}$ and $\hat\beta$ are uncorrelated, and deduce the variance of an adjusted mean.
- 3 Two groups of three: group 1 $x=(2,4,6)$, $y=(5,8,10)$; group 2 $x=(5,7,9)$, $y=(9,12,14)$. Compute the adjusted difference of means and test it at $5\%$. Compare with the unadjusted analysis.
- 4 Show that if all group means $\bar x_{i\cdot}$ of the covariate are equal, the adjusted means equal the raw means and $\text{SSE}_0-\text{SSE}$ equals the one-way treatment sum of squares of $y$. What does the covariate still achieve?
- 5 (Parallelism.) For the data of Example 7, fit a separate slope in each group and test at $5\%$ whether the slopes are equal.
- 6 A trial of a diet uses weight loss as response and "calories eaten during the trial" as a covariate. Explain why this adjustment is misleading.
- 7 Show that $\text{SSE}=E_{yy}(1-r_w^2)$ with $r_w=E_{xy}/\sqrt{E_{xx}E_{yy}}$, and compute $r_w$ for Example 7.
E1. CSIR Part B GATE ST In a one-way ANOVA with $4$ groups of $6$ observations each, the between-groups sum of squares is $36$ and the total sum of squares is $96$. The value of the $F$ statistic is (A) $1.8$; (B) $0.6$; (C) $4.0$; (D) $2.4$.
E2. CSIR Part B In a two-way fixed effects layout with factor A at $3$ levels, factor B at $4$ levels and $2$ observations per cell, the $F$ statistic for testing the interaction has degrees of freedom (A) $(6,23)$; (B) $(6,12)$; (C) $(5,12)$; (D) $(6,18)$.
E3. CSIR Part B GATE ST In a balanced one-way random effects model with $5$ groups of $4$ observations, $\text{MSA}=26$ and $\text{MSE}=6$. The ANOVA estimate of the intraclass correlation $\sigma_a^2/(\sigma_a^2+\sigma^2)$ is closest to (A) $0.77$; (B) $0.23$; (C) $0.83$; (D) $0.45$.
E4. CSIR Part C In the normal one-way fixed effects model with $k\ge2$ groups, which of the following are true? (A) MSW is unbiased for $\sigma^2$ whether or not the group means are equal. (B) $E[\text{MSB}]\ge E[\text{MSW}]$, with equality if and only if all group means are equal. (C) SSB and SSW are independent only when the group means are equal. (D) For $k=2$, the $F$ statistic is the square of the pooled two-sample $t$ statistic.
E5. CSIR Part C Consider a balanced two-way mixed model with A fixed at $a$ levels, B random at $b$ levels, a random interaction (unrestricted model) and $n\ge2$ observations per cell. Which of the following are true? (A) The hypothesis of no A effects is tested by MSA/MSAB on $(a-1,(a-1)(b-1))$ degrees of freedom. (B) The hypothesis $\sigma_{ab}^2=0$ is tested by MSAB/MSE. (C) With $n=1$ observation per cell, the hypothesis of no A effects could not be tested. (D) $E[\text{MSE}]=\sigma^2$.
E6. CSIR Part C In a one-way analysis of covariance with $k$ treatments, $N$ experimental units and one covariate $x$ (common slope), which of the following are true? (A) The error sum of squares has $N-k-1$ degrees of freedom. (B) The adjusted treatment means are $\bar y_{i\cdot}-\hat\beta(\bar x_{i\cdot}-\bar x_{\cdot\cdot})$, where $\hat\beta$ is the pooled within-treatment regression coefficient. (C) If all treatments have the same covariate mean, the adjusted and unadjusted treatment means coincide. (D) The $F$ statistic for treatments in the ANCOVA is always at least as large as the one-way ANOVA $F$ statistic computed from $y$ alone.
Suggested plan — 3 lectures × 60 min.
- One-way ANOVA. Start with the multiplicity problem of many $t$-tests. Definition 6.6 and Theorem 6.11 with full proofs of (1)–(2) and the projection proof of (3); Example 1 with the correction-factor shortcut. Project Interactive 1, first with $\delta=0$ (the $F$ values hover around $1$, about $5\%$ rejected), then with $\delta=0.6$. Contrasts (Theorem 6.12), the three multiple-comparison methods (Theorem 6.13) and Example 2.
- Two-way layouts. Theorem 6.14 and Example 3, stressing the gain from blocking (the same fertiliser data are significant with blocks and invisible without). Definition 6.8(2), Theorem 6.15, Example 4, then Interactive 2: raise the interaction slider until the profiles cross and watch $F$ for A fall. The strategy box on how to read a two-way table.
- Random effects and ANCOVA. Definition 6.9, Theorem 6.16 with proof, Example 5 (point out how wide the interval for $\theta$ is). Theorem 6.17 as a table with the rule for denominators; Example 6 as a class discussion. ANCOVA: Theorem 6.18 via the two-stage minimisation, Example 7. Finish with E5 and E6.
Discussion questions. (1) A company tests four suppliers chosen because they are the only ones available. Should "supplier" be fixed or random? What if they were drawn from a list of fifty? (2) Why can a factor have a large effect at every level of another factor and still have a main-effect $F$ near zero? (3) In Example 7, would you report the raw or the adjusted difference between methods 2 and 1, and why?
Misconceptions to watch for. Using MSE as the denominator for every test in random and mixed models; interpreting main effects in the presence of strong interaction; believing that a significant $F$ identifies the differing pair; counting error degrees of freedom as $N-k$ in ANCOVA; adjusting for covariates measured after treatment; reporting a negative variance component without comment.
Suggested homework. I(2, 4, 5, 7), II(1, 4, 6, 8), III(3, 4, 6, 7), IV(3, 4, 5); Exam corner E1, E5, E6.
Research question. How robust is the one-way ANOVA $F$-test? Estimate by simulation its actual type I error rate when the errors are skewed or heavy-tailed, and when the group variances are unequal, for balanced and unbalanced designs. When should Welch's ANOVA or a rank test (Kruskal–Wallis) be used instead?
Why it matters. ANOVA is applied every day in agriculture, medicine, psychology and engineering, often to data that are not normal and have unequal spreads. The exact theory of Theorem 6.11 assumes normality and equal variances; whether a published "significant difference" can be trusted depends on how badly the test behaves when these fail. Simulation studies of this kind are a standard part of methodological research and of the guidelines statistical societies give to applied scientists.
What to do.
- Write a simulation that draws $k=3$ groups with equal means from a chosen population, computes the $F$ statistic and records whether $F\gt F_{k-1,N-k,0.05}$. Use $10\,000$ replicates per setting.
- Error laws: normal, exponential (skewed), $t_3$ (heavy-tailed). Designs: $n_i=(10,10,10)$ and $(5,10,20)$. Variances: equal, and standard deviations in the ratio $1:2:4$ (assign the large variance once to the small group, once to the large group).
- Tabulate the estimated sizes with their Monte Carlo standard errors ($\approx\sqrt{0.05\times0.95/10\,000}=0.002$).
- Repeat with Welch's test and the Kruskal–Wallis test, and write recommendations.
Tools: R (aov, oneway.test, kruskal.test) or Python (scipy.stats.f_oneway, scipy.stats.kruskal); Interactive 1 shows the null behaviour for normal data.
Hand in: A table of estimated sizes for each test and setting, a short explanation of the pattern, and a one-paragraph recommendation for practitioners.
Going further. Read about Box's (1954) approximations for the effect of unequal variances, and about permutation tests, which give exact size under exchangeability without assuming normality.
🧪 Self-check quiz
Q1. A one-way ANOVA compares $k$ groups with $N$ observations in total. The $F$ statistic has degrees of freedom
Q2. In the one-way fixed effects model, $E[\text{MSB}]$ equals
Q3. In a balanced two-way random effects model with interaction, the main effect of A is tested by
Q4. A balanced one-way random effects analysis with $n=5$ observations per group gives $\text{MSA}=19$ and $\text{MSE}=4$. The ANOVA estimate of $\sigma_a^2$ is
Q5. In a one-way analysis of covariance with $k=4$ treatments, $N=24$ units and one covariate, the error degrees of freedom are