8.2 PPS Sampling; Ratio and Regression Estimators
When units differ in size, or a cheap auxiliary variable is known for every unit, equal-probability sampling wastes information. Sampling with probability proportional to size (Hansen–Hurwitz, Horvitz–Thompson) and the ratio and regression estimators put that information to work.
Suppose we want the total milk production of the $250$ villages of a district. The villages differ enormously: a few large ones have thousands of cattle, many small ones a few dozen. From the last livestock census we know, for every village, the number of households, a size measure roughly proportional to what we want. Section 8.1 would draw villages with equal probabilities and multiply the sample mean by $N$. Its error would then be dominated by the luck of catching or missing the few big villages. Two better ideas use the size measure. The first is to select big villages with higher probability and divide each observation by its selection probability; this is probability proportional to size (PPS) sampling. The second is to keep the simple random sample but adjust the estimate: if the sampled villages turn out to be larger than average, scale the estimate down. These are the ratio and regression estimators.
Both ideas recur throughout survey practice: crop-cutting surveys with area as the auxiliary variable, business surveys with last year's turnover, forest inventories with basal area, household surveys with census counts. Both have characteristic failure modes, and CSIR NET loves those. An unbiased estimator can be absurd (Basu's elephants). A variance estimator can be negative. A positive correlation is not enough for the ratio estimator to help. This section develops the exact theory (unbiasedness and variance under the design) and the first-order theory (bias and mean squared error of the ratio and regression estimators), and says precisely when each method wins.
PPS Sampling with Replacement: the Hansen–Hurwitz Estimator
We keep the notation of §8.1: the population is $U=\{1,\dots,N\}$ with values $Y_i$, total $Y$ and mean $\bar Y$. In addition, a positive auxiliary (size) variable $X_i\gt0$ is known for every unit before sampling, with total $X=\sum X_i$ and mean $\bar X$.
Let $p_i=X_i/X$ (the normed size, $\sum p_i=1$). In PPS sampling with replacement (PPSWR) we make $n$ independent draws, each selecting unit $i$ with probability $p_i$. If the $j$-th draw selects a unit with value $y_j$ and normed size $p_j'$, the Hansen–Hurwitz estimator of the total is
$$\hat Y_{\mathrm{HH}}=\frac1n\sum_{j=1}^n\frac{y_j}{p_j'} .$$A unit drawn twice is counted twice. The mean is estimated by $\hat Y_{\mathrm{HH}}/N$.
Under PPSWR with $n$ draws:
- $\hat Y_{\mathrm{HH}}$ is unbiased for $Y$;
- its variance is $$\operatorname{Var}(\hat Y_{\mathrm{HH}})=\frac1n\sum_{i=1}^Np_i\Big(\frac{Y_i}{p_i}-Y\Big)^2=\frac1n\Big(\sum_{i=1}^N\frac{Y_i^2}{p_i}-Y^2\Big);$$
- for $n\ge2$, with $z_j=y_j/p_j'$, an unbiased estimator of this variance is $$v(\hat Y_{\mathrm{HH}})=\frac{1}{n(n-1)}\sum_{j=1}^n\big(z_j-\hat Y_{\mathrm{HH}}\big)^2 ;$$
- the variance is zero if and only if $Y_i/X_i$ is the same for every unit, i.e. $Y_i\propto X_i$.
Proof. Each draw produces the random variable $z=Y_I/p_I$, where $P(I=i)=p_i$, and the $n$ values $z_1,\dots,z_n$ are iid. Now $E z=\sum_ip_i\frac{Y_i}{p_i}=Y$ and $\operatorname{Var}z=\sum_ip_i\left(\frac{Y_i}{p_i}-Y\right)^2$. Expanding the square gives $\sum Y_i^2/p_i-2Y^2+Y^2$. So $\hat Y_{\mathrm{HH}}=\bar z$ is the mean of an iid sample, which gives (1) and (2), and the sample variance of the $z_j$ is unbiased for $\operatorname{Var}z$ (§4.1), which gives (3). For (4), $\sum p_i(Y_i/p_i-Y)^2=0$ iff $Y_i/p_i=Y$ for every $i$ (all $p_i\gt0$), i.e. $Y_i=Yp_i=(Y/X)X_i$. ∎
Part (4) is the whole idea. PPS turns the estimation of $Y$ into the estimation of the mean of the ratios $Y_i/p_i$. These are nearly constant when $Y$ is nearly proportional to the size, so the estimator is very precise. With $p_i=1/N$ the estimator reduces to $N\bar y$ under SRSWR, and Theorem 8.12 reproduces Theorem 8.3.
How to draw units with probability proportional to $X_i$. When the sizes $X_i$ are integers there are two classical schemes.
- Cumulative total method. Form the cumulative totals $T_i=X_1+\dots+X_i$ ($T_0=0$, $T_N=X$). Draw a random integer $R$ uniformly from $\{1,\dots,X\}$ and select the unit $i$ with $T_{i-1}\lt R\le T_i$. Each unit owns $X_i$ of the $X$ numbers, so it is selected with probability $X_i/X$.
- Lahiri's method (1951). Let $M\ge\max_iX_i$. Draw a pair $(i,R)$ with $i$ uniform on $\{1,\dots,N\}$ and $R$ uniform on $\{1,\dots,M\}$, independently. If $R\le X_i$, select unit $i$; otherwise reject the pair and draw a new one. Repeat until a unit is selected.
The cumulative total method needs all cumulative totals to be computed and listed, which is tedious for large $N$. Lahiri's method needs only the individual sizes and an upper bound $M$.
In Lahiri's method each trial selects some unit with probability $\theta=\frac{X}{NM}$. The selected unit is $i$ with probability exactly $X_i/X$, and the number of trials needed has the geometric distribution with mean $NM/X=M/\bar X$.
Proof. In a single trial, $P(\text{unit }i\text{ chosen and accepted})=\frac1N\cdot\frac{X_i}{M}$, because $R\le X_i\le M$ has probability $X_i/M$. Summing over $i$, a trial succeeds with probability $\theta=\frac{X}{NM}\gt0$. Trials are independent, so the first success occurs at trial $t$ with probability $(1-\theta)^{t-1}\theta$, a geometric law with mean $1/\theta$. The selected unit is the one accepted at the first successful trial, and by independence $$P(\text{selected unit}=i)=\sum_{t\ge1}(1-\theta)^{t-1}\cdot\frac{X_i}{NM}=\frac{X_i/(NM)}{\theta}=\frac{X_i}{X}.\ ∎$$
Choosing $M=\max X_i$ minimises the expected number of rejections. If one uses an $M$ that is smaller than some $X_i$, those units are under-selected and the method is wrong.
Sampling without Replacement: Inclusion Probabilities and the Horvitz–Thompson Estimator
Sampling with replacement wastes effort on repeated units. For designs without replacement the natural tool is the inclusion probability $\pi_i=P(i\in s)$, which we met in Definition 8.1. Horvitz and Thompson (1952) observed that weighting each sampled value by $1/\pi_i$ gives an unbiased estimator of the total for any design.
Let $p(\cdot)$ be a sampling design without replacement, with first- and second-order inclusion probabilities $\pi_i$ and $\pi_{ij}$ ($i\ne j$), and let $Z_i=\mathbf 1\{i\in s\}$. If every $\pi_i\gt0$, the Horvitz–Thompson (HT) estimator of $Y$ is
$$\hat Y_{\mathrm{HT}}=\sum_{i\in s}\frac{y_i}{\pi_i}=\sum_{i=1}^NZ_i\frac{Y_i}{\pi_i}.$$The design has fixed size $n$ if every possible sample has exactly $n$ units. A fixed-size design with $\pi_i=nX_i/X$ is called a $\pi$ps design.
For any design without replacement, $\sum_i\pi_i=E[n(s)]$, the expected sample size. If the design has fixed size $n$, then for every $i$
$$\sum_{i=1}^N\pi_i=n,\qquad \sum_{j\ne i}\pi_{ij}=(n-1)\pi_i,\qquad \sum_i\sum_{j\ne i}\pi_{ij}=n(n-1).$$Proof. $n(s)=\sum_iZ_i$, so $E[n(s)]=\sum_iE Z_i=\sum\pi_i$. For a fixed-size design $\sum_{j\ne i}Z_j=n-Z_i$, so $Z_i\sum_{j\ne i}Z_j=Z_i(n-1)$ (using $Z_i^2=Z_i$). Taking expectations gives $\sum_{j\ne i}\pi_{ij}=(n-1)\pi_i$, and summing over $i$ gives $n(n-1)$. ∎
Let every $\pi_i\gt0$.
- $\hat Y_{\mathrm{HT}}$ is unbiased for $Y$, for every population $(Y_1,\dots,Y_N)$.
- Its variance is $$\operatorname{Var}(\hat Y_{\mathrm{HT}})=\sum_{i}\frac{1-\pi_i}{\pi_i}Y_i^2+\sum_{i\ne j}\frac{\pi_{ij}-\pi_i\pi_j}{\pi_i\pi_j}Y_iY_j .$$
- (Sen–Yates–Grundy form.) If the design has fixed size $n$, $$\operatorname{Var}(\hat Y_{\mathrm{HT}})=\frac12\sum_{i\ne j}\big(\pi_i\pi_j-\pi_{ij}\big)\Big(\frac{Y_i}{\pi_i}-\frac{Y_j}{\pi_j}\Big)^2 .$$ In particular the variance is $0$ when $\pi_i\propto Y_i$.
- If every $\pi_{ij}\gt0$, the following are unbiased estimators of the variance (the second for fixed-size designs): $$v_{\mathrm{HT}}=\sum_{i\in s}\frac{1-\pi_i}{\pi_i^2}y_i^2+\sum_{\substack{i,j\in s\\ i\ne j}}\frac{\pi_{ij}-\pi_i\pi_j}{\pi_{ij}\,\pi_i\pi_j}y_iy_j,$$ $$v_{\mathrm{SYG}}=\sum_{\substack{i,j\in s\\ i\lt j}}\frac{\pi_i\pi_j-\pi_{ij}}{\pi_{ij}}\Big(\frac{y_i}{\pi_i}-\frac{y_j}{\pi_j}\Big)^2 .$$
Proof. Put $a_i=Y_i/\pi_i$, so $\hat Y_{\mathrm{HT}}=\sum_iZ_ia_i$ with $E Z_i=\pi_i$, $\operatorname{Var}Z_i=\pi_i(1-\pi_i)$ and $\operatorname{Cov}(Z_i,Z_j)=\pi_{ij}-\pi_i\pi_j$. (1) $E\hat Y_{\mathrm{HT}}=\sum\pi_ia_i=\sum Y_i$. (2) The variance of a linear combination of the $Z_i$ is $$\sum_ia_i^2\pi_i(1-\pi_i)+\sum_{i\ne j}a_ia_j(\pi_{ij}-\pi_i\pi_j),$$ which is the stated formula.
(3) Expand $(a_i-a_j)^2=a_i^2+a_j^2-2a_ia_j$. By symmetry in $i,j$, $$\tfrac12\sum_{i\ne j}(\pi_i\pi_j-\pi_{ij})(a_i-a_j)^2=\sum_ia_i^2\sum_{j\ne i}(\pi_i\pi_j-\pi_{ij})-\sum_{i\ne j}(\pi_i\pi_j-\pi_{ij})a_ia_j .$$ By Theorem 8.14 the inner sum is $\pi_i(n-\pi_i)-(n-1)\pi_i=\pi_i(1-\pi_i)$. So the first term is $\sum a_i^2\pi_i(1-\pi_i)$, and the second is $\sum_{i\ne j}(\pi_{ij}-\pi_i\pi_j)a_ia_j$; together they give (2). If $\pi_i=cY_i$, all $a_i$ are equal and every term vanishes.
(4) For any numbers $g_i$ and $h_{ij}$, $E\sum_{i\in s}g_i=\sum_i\pi_ig_i$ and $E\sum_{i\ne j\in s}h_{ij}=\sum_{i\ne j}\pi_{ij}h_{ij}$. Dividing each term of (2) or (3) by the probability that it is observed ($\pi_i$ or $\pi_{ij}$) therefore gives unbiased estimators. ∎
Two remarks. First, the HT estimator is unbiased only if every $\pi_i\gt0$: a unit that can never be sampled contributes nothing, and its value is invisible to every estimator. Second, unbiased variance estimation needs $\pi_{ij}\gt0$ for all pairs. Systematic sampling (§8.1) fails this. For SRSWOR, $\pi_i=n/N$ and $\hat Y_{\mathrm{HT}}=N\bar y$, and Theorem 8.15 reproduces Theorem 8.2 (Exercise II.3). The condition $\pi_i\pi_j\ge\pi_{ij}$ for all pairs (the Sen–Yates–Grundy condition) guarantees $v_{\mathrm{SYG}}\ge0$. It holds for SRSWOR and for the standard $\pi$ps schemes, but not for every design.
Draw-by-draw PPS without replacement. The simplest scheme draws the first unit with probabilities $p_i$, the second with probabilities $p_j/(1-p_i)$ among the remaining units, and so on. For $n=2$,
$$\pi_{ij}=p_ip_j\Big(\frac1{1-p_i}+\frac1{1-p_j}\Big),\qquad \pi_i=p_i\Big(1+\sum_{j\ne i}\frac{p_j}{1-p_j}\Big).$$These $\pi_i$ are not equal to $2p_i$ in general. Using $y_i/(2p_i)$ as if they were is a classic error.
The Ratio Estimator
Now return to simple random sampling, but observe the auxiliary variable on each sampled unit together with $y$. If the sample happens to contain unusually large units, then $\bar x\gt\bar X$, and $\bar y$ is probably too large as well. The ratio estimator corrects for this.
Let $(y_i,x_i)$, $i=1,\dots,n$, be an SRSWOR from the population of pairs $(Y_i,X_i)$ with $X_i\gt0$ and $\bar X$ known. Write $R=\bar Y/\bar X=Y/X$ for the population ratio, $S_x^2,S_y^2$ for the population variances, $S_{xy}=\frac1{N-1}\sum(Y_i-\bar Y)(X_i-\bar X)$, $\rho=S_{xy}/(S_xS_y)$, and $C_x=S_x/\bar X$, $C_y=S_y/\bar Y$ for the coefficients of variation. The ratio estimators of $R$, $\bar Y$ and $Y$ are
$$\hat R=\frac{\bar y}{\bar x},\qquad \bar y_R=\hat R\,\bar X=\frac{\bar y}{\bar x}\bar X,\qquad \hat Y_R=\hat R\,X .$$Under SRSWOR,
$$E[\hat R]-R=-\frac{\operatorname{Cov}(\hat R,\bar x)}{\bar X},$$and consequently
$$\frac{|E[\bar y_R]-\bar Y|}{\sqrt{\operatorname{Var}(\bar y_R)}}\le\operatorname{CV}(\bar x)=\sqrt{\frac{1-f}{n}}\;C_x .$$So the bias is negligible compared with the standard error whenever the coefficient of variation of $\bar x$ is small.
Proof. Since $\hat R\bar x=\bar y$, we have $E[\hat R\bar x]=\bar Y=R\bar X$. Therefore $$\operatorname{Cov}(\hat R,\bar x)=E[\hat R\bar x]-E[\hat R]\,\bar X=\bar X\big(R-E\hat R\big).$$ By the Cauchy–Schwarz inequality, $|\operatorname{Cov}(\hat R,\bar x)|\le\sqrt{\operatorname{Var}\hat R}\sqrt{\operatorname{Var}\bar x}$. Hence $|E\hat R-R|\le\sqrt{\operatorname{Var}\hat R}\cdot\sqrt{\operatorname{Var}\bar x}/\bar X$, and $\sqrt{\operatorname{Var}\bar x}/\bar X=\sqrt{(1-f)/n}\,C_x$ by Theorem 8.2. Multiplying $\hat R$ by the constant $\bar X$ multiplies the bias and the standard deviation by the same factor. ∎
Under SRSWOR, to first order in $1/n$ (that is, neglecting terms of order $n^{-2}$),
$$E[\bar y_R]-\bar Y\approx\frac{1-f}{n}\,\bar Y\big(C_x^2-\rho\,C_xC_y\big),$$ $$\operatorname{MSE}(\bar y_R)\approx\frac{1-f}{n}\big(S_y^2+R^2S_x^2-2R\rho S_xS_y\big)=\frac{1-f}{n}\cdot\frac{1}{N-1}\sum_{i=1}^N\big(Y_i-RX_i\big)^2 .$$Consequently $\bar y_R$ is more precise than $\bar y$ (to this order) if and only if
$$\rho\gt\frac{C_x}{2C_y}=\frac{R\,S_x}{2S_y}.$$The MSE is estimated by $v(\bar y_R)=\frac{1-f}{n}\cdot\frac1{n-1}\sum_{i=1}^n(y_i-\hat Rx_i)^2$.
Proof (careful sketch). Write $\bar y=\bar Y(1+\varepsilon_0)$ and $\bar x=\bar X(1+\varepsilon_1)$. Then $E\varepsilon_0=E\varepsilon_1=0$, and by Theorem 8.2 and Exercise IV.3 of §8.1, $$E\varepsilon_0^2=\tfrac{1-f}{n}C_y^2,\qquad E\varepsilon_1^2=\tfrac{1-f}{n}C_x^2,\qquad E\varepsilon_0\varepsilon_1=\tfrac{1-f}{n}\rho C_xC_y .$$ On the event $|\varepsilon_1|\lt1$, which has probability tending to $1$, expand $$\bar y_R=\bar Y(1+\varepsilon_0)(1+\varepsilon_1)^{-1}=\bar Y\big(1+\varepsilon_0-\varepsilon_1-\varepsilon_0\varepsilon_1+\varepsilon_1^2-\cdots\big).$$ The moments of $\varepsilon_0,\varepsilon_1$ of order $k$ are $O(n^{-k/2})$ (for SRSWOR as for iid sampling). Keeping the terms of order $1/n$: the bias is $\bar Y(E\varepsilon_1^2-E\varepsilon_0\varepsilon_1)$, and the MSE is $\bar Y^2E(\varepsilon_0-\varepsilon_1)^2=\bar Y^2\frac{1-f}{n}(C_y^2+C_x^2-2\rho C_xC_y)$. Since $\bar YC_y=S_y$ and $\bar YC_x=RS_x$, this is the stated form. The second form follows by expanding $\sum(Y_i-RX_i)^2=\sum[(Y_i-\bar Y)-R(X_i-\bar X)]^2$, using $\bar Y=R\bar X$. The MSE of $\bar y_R$ is less than $\frac{1-f}{n}S_y^2$ iff $R^2S_x^2\lt2R\rho S_xS_y$, i.e. $\rho\gt RS_x/(2S_y)=C_x/(2C_y)$ (for $R\gt0$). The estimator $v(\bar y_R)$ replaces $Y_i-RX_i$ by the sample residuals $y_i-\hat Rx_i$, in the same way as $s^2$ replaces $S^2$. ∎
A rigorous version of this argument controls the remainder on the event $|\varepsilon_1|\ge\frac12$ by a large-deviation bound. That is the content of the usual "for large $n$" qualification. In CSIR problems the first-order formulae are always intended.
When is the ratio estimator the right choice? The MSE involves $Y_i-RX_i$, the deviations from a line through the origin with slope $R$. The next theorem makes this precise: the ratio estimator is optimal under exactly such a model.
Suppose that, given the $X_i$, the values follow the model $Y_i=\beta X_i+\varepsilon_i$, where the $\varepsilon_i$ are uncorrelated with $E\varepsilon_i=0$ and $\operatorname{Var}\varepsilon_i=\sigma^2X_i$. Then, for any fixed sample $s$, the BLUE of $\beta$ is $\hat\beta=\sum_{i\in s}y_i\big/\sum_{i\in s}x_i=\hat R$. The best linear unbiased predictor of the total $Y=\sum_{i\in s}y_i+\sum_{i\notin s}Y_i$ is $\hat RX$, the ratio estimator.
Proof. In matrix form, $y=x\beta+\varepsilon$ with $\operatorname{Cov}\varepsilon=\sigma^2V$, $V=\operatorname{diag}(x_i)$. By Aitken's generalised least squares theorem (§6.1), the BLUE is $$\hat\beta=(x^{\mathsf T}V^{-1}x)^{-1}x^{\mathsf T}V^{-1}y=\frac{\sum x_i\cdot y_i/x_i}{\sum x_i^2/x_i}=\frac{\sum y_i}{\sum x_i}.$$ The unobserved part $\sum_{i\notin s}Y_i$ has model mean $\beta\sum_{i\notin s}X_i$, and its errors are uncorrelated with the sample errors. So the best linear unbiased predictor replaces $\beta$ by $\hat\beta$ (Gauss–Markov applied to the linear function $\beta\sum_{i\notin s}X_i$). This gives $\sum_sy_i+\hat R(X-\sum_sx_i)=\hat RX$, because $\sum_sy_i=\hat R\sum_sx_i$. ∎
The ratio estimator is biased under SRSWOR. A clever change of design removes the bias altogether.
In the Midzuno scheme, the first unit is drawn with probability $p_i=X_i/X$ and the remaining $n-1$ units by SRSWOR from the other $N-1$. Then every sample $s$ of size $n$ has probability
$$P(s)=\frac{\sum_{i\in s}p_i}{\binom{N-1}{n-1}}=\frac{\bar x_s}{\bar X}\cdot\frac1{\binom Nn}.$$Hence $\bar y_R=\frac{\bar y}{\bar x}\bar X$ is exactly unbiased for $\bar Y$, and $\pi_i=p_i+(1-p_i)\frac{n-1}{N-1}$.
Proof. The sample $s$ arises when the first draw is some $i\in s$ (probability $p_i$) and the remaining draws give exactly $s\setminus\{i\}$ (probability $1/\binom{N-1}{n-1}$). Summing over $i\in s$ gives the first expression. Since $\sum_{i\in s}p_i=n\bar x_s/(N\bar X)$ and $\binom Nn=\frac Nn\binom{N-1}{n-1}$, the second form follows. Then $$E[\bar y_R]=\sum_sP(s)\frac{\bar y_s}{\bar x_s}\bar X=\frac{1}{\binom Nn}\sum_s\bar y_s=\bar Y,$$ the last step being the unbiasedness of $\bar y$ under SRSWOR. Unit $i$ is included if it is drawn first (probability $p_i$), or if it is not drawn first and is then among the $n-1$ units chosen from $N-1$. ∎
The Regression Estimator and Comparisons
When the relation between $y$ and $x$ is a straight line that does not pass through the origin, the ratio adjustment over- or under-corrects. The remedy is to adjust along the regression line instead.
For a constant $b_0$, the difference-type estimator is $\bar y_{d}=\bar y+b_0(\bar X-\bar x)$; $b_0=1$ gives the difference estimator. The (linear) regression estimator uses the sample least squares slope $b=s_{xy}/s_x^2$:
$$\bar y_{lr}=\bar y+b\,(\bar X-\bar x),\qquad b=\frac{\sum_{i=1}^n(x_i-\bar x)(y_i-\bar y)}{\sum_{i=1}^n(x_i-\bar x)^2}.$$Geometrically, $\bar y_{lr}$ is the height at $x=\bar X$ of the least squares line fitted to the sample (the simple regression of §6.3). Write $B=S_{xy}/S_x^2$ for the population slope.
Under SRSWOR:
- for fixed $b_0$, $\bar y_d$ is unbiased with $$\operatorname{Var}(\bar y_d)=\frac{1-f}{n}\big(S_y^2-2b_0S_{xy}+b_0^2S_x^2\big),$$ minimised at $b_0=B$, with minimum $\frac{1-f}{n}S_y^2(1-\rho^2)$;
- with the estimated slope $b$, $\bar y_{lr}$ has bias of order $1/n$, and to first order $$\operatorname{MSE}(\bar y_{lr})\approx\frac{1-f}{n}S_y^2(1-\rho^2);$$
- to first order, $V_{lr}\le\operatorname{MSE}(\bar y_R)$ and $V_{lr}\le\operatorname{Var}(\bar y)$, with $$\operatorname{Var}(\bar y)-V_{lr}=\frac{1-f}{n}\rho^2S_y^2,\qquad \operatorname{MSE}(\bar y_R)-V_{lr}=\frac{1-f}{n}\big(RS_x-\rho S_y\big)^2 .$$ The second difference is $0$ iff $B=R$, i.e. iff the population regression line passes through the origin.
The MSE of $\bar y_{lr}$ is estimated by $v(\bar y_{lr})=\frac{1-f}{n}\cdot\frac1{n-2}\sum_{i=1}^n\big[(y_i-\bar y)-b(x_i-\bar x)\big]^2$.
Proof. (1) $\bar y_d=b_0\bar X+\overline{(y-b_0x)}$ is a constant plus the sample mean of the variable $U_i=Y_i-b_0X_i$. By Theorem 8.2 it is unbiased for $b_0\bar X+\bar Y-b_0\bar X=\bar Y$, with variance $\frac{1-f}{n}S_U^2$, and $S_U^2=S_y^2-2b_0S_{xy}+b_0^2S_x^2$. This quadratic in $b_0$ is minimised at $b_0=S_{xy}/S_x^2=B$, with value $S_y^2-S_{xy}^2/S_x^2=S_y^2(1-\rho^2)$.
(2) Write $\bar y_{lr}-\bar Y=\big[(\bar y-\bar Y)-B(\bar x-\bar X)\big]-(b-B)(\bar x-\bar X)$. The bracket is $\bar y_d-\bar Y$ with $b_0=B$, which has mean $0$ and the variance of (1). The sample moments $s_{xy}$ and $s_x^2$ converge to $S_{xy}$ and $S_x^2$ with errors $O_p(n^{-1/2})$, so $b-B=O_p(n^{-1/2})$; also $\bar x-\bar X=O_p(n^{-1/2})$. Their product is $O_p(n^{-1})$. It contributes $O(1/n)$ to the bias and $O(n^{-3/2})$ to the MSE, both negligible at first order.
(3) The first difference is $\frac{1-f}{n}[S_y^2-S_y^2(1-\rho^2)]$. For the second, divide by $\frac{1-f}{n}$:
$$\big(S_y^2+R^2S_x^2-2R\rho S_xS_y\big)-S_y^2(1-\rho^2)=(RS_x-\rho S_y)^2 .$$It vanishes iff $R=\rho S_y/S_x=B$. Since $\bar Y=R\bar X$ always, the line $y=\bar Y+B(x-\bar X)$ passes through the origin iff $B=R$. ∎
- Size known for every unit before sampling, $y$ roughly proportional to size → select by PPS. Use Hansen–Hurwitz (with replacement), or Horvitz–Thompson with a $\pi$ps design (without replacement).
- Auxiliary $x$ observed on the sample, $\bar X$ known, $y\approx\beta x$ (line through the origin, spread growing with $x$) → ratio estimator. It beats $\bar y$ iff $\rho\gt C_x/(2C_y)$. Check $\operatorname{CV}(\bar x)$ to make sure the bias is negligible.
- Linear relation with an intercept → regression estimator: always at least as good as both, to first order. Use the difference estimator when the slope is known to be about $1$ (e.g. this year versus last year).
- Negative correlation → the product estimator $\bar y\,\bar x/\bar X$ (Exercise III.6), or the regression estimator.
- Small $n$ and unbiasedness required → the Midzuno scheme, or the Hartley–Ross estimator (Exercise IV.5).
| $x$ | 30 | 52 | 41 | 60 | 38 | 47 | 55 | 35 | 44 | 48 |
|---|---|---|---|---|---|---|---|---|---|---|
| $y$ | 78 | 104 | 96 | 120 | 74 | 108 | 110 | 84 | 86 | 100 |
- Unbiased does not mean sensible. Basu's elephants (Example 4): the HT estimator is unbiased but gives $4040$ kg for fifty elephants with probability $0.99$.
- The HT estimator is not location-invariant. If every $Y_i=c$, then $\hat Y_{\mathrm{HT}}=c\sum_{i\in s}1/\pi_i$, which equals $Nc$ for every sample only if $\sum_{i\in s}1/\pi_i=N$ always. Under the design of Example 3 it ranges from $3.04c$ to $6.53c$ (Exercise II.8).
- The SYG variance estimator can be negative. Take $N=4$, $n=2$, $P\{1,2\}=P\{3,4\}=0.4$ and probability $0.05$ for each other pair. Then every $\pi_i=\frac12$ and $\pi_{12}=0.4\gt\pi_1\pi_2$, so on $s=\{1,2\}$ we get $v_{\mathrm{SYG}}=-1.5(y_1-y_2)^2\lt0$ (E4). The estimator is unbiased on average, but it can be negative on a single sample.
- PPS with an irrelevant size measure can be disastrous. If $Y$ is constant but sizes vary, SRS is exact while $\hat Y_{\mathrm{HH}}$ varies wildly. With $X=(1,2,3,10)$, $Y=(10,12,9,11)$ and $n=1$: $\operatorname{Var}\hat Y_{\mathrm{HH}}=1613.6$ against $20$ for $N\bar y$ (Exercise IV.3).
- Draw-by-draw PPS without replacement does not give $\pi_i=np_i$ (Example 3). Using $\sum y_i/(np_i)$ as an "HT" estimator is then biased.
- Lahiri's method with $M\lt\max X_i$ is wrong. A unit with $X_i\gt M$ is accepted with probability $1$ instead of $X_i/M$. It is therefore selected in proportion to $M$, not to $X_i$, so the largest units are under-selected.
- Positive correlation is not enough for the ratio estimator. It needs $\rho\gt C_x/(2C_y)$ (Example 6, where $\rho=0.4$ makes it worse than $\bar y$).
- The ratio and regression estimators are biased under SRSWOR. For the regression estimator take $N=3$, $n=2$ and the points $(1,1),(2,3),(4,4)$. The three samples give $\bar y_{lr}=\frac{11}3,\frac73,\frac{19}6$, with mean $3.056\ne\bar Y=2.667$. The bias is of order $1/n$ and is usually negligible, but not for tiny samples.
- Ratio versus regression. The ratio estimator is optimal for a line through the origin (Theorem 8.18). With an intercept it can be beaten by a wide margin (Interactive 2, $a\ne0$), whereas the regression estimator is never worse to first order.
Exercises
- 1 Seven shops have sizes (employees) $X=12,30,8,25,45,10,20$. Using the cumulative total method and the random numbers $37,140,66,101$ from $1$–$150$, select a PPSWR sample of four shops.
- 2 The four shops selected in I.1 report monthly sales (thousand rupees) of $95$ (shop 2), $70$ (shop 7), $80$ (shop 4) and $140$ (shop 5). Compute $\hat Y_{\mathrm{HH}}$ and its estimated standard error.
- 3 For the shops of I.1, apply Lahiri's method with $M=45$ to the pairs $(3,12),(6,9),(1,40),(5,44),(2,31),(4,25)$.
- 4 In I.3, find the probability that a pair is rejected and the expected number of pairs needed per selected unit.
- 5 Prove that $\operatorname{Var}(\hat Y_{\mathrm{HH}})=\frac1n\left(\sum Y_i^2/p_i-Y^2\right)$, and that by Cauchy–Schwarz $\sum Y_i^2/p_i\ge Y^2$ for all $p$ with equality iff $p_i\propto Y_i$ (for $Y_i\gt0$).
- 6 A population has $X=(1,2,3,4)$ and $Y=(3,5,9,11)$. For $n=2$ draws, compare $\operatorname{Var}(\hat Y_{\mathrm{HH}})$ with $\operatorname{Var}(N\bar y)$ under SRSWR and under SRSWOR.
- 7 Show that PPSWR with all $X_i$ equal reduces Theorem 8.12 to Theorem 8.3.
- 8 (Applied.) A health department wants the number of children under five in a district of $120$ villages. Population figures from the last census are known for each village. Describe a PPSWR design, the estimator, how you would compute its standard error, and one situation in which the design could perform badly.
- 1 For draw-by-draw PPS without replacement with $n=2$, derive $\pi_{ij}=p_ip_j\left(\frac1{1-p_i}+\frac1{1-p_j}\right)$ and the formula for $\pi_i$.
- 2 With $N=3$ and $p=(0.2,0.3,0.5)$, compute all $\pi_i$ and $\pi_{ij}$ for II.1 and check both identities of Theorem 8.14.
- 3 Show that for SRSWOR $\hat Y_{\mathrm{HT}}=N\bar y$, and that Theorem 8.15(2) reduces to $N^2(1-f)S^2/n$.
- 4 Derive Theorem 8.15(2) directly from the indicator covariances, without the SYG form.
- 5 Prove that the SYG condition $\pi_i\pi_j\ge\pi_{ij}$ holds for SRSWOR, so that $v_{\mathrm{SYG}}\ge0$ there.
- 6 Poisson sampling: each unit is included independently with probability $\pi_i$. Find $\operatorname{Var}(\hat Y_{\mathrm{HT}})$, show it has the form $\sum\frac{1-\pi_i}{\pi_i}Y_i^2$, and explain why the sample size is random.
- 7 Show that if $\pi_i=0$ for some unit $i$, no estimator can be unbiased for $Y$ for every population.
- 8 For the design of Example 3, compute $\sum_{i\in s}1/\pi_i$ for every sample, and deduce the range of $\hat Y_{\mathrm{HT}}$ when every $Y_i=c$.
- 1 From the $N=120$ blocks of a town, an SRSWOR of $8$ blocks gives the number of households $x$ (known for every block from the census, $\bar X=20.5$) and this month's electricity consumption $y$ (MWh): $x=12,20,15,25,18,30,22,10$; $y=33,41,42,52,44,66,45,29$. Compute $\bar y$, $\bar y_R$ and their estimated standard errors.
- 2 For the data of III.1, compute $\bar y_{lr}$ and its estimated standard error.
- 3 A population has $\bar Y=80$, $\bar X=40$, $S_y=16$, $S_x=10$, $\rho=0.6$, $N=1000$; $n=50$. Compute the first-order MSEs of $\bar y$, $\bar y_R$ and $\bar y_{lr}$, and explain the ranking with the criterion of Theorem 8.17.
- 4 Derive the first-order bias $\frac{1-f}{n}\bar Y(C_x^2-\rho C_xC_y)$ of $\bar y_R$ and show that it vanishes to first order iff $B=R$.
- 5 For the population of III.3, use Theorem 8.16 to bound $|\text{bias}|/\text{SE}$ of $\bar y_R$, and compare with the first-order bias.
- 6 The product estimator is $\bar y_P=\bar y\,\bar x/\bar X$. Show that $\operatorname{MSE}(\bar y_P)\approx\frac{1-f}{n}(S_y^2+R^2S_x^2+2R\rho S_xS_y)$, that it beats $\bar y$ iff $\rho\lt-C_x/(2C_y)$, and evaluate it for III.3 with $\rho=-0.8$.
- 7 Show that the difference-type estimator with $b_0=0$, $b_0=R$ and $b_0=B$ gives, respectively, $\bar y$, an estimator with the same first-order MSE as $\bar y_R$, and the minimum-variance member of the family.
- 8 Prove Theorem 8.18 without matrices, by minimising $\sum_{i\in s}(y_i-\beta x_i)^2/x_i$.
- 9 With $X=(1,2,3,4)$, $Y=(2,5,5,9)$ and $n=2$, compute the six Midzuno probabilities $P(s)$ and verify that $E\bar y_R=\bar Y$ and $\pi_i=p_i+(1-p_i)\frac{n-1}{N-1}$.
- 1 In stratified sampling, compare the separate ratio estimator $\bar y_{RS}=\sum_hW_h\frac{\bar y_h}{\bar x_h}\bar X_h$ with the combined one $\bar y_{RC}=\frac{\bar y_{\mathrm{st}}}{\bar x_{\mathrm{st}}}\bar X$. Write their first-order MSEs and explain why the separate estimator is risky when the $n_h$ are small.
- 2 Des Raj's ordered estimator for draw-by-draw PPS with $n=2$: $t_1=y_1/p_1$, $t_2=y_1+y_2(1-p_1)/p_2$, $t=\frac12(t_1+t_2)$, where $y_1$ is the first unit drawn. Prove that $t$ is unbiased for $Y$, and check it on the population of Example 7.
- 3 Give an example in which PPSWR with $n=1$ is much worse than SRSWR with $n=1$, and explain it with Theorem 8.12(2).
- 4 Show that for $n=1$ the HT estimator under PPS sampling coincides with the Hansen–Hurwitz estimator.
- 5 Hartley–Ross. Let $r_i=y_i/x_i$ and $\bar r$ be their sample mean. Show that $\bar r\bar X$ is biased for $\bar Y$, and that $t_{HR}=\bar r\bar X+\frac{n(N-1)}{N(n-1)}(\bar y-\bar r\bar x)$ is exactly unbiased under SRSWOR. Verify it on the population of Example 7.
E1. CSIR Part B GATE ST From a population of $N=10$ units, two draws are made with replacement with probabilities proportional to size. The first draw gives a unit with $p=0.10$ and $y=30$; the second gives a unit with $p=0.25$ and $y=60$. The Hansen–Hurwitz estimate of the population total is (A) $270$; (B) $257.1$; (C) $450$; (D) $540$.
E2. CSIR Part B In Lahiri's method a population has $N=10$ units with total size $X=250$, and $M=50$. The expected number of pairs $(i,R)$ that must be drawn to select one unit is (A) $0.5$; (B) $2$; (C) $5$; (D) $25$.
E3. CSIR Part B GATE ST An SRSWOR of $n=20$ is drawn from $N=500$ units. For the population $S_y=12$, $S_x=10$, $\rho=0.8$ and $R=\bar Y/\bar X=1.5$. To first order, the mean squared error of the ratio estimator $\bar y_R$ is (A) $3.888$; (B) $6.912$; (C) $2.488$; (D) $31.536$.
E4. CSIR Part C Consider a sampling design without replacement in which every unit has $\pi_i\gt0$, and the Horvitz–Thompson estimator $\hat Y_{\mathrm{HT}}$. Which of the following are true? (One or more options may be correct.) (A) If the design has fixed size $n$, then $\sum_i\pi_i=n$. (B) $\hat Y_{\mathrm{HT}}$ is unbiased for the population total. (C) The Sen–Yates–Grundy variance estimator is non-negative for every fixed-size design with all $\pi_{ij}\gt0$. (D) Under PPS sampling with replacement, the Hansen–Hurwitz estimator has zero variance when $Y_i\propto X_i$.
E5. CSIR Part C Let $\bar y_R=(\bar y/\bar x)\bar X$ under SRSWOR, with all $X_i\gt0$. Which of the following are true? (A) $\bar y_R$ is unbiased for $\bar Y$ for every population. (B) $|\text{bias}(\bar y_R)|\le\operatorname{CV}(\bar x)\cdot\operatorname{SD}(\bar y_R)$. (C) To first order, $\bar y_R$ has smaller MSE than $\bar y$ if and only if $\rho\gt C_x/(2C_y)$. (D) If $Y_i=cX_i$ for all $i$, then $\bar y_R=\bar Y$ for every sample.
E6. CSIR Part C Under SRSWOR with $\bar X$ known, consider $\bar y_d=\bar y+b_0(\bar X-\bar x)$ for a constant $b_0$ and the regression estimator $\bar y_{lr}$ with the sample slope. Which of the following are true? (A) $\bar y_d$ is unbiased for $\bar Y$ for every $b_0$. (B) $\min_{b_0}\operatorname{Var}(\bar y_d)=\frac{1-f}{n}S_y^2(1-\rho^2)$. (C) To first order, $\operatorname{MSE}(\bar y_{lr})\le\operatorname{MSE}(\bar y_R)$, with equality iff the population regression line passes through the origin. (D) $\bar y_{lr}$ is exactly unbiased for every population.
Suggested plan — 3 lectures × 60 min.
- PPS with replacement. Motivate with villages of very different sizes. Prove Theorem 8.12 (the iid trick) and emphasise part (4). Cumulative totals and Lahiri (Theorem 8.13 with the geometric argument), then Examples 1–2 on the board. Project Interactive 1, first "proportional", then "decreasing".
- Inclusion probabilities and HT. Theorem 8.14 from indicators, the HT theorem with the SYG proof in full, Example 3 (assign the six samples to six groups of students), and Basu's elephants as a discussion. Finish with the first three bullets of the ⚠️ box.
- Ratio and regression. The Hartley–Ross identity (two lines), the first-order expansion, the criterion $\rho\gt C_x/(2C_y)$, the model view (Theorem 8.18), Midzuno (Theorem 8.19), and then the regression estimator and the comparison theorem. Example 5 with a calculator, Example 6 as a poll. Project Interactive 2: $a=0$, then $a=100$, then a large $\sigma$. End with E4–E6.
Discussion questions. (1) The HT estimator is unbiased for every design, so why do we not simply use it with any convenient design? (2) Under what model would you prefer the ratio estimator to the regression estimator, even with an intercept present (hint: small $n$, heteroscedasticity)? (3) A size measure is ten years old. How does its age affect PPS and ratio estimation differently?
Misconceptions to watch for. Using $y_i/(np_i)$ with draw-by-draw $\pi_i$; believing unbiasedness implies a good estimator; thinking any positive correlation helps the ratio estimator; treating the ratio estimator as unbiased; forgetting that $\bar X$ must be known exactly; using $n-1$ instead of $n-2$ in the residual variance of the regression estimator; applying Lahiri with $M\lt\max X_i$.
Suggested homework. I(1, 2, 4, 6), II(1, 2, 3, 5, 8), III(1, 2, 3, 6, 9), IV(2, 5); Exam corner E1, E4, E6.
Research question. Satellite vegetation indices (such as NDVI) are available for every field at almost no cost, while yields come from expensive crop-cutting experiments. How much can the satellite index improve a crop-yield estimate, and which estimator — PPS, ratio or regression — makes the best use of it?
Why it matters. National agricultural statistics in India, the United States, the European Union and many African countries combine field measurements with remote sensing. The choice of estimator decides how many costly field plots are needed for a given precision, and the wrong choice can introduce bias in years when the yield–NDVI relationship shifts (drought, floods).
What to do.
- Simulate a population of $N=2000$ fields with $x_i=\text{NDVI}_i\sim U(0.2,0.8)$ and yield $y_i=a+\beta x_i+\varepsilon_i$, with $\operatorname{Var}\varepsilon_i\propto x_i$. Take three scenarios: $a=0$; $a=1.5$ t/ha (a yield floor); and a "drought year" in which $\beta$ is halved.
- For $n=30$, run $5000$ replications of each design: SRS with $\bar y$, SRS with $\bar y_R$ and $\bar y_{lr}$, and PPSWR by $x$ with Hansen–Hurwitz. Record the bias and the MSE.
- Compare the simulated MSEs with the first-order formulas of Theorems 8.12, 8.17 and 8.20. Where does the approximation break down? Try $n=8$.
- Compute the sample size each method needs for a $2\%$ relative standard error, and translate it into crop-cutting cost.
Tools: Python (numpy) or R; any plotting library.
Hand in: A table of bias and MSE for the four estimators in the three scenarios, a plot of simulated versus first-order MSE, and a recommendation for the agency.
Going further. Read about the generalised regression (GREG) estimator and calibration estimators, which extend $\bar y_{lr}$ to several auxiliary variables and arbitrary designs, and about small-area estimation, where the same idea borrows strength across districts.
🧪 Self-check quiz
Q1. Under PPS sampling with replacement ($n$ draws, normed sizes $p_i$), the Hansen–Hurwitz estimator of the total is
Q2. In Lahiri's method a pair $(i,R)$ with $R$ uniform on $\{1,\dots,M\}$ leads to selection of unit $i$ when
Q3. The Horvitz–Thompson estimator $\sum_{i\in s}y_i/\pi_i$ is unbiased for the total
Q4. To first order, the ratio estimator $\bar y_R$ is more efficient than $\bar y$ if and only if
Q5. The minimum over $b_0$ of $\operatorname{Var}\big(\bar y+b_0(\bar X-\bar x)\big)$ under SRSWOR equals