1.4 Joint, Marginal and Conditional Distributions
Several random variables at once: joint, marginal and conditional distributions, independence, covariance and correlation, conditional expectation with the tower property and the law of total variance, Jacobian transformations, convolutions, maxima and minima, and the bivariate normal distribution.
Interesting questions involve more than one random quantity. A doctor wants the blood pressure of a patient given the patient's age; an engineer wants the lifetime of a system built from several components; an actuary wants the total of a random number of random claims; a geneticist wants to know how strongly the heights of parents and children are related. Each needs the joint distribution of several random variables, which records not just how each one behaves but how they behave together. From it we recover each variable's own (marginal) law, decide whether the variables are independent, measure linear dependence by covariance and correlation, and predict one variable from another by conditional expectation.
This section builds the toolkit and proves its central rules: the factorisation criterion for independence, the tower property $E[E[X\mid Y]]=E[X]$, the law of total variance, the bound $|\rho|\le1$, the Jacobian formula for transformations, the convolution formula for sums, and the product formulas for maxima and minima. It ends with the bivariate normal distribution, the most important joint law in statistics, whose conditional means are straight lines (the regression lines of §1.1) and for which, uniquely, uncorrelated means independent. CSIR examiners love this material. Part B asks for a marginal density, a conditional mean, a covariance or the law of a maximum; Part C asks whether "uncorrelated implies independent", whether normal marginals make a normal pair, or whether a product-form density on a triangle gives independence.
Joint Distributions, Marginals and Independence
- $F$ is nondecreasing and right-continuous in each argument.
- $F(x,y)\to0$ as $x\to-\infty$ or as $y\to-\infty$; and $F(x,y)\to1$ as $x,y\to\infty$ together.
- (Marginals) $F_X(x)=\lim_{y\to\infty}F(x,y)$ and $F_Y(y)=\lim_{x\to\infty}F(x,y)$.
- (Rectangle rule) For $a\lt b$ and $c\lt d$, $$P(a\lt X\le b,\ c\lt Y\le d)=F(b,d)-F(a,d)-F(b,c)+F(a,c)\ge0.$$
- (Fréchet bounds) $\max\{F_X(x)+F_Y(y)-1,0\}\le F(x,y)\le\min\{F_X(x),F_Y(y)\}$.
Conversely (statement), every function with properties 1, 2 and 4 is the joint cdf of some random vector. Property 4 does not follow from 1 and 2.
Proof. (1)–(3) are proved exactly as Theorem 1.23, using continuity of $P$. For (3), the events $\{X\le x,\ Y\le n\}$ increase to $\{X\le x\}$ as $n\to\infty$.
(4) Let $A=\{X\le b,\ Y\le d\}$. Remove from $A$ the events $B=\{X\le a,\ Y\le d\}$ and $C=\{X\le b,\ Y\le c\}$, which overlap in $B\cap C=\{X\le a,\ Y\le c\}$. What is left is the rectangle event, so its probability is $P(A)-P(B)-P(C)+P(B\cap C)$, by inclusion–exclusion for $B\cup C\subseteq A$.
(5) $\{X\le x,\ Y\le y\}$ lies inside both $\{X\le x\}$ and $\{Y\le y\}$, which gives the upper bound. The lower bound is $P(A\cap B)\ge P(A)+P(B)-1$ (§1.2, Exercise I.3). ∎
To see that property 4 is a genuine extra condition, let $F(x,y)=1$ if $x+y\ge0$ and $F(x,y)=0$ otherwise. This $F$ is nondecreasing and right-continuous in each variable and has the right limits. But the rectangle rule on $(-1,1]\times(-1,1]$ gives $F(1,1)-F(-1,1)-F(1,-1)+F(-1,-1)$ $=1-1-1+0=-1$, a negative "probability".
(a) $(X,Y)$ is jointly discrete with joint pmf $p(x,y)=P(X=x,\ Y=y)$ if $\sum_{x,y}p(x,y)=1$ over a countable set. The marginal pmfs are $p_X(x)=\sum_yp(x,y)$ and $p_Y(y)=\sum_xp(x,y)$.
(b) $(X,Y)$ is jointly continuous with joint density $f\ge0$ if $$P((X,Y)\in B)=\iint_Bf(x,y)\,dx\,dy$$ for every Borel set $B\subseteq\mathbb{R}^2$ (it suffices to require this for $B=(-\infty,x]\times(-\infty,y]$). Then $f=\frac{\partial^2F}{\partial x\,\partial y}$ wherever $f$ is continuous, and the marginal densities are $$f_X(x)=\int_{-\infty}^\infty f(x,y)\,dy,\qquad f_Y(y)=\int_{-\infty}^\infty f(x,y)\,dx.$$
The marginal formula is Fubini's theorem: $P(X\le x)=\int_{-\infty}^x\left(\int f(s,y)\,dy\right)ds$. Note that two continuous random variables need not be jointly continuous: $(X,X)$ lives on the diagonal, a set of area zero, so it has no joint density although $X$ has one.
- $X$ and $Y$ are independent iff $F(x,y)=F_X(x)F_Y(y)$ for all $x,y$.
- Jointly discrete $X,Y$ are independent iff $p(x,y)=p_X(x)p_Y(y)$ for all $x,y$.
- Jointly continuous $X,Y$ are independent iff $f(x,y)=f_X(x)f_Y(y)$ for (almost) all $x,y$. (Factorisation) This holds iff $f(x,y)=g(x)h(y)$ for some functions $g,h\ge0$, for all $(x,y)\in\mathbb{R}^2$. In particular, the set where $f\gt0$ must then be a product set $A\times B$.
- If $X$ and $Y$ are independent, so are $\phi(X)$ and $\psi(Y)$ for Borel functions $\phi,\psi$, and $$E[\phi(X)\psi(Y)]=E[\phi(X)]\,E[\psi(Y)]$$ whenever both expectations on the right are finite.
Proof. (1) "Only if": take $B_1=(-\infty,x]$, $B_2=(-\infty,y]$. "If" (sketch): by the rectangle rule, a product formula for $F$ gives a product formula for $P(X\in I,\ Y\in J)$ for half-open intervals $I,J$, hence for finite disjoint unions of them. The two probability measures $B\mapsto P((X,Y)\in B)$ and $B\mapsto(P_X\otimes P_Y)(B)$ therefore agree on a class of sets that is closed under intersection and generates $\mathcal{B}(\mathbb{R}^2)$, and by the uniqueness theorem for measures (Dynkin's π–λ theorem) they agree everywhere.
(2) "Only if" is the definition with singletons. "If": $P(X\in A,\ Y\in B)=\sum_{x\in A}\sum_{y\in B}p_X(x)p_Y(y)$ $=P(X\in A)P(Y\in B)$.
(3) If $f=f_Xf_Y$, then $F(x,y)=\int_{-\infty}^x\int_{-\infty}^yf_X(s)f_Y(t)\,dt\,ds=F_X(x)F_Y(y)$, and (1) applies. Conversely, independence gives $F=F_XF_Y=\int\int f_X(s)f_Y(t)$, so $f_Xf_Y$ is a joint density, and densities are unique up to sets of area zero. For the factorisation criterion, suppose $f=g(x)h(y)$ everywhere. Let $G=\int g$ and $H=\int h$. Then $1=\iint f=GH$, $f_X(x)=g(x)H$ and $f_Y(y)=h(y)G$, so $$f_X(x)f_Y(y)=g(x)h(y)\,GH=f(x,y).$$ If $f=g(x)h(y)$, then $\{f\gt0\}=\{g\gt0\}\times\{h\gt0\}$, a product set.
(4) $\{\phi(X)\in A\}=\{X\in\phi^{-1}(A)\}$, so the defining product rule for $X,Y$ gives it for $\phi(X),\psi(Y)$. For the expectation, in the jointly continuous case and for $\phi,\psi\ge0$, Tonelli's theorem gives $$\iint\phi(x)\psi(y)f_X(x)f_Y(y)\,dx\,dy=\int\phi f_X\cdot\int\psi f_Y.$$ The general case follows by splitting into positive and negative parts, and the discrete case is the same computation with double sums. ∎
Expectations, Covariance and Correlation
Proof. This is the two-dimensional law of the unconscious statistician, and the proof of Theorem 1.28(1) applies word for word: for $g\ge0$, $$E[g(X,Y)]=\int_0^\infty P(g(X,Y)\gt t)\,dt=\iint g\,f,$$ by Tonelli. Linearity follows by taking $g(x,y)=x+y$ and splitting the integral. ∎
Let all variables have finite variances.
- $\operatorname{Cov}(X,Y)=E[XY]-E[X]E[Y]$; covariance is symmetric and bilinear, $\operatorname{Cov}(X,X)=\operatorname{Var}(X)$, and $\operatorname{Cov}(aX+b,\ cY+d)=ac\operatorname{Cov}(X,Y)$. Hence $\rho(aX+b,\ cY+d)=\operatorname{sign}(ac)\,\rho(X,Y)$ for $ac\ne0$.
- $\operatorname{Var}\left(\sum_ia_iX_i\right)=\sum_ia_i^2\operatorname{Var}(X_i)+2\sum_{i\lt j}a_ia_j\operatorname{Cov}(X_i,X_j)$. Independent variables are uncorrelated, so for them the variance of a sum is the sum of the variances.
- $|\rho(X,Y)|\le1$, with $|\rho|=1$ iff $Y=a+bX$ almost surely for constants with $b\ne0$; then $\operatorname{sign}(b)=\rho$.
- (Best linear predictor) $E[(Y-a-bX)^2]$ is minimised by $b^*=\frac{\operatorname{Cov}(X,Y)}{\operatorname{Var}(X)}$ and $a^*=\mu_Y-b^*\mu_X$, and the minimum is $\sigma_Y^2(1-\rho^2)$.
Proof. (1) Expand $(X-\mu_X)(Y-\mu_Y)$ and use linearity. Bilinearity is linearity of $E$; shifts cancel in $X-\mu_X$.
(2) $\operatorname{Var}(\sum a_iX_i)=\operatorname{Cov}(\sum_ia_iX_i,\sum_ja_jX_j)$; expand by bilinearity. If $X,Y$ are independent, Theorem 1.34(4) gives $E[XY]=E[X]E[Y]$.
(3) Let $U=(X-\mu_X)/\sigma_X$ and $V=(Y-\mu_Y)/\sigma_Y$, so $E[U^2]=E[V^2]=1$ and $E[UV]=\rho$. Then $$0\le E[(V-\rho U)^2]=1-2\rho^2+\rho^2=1-\rho^2.$$ So $\rho^2\le1$, and $\rho^2=1$ iff $E[(V-\rho U)^2]=0$, i.e. $V=\rho U$ almost surely (Theorem 1.28(4)), which says that $Y$ is an affine function of $X$ with slope of the sign of $\rho$. Conversely, if $Y=a+bX$ then $V=\operatorname{sign}(b)U$ and $\rho=\operatorname{sign}(b)$.
(4) For fixed $b$, the best $a$ is $E[Y-bX]$ (Theorem 1.29(2)), and then the error is $\operatorname{Var}(Y-bX)=\sigma_Y^2-2b\operatorname{Cov}+b^2\sigma_X^2$. This quadratic in $b$ is minimised at $b^*=\operatorname{Cov}/\sigma_X^2$, with value $\sigma_Y^2-\operatorname{Cov}^2/\sigma_X^2=\sigma_Y^2(1-\rho^2)$. ∎
These are the population versions of the results on $r$ and the least-squares line in §1.1. Correlation measures only linear dependence: it is invariant under changes of origin and scale (up to sign), it vanishes for independent variables, but it can vanish for strongly dependent ones too.
Conditional Distributions and Conditional Expectation
(a) Discrete. For $p_Y(y)\gt0$, the conditional pmf of $X$ given $Y=y$ is $p_{X\mid Y}(x\mid y)=\dfrac{p(x,y)}{p_Y(y)}$.
(b) Continuous. For $f_Y(y)\gt0$, the conditional density of $X$ given $Y=y$ is $f_{X\mid Y}(x\mid y)=\dfrac{f(x,y)}{f_Y(y)}$.
(c) The conditional expectation of $X$ given $Y=y$ is the mean of this conditional law, $$m(y)=E[X\mid Y=y]=\sum_xx\,p_{X\mid Y}(x\mid y)\quad\text{or}\quad\int x\,f_{X\mid Y}(x\mid y)\,dx.$$ The random variable $E[X\mid Y]=m(Y)$ is the conditional expectation of $X$ given $Y$. Similarly $\operatorname{Var}(X\mid Y)=E[X^2\mid Y]-\left(E[X\mid Y]\right)^2$.
The conditional density is a density in $x$ for each fixed $y$: it is nonnegative and integrates to $f_Y(y)/f_Y(y)=1$. It is the natural limit of $P(X\le x\mid y\le Y\le y+h)$ as $h\downarrow0$, but beware: conditioning on an event of probability zero depends on how that event is approached (pitfall 9 below). Conditional expectation is the right object for prediction: the best guess of $X$ when $Y$ is observed.
Let $E|X|\lt\infty$, and let $(X,Y)$ be jointly discrete or jointly continuous.
- (Tower property, or law of total expectation) $E\left[E[X\mid Y]\right]=E[X]$.
- (Taking out what is known) $E[\psi(Y)X\mid Y]=\psi(Y)E[X\mid Y]$, and hence $E[\psi(Y)X]=E[\psi(Y)E[X\mid Y]]$, when $\psi(Y)X$ is integrable.
- If $X$ and $Y$ are independent, $E[X\mid Y]=E[X]$.
- (Best predictor) If $E[X^2]\lt\infty$, then for every function $h$ with $E[h(Y)^2]\lt\infty$, $$E[(X-h(Y))^2]=E[(X-m(Y))^2]+E[(m(Y)-h(Y))^2].$$ So $m(Y)=E[X\mid Y]$ minimises the mean squared prediction error among all functions of $Y$.
Proof. We give the continuous case; the discrete one replaces integrals by sums. (1) By Fubini (justified by $E|X|\lt\infty$), $$E[m(Y)]=\int\left(\int x\frac{f(x,y)}{f_Y(y)}\,dx\right)f_Y(y)\,dy=\iint x\,f(x,y)\,dx\,dy=E[X].$$ (2) Given $Y=y$, $\psi(Y)=\psi(y)$ is a constant: $\int\psi(y)x\,f_{X\mid Y}(x\mid y)\,dx=\psi(y)m(y)$. Taking expectations and using (1) gives the second statement. (3) Independence gives $f(x,y)=f_X(x)f_Y(y)$, so $f_{X\mid Y}(x\mid y)=f_X(x)$ and $m(y)=E[X]$. (4) Write $X-h(Y)=(X-m(Y))+(m(Y)-h(Y))$ and expand the square. The cross term is $$2E\left[(m(Y)-h(Y))(X-m(Y))\right]=2E\left[(m(Y)-h(Y))\,E[X-m(Y)\mid Y]\right]=0,$$ by (2) with $\psi=m-h$, because $E[X-m(Y)\mid Y]=m(Y)-m(Y)=0$. ∎
Proof. Put $m=m(Y)=E[X\mid Y]$. By the tower property applied to $X^2$, $E[\operatorname{Var}(X\mid Y)]=E[E[X^2\mid Y]]-E[m^2]$ $=E[X^2]-E[m^2]$. Also $E[m]=E[X]$, so $\operatorname{Var}(m)=E[m^2]-(E[X])^2$. Adding the two lines gives $E[X^2]-(E[X])^2=\operatorname{Var}(X)$. ∎
This is the probabilistic form of the "within plus between" decomposition for grouped data (Theorem 1.6 of §1.1), and the seed of analysis of variance (§6.2). The inequality $\operatorname{Var}(E[X\mid Y])\le\operatorname{Var}(X)$ drives the Rao–Blackwell theorem (§4.2): averaging over what you do not need never increases variance.
Transformations: Jacobians, Sums, Maxima and Minima
Let $(X,Y)$ have joint density $f$ vanishing outside an open set $D\subseteq\mathbb{R}^2$. Let $g=(g_1,g_2)$ be a one-to-one map of $D$ onto an open set $R$ whose inverse $(u,v)\mapsto(x(u,v),y(u,v))$ is continuously differentiable on $R$, with Jacobian $$J(u,v)=\det\begin{pmatrix}\partial x/\partial u&\partial x/\partial v\\ \partial y/\partial u&\partial y/\partial v\end{pmatrix}\ne0.$$ Then $(U,V)=g(X,Y)$ has joint density $$f_{U,V}(u,v)=f\big(x(u,v),y(u,v)\big)\,|J(u,v)|\quad\text{on }R,$$ and $0$ outside $R$. If $g$ is not one-to-one but $D$ splits into pieces on which it is, add one such term per piece.
Proof. For a Borel set $B\subseteq R$, $P((U,V)\in B)=P((X,Y)\in g^{-1}(B))$ $=\iint_{g^{-1}(B)}f(x,y)\,dx\,dy$. The change-of-variables theorem for double integrals, with $(x,y)=(x(u,v),y(u,v))$, turns this into $\iint_Bf(x(u,v),y(u,v))\,|J(u,v)|\,du\,dv$. Since this holds for every $B$, the integrand is a density of $(U,V)$. ∎
To find the law of a single function $U=g_1(X,Y)$, invent a convenient companion $V$ (often $V=X$ or $V=Y$), apply the theorem, and integrate $v$ out. This is how the next theorem is proved.
Let $X$ and $Y$ be independent.
- If they take integer values, $P(X+Y=n)=\sum_kP(X=k)\,P(Y=n-k)$.
- If they have densities, $X+Y$ has density $$f_{X+Y}(z)=\int_{-\infty}^\infty f_X(x)\,f_Y(z-x)\,dx.$$
- $G_{X+Y}=G_XG_Y$ for nonnegative integer variables, and $M_{X+Y}(t)=M_X(t)M_Y(t)$ wherever both mgfs are finite.
Proof. (1) Split $\{X+Y=n\}$ according to the value $k$ of $X$, and use independence. (2) Take $U=X+Y$, $V=X$, a linear bijection of $\mathbb{R}^2$ with inverse $x=v$, $y=u-v$ and $|J|=1$. Theorem 1.39 gives $f_{U,V}(u,v)=f_X(v)f_Y(u-v)$; integrate out $v$. (3) $s^{X+Y}=s^Xs^Y$ and $e^{t(X+Y)}=e^{tX}e^{tY}$ are products of functions of $X$ and of $Y$, so Theorem 1.34(4) applies. ∎
Together with the uniqueness theorems for pgfs and mgfs, (3) is the quickest way to identify the law of a sum. For example, if $X$ and $Y$ are independent Poisson variables with means $\lambda$ and $\mu$, then $G_{X+Y}(s)=e^{\lambda(s-1)}e^{\mu(s-1)}=e^{(\lambda+\mu)(s-1)}$, so $X+Y$ is Poisson with mean $\lambda+\mu$ (Exercise IV.3).
Let $X_1,\dots,X_n$ be independent with cdfs $F_1,\dots,F_n$, and let $M=\max_iX_i$, $L=\min_iX_i$. Then $$F_M(x)=\prod_{i=1}^nF_i(x),\qquad P(L\gt x)=\prod_{i=1}^n\left(1-F_i(x)\right).$$ If the $X_i$ are iid with cdf $F$ and density $f$, then $f_M=nF^{n-1}f$ and $f_L=n(1-F)^{n-1}f$.
(Competing exponentials) If $X_i$ is exponential with rate $\lambda_i$, then $L$ is exponential with rate $\lambda=\sum_i\lambda_i$, and $P(L=X_j)=\lambda_j/\lambda$.
Proof. $\{M\le x\}=\bigcap_i\{X_i\le x\}$ and $\{L\gt x\}=\bigcap_i\{X_i\gt x\}$; apply independence. Differentiate for the iid densities. For exponentials, $P(L\gt x)=\prod e^{-\lambda_ix}=e^{-\lambda x}$. Finally $X_j$ is the minimum iff the others exceed it, so $$P(L=X_j)=\int_0^\infty\lambda_je^{-\lambda_jx}\prod_{i\ne j}e^{-\lambda_ix}\,dx=\int_0^\infty\lambda_je^{-\lambda x}\,dx=\frac{\lambda_j}{\lambda}.$$ (Ties have probability $0$.) ∎
The Bivariate Normal Distribution
Recall that $Z$ is standard normal if it has density $\varphi(z)=\frac{1}{\sqrt{2\pi}}e^{-z^2/2}$, and that $X=\mu+\sigma Z$ ($\sigma\gt0$) is normal $N(\mu,\sigma^2)$, with density $\frac{1}{\sigma}\varphi\left(\frac{x-\mu}{\sigma}\right)$, mean $\mu$ and variance $\sigma^2$. Completing the square, $\int e^{tz}\varphi(z)\,dz=e^{t^2/2}\int\varphi(z-t)\,dz$, so $M_Z(t)=e^{t^2/2}$ and $M_X(t)=e^{\mu t+\sigma^2t^2/2}$. (The full theory of the normal law is in §2.1.)
Let $(X,Y)\sim\mathrm{BVN}(\mu_1,\mu_2,\sigma_1^2,\sigma_2^2,\rho)$.
- (Marginals) $X\sim N(\mu_1,\sigma_1^2)$ and $Y\sim N(\mu_2,\sigma_2^2)$.
- (Conditionals) Given $X=x$, $$Y\sim N\left(\mu_2+\rho\frac{\sigma_2}{\sigma_1}(x-\mu_1),\ \sigma_2^2(1-\rho^2)\right).$$ The regression of $Y$ on $X$ is linear, and the conditional variance does not depend on $x$.
- (Correlation) $\rho(X,Y)=\rho$, and $X,Y$ are independent iff $\rho=0$.
- (Construction) If $Z_1,Z_2$ are iid $N(0,1)$, then $X=\mu_1+\sigma_1Z_1$, $Y=\mu_2+\sigma_2\left(\rho Z_1+\sqrt{1-\rho^2}\,Z_2\right)$ has this distribution.
- (Linear combinations) For $(a,b)\ne(0,0)$, $aX+bY$ is normal with mean $a\mu_1+b\mu_2$ and variance $a^2\sigma_1^2+2ab\rho\sigma_1\sigma_2+b^2\sigma_2^2$.
Proof. (1), (2) Complete the square in $z_2$: $$z_1^2-2\rho z_1z_2+z_2^2=(z_2-\rho z_1)^2+(1-\rho^2)z_1^2.$$ Hence $f(x,y)$ is the product of $$\frac{1}{\sigma_1\sqrt{2\pi}}e^{-z_1^2/2}$$ and $$\frac{1}{\sigma_2\sqrt{1-\rho^2}\sqrt{2\pi}}\exp\left(-\frac{(z_2-\rho z_1)^2}{2(1-\rho^2)}\right).$$ The first factor is the $N(\mu_1,\sigma_1^2)$ density in $x$. For fixed $x$, since $z_2-\rho z_1=\frac{y-\mu_2-\rho\sigma_2z_1}{\sigma_2}$, the second factor is the density in $y$ of $N(\mu_2+\rho\sigma_2z_1,\ \sigma_2^2(1-\rho^2))$, so it integrates to $1$ over $y$. Integrating $f$ over $y$ leaves the first factor, which is (1) for $X$ (and $Y$ follows by symmetry). Dividing $f$ by $f_X$ leaves the second factor, which is (2).
(3) Using (2) and the tower property, $$\operatorname{Cov}(X,Y)=E\left[(X-\mu_1)\,E[Y-\mu_2\mid X]\right]=E\left[(X-\mu_1)\,\rho\tfrac{\sigma_2}{\sigma_1}(X-\mu_1)\right]=\rho\sigma_1\sigma_2.$$ If $\rho=0$ the exponent splits as $-\tfrac12(z_1^2+z_2^2)$ and $f=f_Xf_Y$; conversely independent variables are uncorrelated.
(4) The map $(z_1,z_2)\mapsto(x,y)$ is linear with inverse $z_1=\frac{x-\mu_1}{\sigma_1}$, $z_2=\frac{(y-\mu_2)/\sigma_2-\rho z_1}{\sqrt{1-\rho^2}}$ and Jacobian $\frac{1}{\sigma_1\sigma_2\sqrt{1-\rho^2}}$. The joint density of $(Z_1,Z_2)$ is $\frac{1}{2\pi}e^{-(z_1^2+z_2^2)/2}$, and substituting gives exactly the exponent $z_1^2+\frac{(z_2-\rho z_1)^2}{1-\rho^2}$ of (1)–(2) (with $z_2$ there meaning $\frac{y-\mu_2}{\sigma_2}$). Theorem 1.39 completes the proof.
(5) By (4), $aX+bY=c+\alpha Z_1+\beta Z_2$ with $c=a\mu_1+b\mu_2$, $\alpha=a\sigma_1+b\rho\sigma_2$, $\beta=b\sigma_2\sqrt{1-\rho^2}$, not both zero. By Theorem 1.40(3), its mgf is $$e^{ct}e^{\alpha^2t^2/2}e^{\beta^2t^2/2}=e^{ct+(\alpha^2+\beta^2)t^2/2},$$ the mgf of $N(c,\alpha^2+\beta^2)$. By uniqueness (Theorem 1.32(5)), $aX+bY$ has this law, and $\alpha^2+\beta^2$ expands to the stated variance. ∎
The slope $\rho\sigma_2/\sigma_1$ in (2) is the population regression coefficient, and $\sigma_2^2(1-\rho^2)$ is the minimum mean squared error of Theorem 1.36(4). For the bivariate normal the best predictor (Theorem 1.37(4)) is linear, so the best linear predictor is the best predictor of all. Because $|\rho|\lt1$, the predicted $Y$ lies fewer standard deviations from its mean than the observed $X$: this is Galton's regression to the mean.
- Marginals do not determine the joint law. With $U$ uniform on $(0,1)$, the pairs $(U,U)$, $(U,1-U)$ and $(U,U')$ with $U'$ an independent copy all have uniform marginals. They attain the upper Fréchet bound, the lower one, and the product $xy$ respectively (Exercise I.7).
- Each variable continuous, but no joint density. $(X,X)$ is concentrated on a line of area $0$.
- A product formula on a non-product support is not independence. $f(x,y)=8xy$ on $0\lt x\lt y\lt1$ "factorises", but the support is a triangle, so $X$ and $Y$ are dependent (Exercise I.8). The factorisation must hold on all of $\mathbb{R}^2$, indicator functions included.
- Uncorrelated does not imply independent. $X$ uniform on $(-1,1)$ and $Y=X^2$ (Example 2(b)); or $(X,Y)$ uniform on the unit disc (Exercise I.6). Likewise $E[XY]=E[X]E[Y]$ does not imply independence.
- Normal marginals and zero correlation do not make an independent pair. Let $X\sim N(0,1)$, $W=\pm1$ with probability $\tfrac12$ each, independent of $X$, and $Y=WX$. Then $Y\sim N(0,1)$ and $\operatorname{Cov}(X,Y)=E[W]E[X^2]=0$, but $|Y|=|X|$. And $(X,Y)$ is not bivariate normal: $P(X+Y=0)=\tfrac12$ (Exercise V.6). Theorem 1.42(3) needs joint normality.
- A function that is monotone in each variable need not be a joint cdf. $F(x,y)=\mathbf{1}\{x+y\ge0\}$ violates the rectangle rule (after Theorem 1.33).
- $E[X\mid Y]$ is a random variable, the function $m(Y)$, not a number. Only $E[X\mid Y=y]$ is a number, and only $E[E[X\mid Y]]$ equals $E[X]$.
- The variance of a difference. $\operatorname{Var}(X-Y)=\operatorname{Var}X+\operatorname{Var}Y-2\operatorname{Cov}(X,Y)$. For independent variables it is the sum of the variances, never their difference.
- Conditioning on a null event is ambiguous (Borel–Kolmogorov). For $X,Y$ iid exponential(1), "given $X=Y$" computed via $D=X-Y=0$ gives the conditional density $2e^{-2x}$ (mean $\tfrac12$), but via $R=X/Y=1$ it gives $4xe^{-2x}$ (mean $1$). A conditional density is attached to a conditioning variable, not to an event of probability zero.
- The maximum and minimum of iid variables are dependent. Always $L\le M$; for two uniforms $P(L\gt\tfrac12,\ M\lt\tfrac12)=0$ though both factors are positive.
- The two regression lines are different. $E[Y\mid X=x]$ has slope $\rho\sigma_2/\sigma_1$, while solving $E[X\mid Y=y]$ for $y$ gives slope $\sigma_2/(\rho\sigma_1)$. They coincide only when $|\rho|=1$, and they are perpendicular (the two axes) when $\rho=0$ (Interactive 2).
- Pairwise independence of random variables is not mutual independence, exactly as for events (§1.2): two fair $\pm1$ coins $W_1,W_2$ and their product $W_1W_2$ are pairwise but not mutually independent.
Exercises
- 1 $f(x,y)=c(x^2+y)$ on $(0,1)^2$. Find $c$, both marginal densities and $P(X\gt Y)$.
- 2 $f(x,y)=e^{-y}$ for $0\lt x\lt y$. Find the marginal densities, decide independence and find $P(Y\gt2X)$.
- 3 $p(x,y)=c(x+y)$ for $x,y\in\{0,1,2\}$. Find $c$ and the marginal pmfs, and decide whether $X$ and $Y$ are independent.
- 4 Show that $F(x,y)=1-e^{-x-y}$ for $x,y\ge0$ (and $0$ otherwise) is not a joint distribution function.
- 5 Prove that a random variable independent of itself is almost surely constant.
- 6 $(X,Y)$ is uniform on the unit disc. Find $f_X$, show that $\operatorname{Cov}(X,Y)=0$, and show that $X$ and $Y$ are not independent.
- 7 $U$ is uniform on $(0,1)$. Compute the joint cdfs of $(U,U)$ and of $(U,1-U)$, and show that they attain the upper and lower Fréchet bounds.
- 8 $f(x,y)=8xy$ for $0\lt x\lt y\lt1$. Check that $f$ is a density, find the marginals, and explain why $X$ and $Y$ are dependent although $f$ is a product $g(x)h(y)$ on its support.
- 1 For $f(x,y)=2$ on $0\lt x\lt y\lt1$, find $E[X]$, $E[Y]$, the variances, $\operatorname{Cov}(X,Y)$ and $\rho$.
- 2 Show that if $\operatorname{Var}(X)=\operatorname{Var}(Y)$, then $X+Y$ and $X-Y$ are uncorrelated.
- 3 $X_1,\dots,X_n$ have common variance $\sigma^2\gt0$ and common pairwise correlation $\rho$. Find $\operatorname{Var}(X_1+\cdots+X_n)$ and deduce that $\rho\ge-\frac{1}{n-1}$.
- 4 For the distribution of II.1, find the best linear predictor $a+bX$ of $Y$, and compare it with $E[Y\mid X]$.
- 5 Give two random variables with $\rho=0$ such that one is a function of the other, other than Example 2(b).
- 6 Show that two indicator variables $\mathbf{1}_A$ and $\mathbf{1}_B$ are uncorrelated iff the events $A$ and $B$ are independent.
- 1 For $f(x,y)=e^{-y}$, $0\lt x\lt y$, find $f_{Y\mid X}(y\mid x)$, $E[Y\mid X]$, $f_{X\mid Y}(x\mid y)$ and $E[X\mid Y]$, and check the tower property both ways.
- 2 $Y$ is uniform on $(0,1)$, and given $Y=y$, $X$ is uniform on $(0,y)$. Find $f_X$, $E[X]$ and $\operatorname{Var}(X)$, the last both directly and by the law of total variance.
- 3 (Insurance) The number $N$ of claims in a year has mean $50$ and variance $50$. Claim sizes are iid with mean $\text{₹}\,4000$ and SD $\text{₹}\,3000$, independent of $N$. Find the mean and SD of the total $S=X_1+\cdots+X_N$.
- 4 Give an example where $E[X\mid Y]=E[X]$ but $X$ and $Y$ are not independent.
- 5 A fair die is rolled, and a fair coin is then tossed as many times as the die shows. Let $X$ be the number of heads. Find $E[X]$, $\operatorname{Var}(X)$ and $P(X=0)$.
- 6 Prove that $\operatorname{Var}(E[X\mid Y])=\operatorname{Var}(X)$ iff $X$ is almost surely a function of $Y$ (in the discrete case: $\operatorname{Var}(X\mid Y=y)=0$ for every $y$ with $P(Y=y)\gt0$).
- 7 Given $\Lambda=\lambda$, $X$ is Poisson with mean $\lambda$, and $\Lambda$ is exponential with rate $1$. Find the pmf, mean and variance of $X$.
- 8 In Example 4 with $n=10$, all ten tosses are heads. Find the posterior mean of $Y$ and the probability that the eleventh toss is a head.
- 1 $X,Y$ iid uniform on $(0,1)$. Find $P(X+Y\le\tfrac12)$ and $P(\tfrac12\le X+Y\le\tfrac32)$.
- 2 $X,Y$ iid exponential with rate $\lambda$. Show that $X-Y$ has the Laplace density $\tfrac\lambda2e^{-\lambda|z|}$.
- 3 $X,Y$ independent Poisson with means $\lambda,\mu$. Show that $X+Y$ is Poisson with mean $\lambda+\mu$ and that, given $X+Y=n$, $X$ is binomial with $n$ trials and success probability $\frac{\lambda}{\lambda+\mu}$.
- 4 $X,Y$ iid $N(0,1)$. Use the Jacobian formula with $V=Y$ to show that $X/Y$ has the Cauchy density $\frac{1}{\pi(1+z^2)}$.
- 5 $X,Y$ iid $N(0,1)$, and $(R,\Theta)$ are their polar coordinates. Show that $R$ and $\Theta$ are independent, $\Theta$ uniform on $(0,2\pi)$ and $R$ with density $re^{-r^2/2}$. Deduce the Box–Muller method: if $U_1,U_2$ are iid uniform, then $\sqrt{-2\log U_1}\cos(2\pi U_2)$ is $N(0,1)$.
- 6 Five bulbs have iid exponential lifetimes with mean $1000$ hours. Find the expected time to the first failure and to the last failure.
- 7 $X_1,\dots,X_n$ are iid uniform on $(0,\theta)$ and $M=\max X_i$. Find $E[M]$ and the smallest $n$ with $P(M\gt0.9\theta)\ge0.95$.
- 8 $X,Y$ iid exponential(1), $L=\min(X,Y)$, $M=\max(X,Y)$. Show that $L$ and $M-L$ are independent, with $L$ exponential(2) and $M-L$ exponential(1).
- 1 $(X,Y)$ is bivariate normal with $\mu_1=1$, $\mu_2=-1$, $\sigma_1=2$, $\sigma_2=3$, $\rho=0.4$. Find $E[Y\mid X=3]$, $\operatorname{Var}(Y\mid X=3)$ and $P(Y\gt0\mid X=3)$.
- 2 For the same pair, find the laws of $X+Y$ and $X-Y$, and decide whether they are independent.
- 3 For a standard bivariate normal pair ($\mu_i=0$, $\sigma_i=1$) with correlation $\rho$, show that $X+Y$ and $X-Y$ are independent.
- 4 For a standard bivariate normal pair with correlation $\rho$, prove that $P(X\gt0,\ Y\gt0)=\tfrac14+\frac{\arcsin\rho}{2\pi}$, and evaluate it for $\rho=\tfrac12$.
- 5 For $(X,Y)$ bivariate normal, find the constant $c$ such that $X-cY$ is independent of $Y$.
- 6 $X\sim N(0,1)$, $W=\pm1$ with probability $\tfrac12$ each independently of $X$, and $Y=WX$. Show that $Y\sim N(0,1)$ and $\operatorname{Cov}(X,Y)=0$, that $X$ and $Y$ are dependent, and that $(X,Y)$ is not bivariate normal.
E1. CSIR Part B $(X,Y)$ has joint density $f(x,y)=6x$ for $0\lt x\lt y\lt1$. Then $E\left[X\mid Y=\tfrac34\right]$ equals (A) $\tfrac12$; (B) $\tfrac38$; (C) $\tfrac23$; (D) $\tfrac14$.
E2. CSIR Part B GATE ST $X$ and $Y$ are iid exponential with mean $1$. Then $E[\max(X,Y)]$ equals (A) $1$; (B) $\tfrac32$; (C) $2$; (D) $\tfrac12$.
E3. CSIR Part B $(X,Y)$ is bivariate normal with $\sigma_X=2$, $\sigma_Y=4$ and $\rho=0.6$. Then $\operatorname{Var}(Y\mid X=x)$ is (A) $5.76$; (B) $9.6$; (C) $10.24$; (D) $16$.
E4. CSIR Part C Let $X$ and $Y$ have finite positive variances. Which of the following are true? (One or more options may be correct.) (A) If $\operatorname{Cov}(X,Y)=0$, then $X$ and $Y$ are independent. (B) If $X$ and $Y$ are independent, then $\operatorname{Var}(X-Y)=\operatorname{Var}(X)+\operatorname{Var}(Y)$. (C) If $(X,Y)$ is bivariate normal and $\operatorname{Cov}(X,Y)=0$, then $X$ and $Y$ are independent. (D) If $X$ and $Y$ are each normally distributed and uncorrelated, then $X+Y$ is normally distributed.
E5. CSIR Part C $Y$ is uniform on $(0,2)$ and, given $Y=y$, $X$ is uniform on $(0,y)$. Which of the following are true? (A) $E[X]=\tfrac12$. (B) $\operatorname{Var}(X)=\tfrac19$. (C) $X$ has density $\tfrac12\log\frac2x$ on $(0,2)$. (D) $X$ and $Y$ are independent.
E6. CSIR Part C GATE ST $X_1,X_2,X_3$ are iid uniform on $(0,1)$, $M=\max X_i$ and $L=\min X_i$. Which of the following are true? (A) $E[M]=\tfrac34$. (B) $P(L\gt\tfrac12)=\tfrac18$. (C) $M$ and $L$ are independent. (D) $X_1+X_2$ has density $\min(z,2-z)$ on $(0,2)$.
Suggested plan: 3 lectures × 60 min.
- Joint laws and covariance. Definition 1.24, Theorem 1.33 with the rectangle rule drawn as four overlapping quadrants, and the $\mathbf{1}\{x+y\ge0\}$ counterexample. Joint densities and marginals; independence and the factorisation criterion (Theorem 1.34), stressing the support. Example 1 on the board (let students guess (b) first). Covariance and correlation (Theorems 1.35–1.36) with the Cauchy–Schwarz proof of $|\rho|\le1$; Example 2.
- Conditioning. Definition 1.28, Example 3 as a table on the board. Tower property, taking out what is known, the best-predictor theorem (Theorem 1.37), and the law of total variance (Theorem 1.38). Example 4 (the uniform marginal is a lovely surprise; ask for predictions). Mention the random-sum formula of Exercise III.3, a frequent exam item.
- Transformations and the bivariate normal. The Jacobian theorem, Example 5, convolution and maxima/minima (Theorems 1.39–1.41), Example 6. Project Interactive 1: uniform sums with $n=1,2,3,6$, then exponential maxima. Definition 1.29 and Theorem 1.42 by completing the square; Example 7; Interactive 2, sweeping $\rho$ and moving the slice. Finish with E4 and E6 live.
Discussion questions. (1) A study finds that sons of very tall fathers are, on average, shorter than their fathers. Does this mean the population is becoming more uniform in height over generations? (2) Why must the factorisation criterion be applied to the density on all of $\mathbb{R}^2$, indicator functions included? (3) Is there a pair of random variables with $E[X\mid Y]=E[X]$ and $E[Y\mid X]=E[Y]$ that is not independent?
Misconceptions to watch for. Reading independence off a formula and ignoring the support; equating "uncorrelated" with "independent"; believing normal marginals imply joint normality; writing $\operatorname{Var}(X-Y)=\operatorname{Var}X-\operatorname{Var}Y$; forgetting $|J|$ or using the Jacobian of the forward map instead of the inverse; treating $E[X\mid Y]$ as a number; assuming the maximum and minimum of an iid sample are independent.
Suggested homework. I(1, 2, 6, 8), II(1, 3, 4), III(1, 2, 3, 5), IV(2, 4, 6, 8), V(1, 4, 6); Exam corner E1, E5, E6.
Research question. Patients enrolled in a trial because their blood pressure was high often improve at the next visit even without treatment. How much of an observed "improvement" is just regression to the mean?
Why it matters. Clinical trials of hypertension, cholesterol and diabetes drugs, sports "Sophomore slumps", speed cameras installed at accident black spots, and school remedial programmes for low scorers all select units because a first measurement was extreme. If repeated measurements are positively but imperfectly correlated, the second measurement drifts back towards the mean by itself. Regulators such as the WHO and national drug agencies insist on randomised control groups largely for this reason.
What to do.
- Model two systolic readings $(X_1,X_2)$ of the same untreated patient as bivariate normal with common mean $\mu=140$ mmHg, common SD $\sigma=15$ mmHg and correlation $\rho=0.7$.
- Patients are enrolled if $X_1\gt160$. Using Theorem 1.42(2) and the tower property, prove that $E[X_2\mid X_1\gt c]=\mu+\rho\,(E[X_1\mid X_1\gt c]-\mu)$, and that $E[X_1\mid X_1\gt c]=\mu+\sigma\lambda(z)$ with $z=\frac{c-\mu}{\sigma}$ and $\lambda(z)=\frac{\varphi(z)}{1-\Phi(z)}$. Deduce the expected "improvement" $(1-\rho)\sigma\lambda(z)$.
- Compute it for $\rho=0.5,0.7,0.9$, and check one case by simulating $10^5$ patients with a seeded generator (use the construction of Theorem 1.42(4); Interactive 2 shows the same geometry).
- Explain why a randomised control group, selected by the same rule, removes this bias from the estimated treatment effect.
Tools: Python (scipy for $\Phi$ and $\varphi$) or a spreadsheet; Interactive 2 on this page.
Hand in: A one-page memo for a trial committee: the formula, a small table of the expected spurious improvement for the three correlations, the simulation check, and a recommendation on trial design.
Going further. Read about Galton's 1886 study of the heights of parents and children, where the word "regression" was born, and about how regression to the mean is estimated and corrected in observational studies of road safety.
🧪 Self-check quiz
Q1. $(X,Y)$ has joint density $4xy$ on $(0,1)^2$. Then $X$ and $Y$ are
Q2. $\operatorname{Var}(X)=1$, $\operatorname{Var}(Y)=4$ and $\operatorname{Cov}(X,Y)=1$. Then $\rho(X,Y)$ is
Q3. For integrable $X$, the random variable $E[X\mid Y]$ always satisfies
Q4. $X$ and $Y$ are iid exponential with rate $\lambda$. Then $\min(X,Y)$ is
Q5. $(X,Y)$ is standard bivariate normal with $\rho=0.8$. Given $X=1$, $Y$ has