Chapter 1 · Probability

1.4 Joint, Marginal and Conditional Distributions

Several random variables at once: joint, marginal and conditional distributions, independence, covariance and correlation, conditional expectation with the tower property and the law of total variance, Jacobian transformations, convolutions, maxima and minima, and the bivariate normal distribution.

Interesting questions involve more than one random quantity. A doctor wants the blood pressure of a patient given the patient's age; an engineer wants the lifetime of a system built from several components; an actuary wants the total of a random number of random claims; a geneticist wants to know how strongly the heights of parents and children are related. Each needs the joint distribution of several random variables, which records not just how each one behaves but how they behave together. From it we recover each variable's own (marginal) law, decide whether the variables are independent, measure linear dependence by covariance and correlation, and predict one variable from another by conditional expectation.

This section builds the toolkit and proves its central rules: the factorisation criterion for independence, the tower property $E[E[X\mid Y]]=E[X]$, the law of total variance, the bound $|\rho|\le1$, the Jacobian formula for transformations, the convolution formula for sums, and the product formulas for maxima and minima. It ends with the bivariate normal distribution, the most important joint law in statistics, whose conditional means are straight lines (the regression lines of §1.1) and for which, uniquely, uncorrelated means independent. CSIR examiners love this material. Part B asks for a marginal density, a conditional mean, a covariance or the law of a maximum; Part C asks whether "uncorrelated implies independent", whether normal marginals make a normal pair, or whether a product-form density on a triangle gives independence.

Joint Distributions, Marginals and Independence

📘 Definition 1.24 — Random vectors and joint distribution functions
A random vector $(X_1,\dots,X_n)$ is a list of random variables on the same probability space. Its joint distribution function is $$F(x_1,\dots,x_n)=P(X_1\le x_1,\dots,X_n\le x_n).$$ For two variables we write $(X,Y)$ and $F(x,y)=P(X\le x,\ Y\le y)$. The marginal cdfs are those of the individual components.
📐 Theorem 1.33 — Properties of a bivariate distribution function
  1. $F$ is nondecreasing and right-continuous in each argument.
  2. $F(x,y)\to0$ as $x\to-\infty$ or as $y\to-\infty$; and $F(x,y)\to1$ as $x,y\to\infty$ together.
  3. (Marginals) $F_X(x)=\lim_{y\to\infty}F(x,y)$ and $F_Y(y)=\lim_{x\to\infty}F(x,y)$.
  4. (Rectangle rule) For $a\lt b$ and $c\lt d$, $$P(a\lt X\le b,\ c\lt Y\le d)=F(b,d)-F(a,d)-F(b,c)+F(a,c)\ge0.$$
  5. (Fréchet bounds) $\max\{F_X(x)+F_Y(y)-1,0\}\le F(x,y)\le\min\{F_X(x),F_Y(y)\}$.

Conversely (statement), every function with properties 1, 2 and 4 is the joint cdf of some random vector. Property 4 does not follow from 1 and 2.

Proof. (1)–(3) are proved exactly as Theorem 1.23, using continuity of $P$. For (3), the events $\{X\le x,\ Y\le n\}$ increase to $\{X\le x\}$ as $n\to\infty$.

(4) Let $A=\{X\le b,\ Y\le d\}$. Remove from $A$ the events $B=\{X\le a,\ Y\le d\}$ and $C=\{X\le b,\ Y\le c\}$, which overlap in $B\cap C=\{X\le a,\ Y\le c\}$. What is left is the rectangle event, so its probability is $P(A)-P(B)-P(C)+P(B\cap C)$, by inclusion–exclusion for $B\cup C\subseteq A$.

(5) $\{X\le x,\ Y\le y\}$ lies inside both $\{X\le x\}$ and $\{Y\le y\}$, which gives the upper bound. The lower bound is $P(A\cap B)\ge P(A)+P(B)-1$ (§1.2, Exercise I.3). ∎

To see that property 4 is a genuine extra condition, let $F(x,y)=1$ if $x+y\ge0$ and $F(x,y)=0$ otherwise. This $F$ is nondecreasing and right-continuous in each variable and has the right limits. But the rectangle rule on $(-1,1]\times(-1,1]$ gives $F(1,1)-F(-1,1)-F(1,-1)+F(-1,-1)$ $=1-1-1+0=-1$, a negative "probability".

📘 Definition 1.25 — Joint pmf, joint density and marginals

(a) $(X,Y)$ is jointly discrete with joint pmf $p(x,y)=P(X=x,\ Y=y)$ if $\sum_{x,y}p(x,y)=1$ over a countable set. The marginal pmfs are $p_X(x)=\sum_yp(x,y)$ and $p_Y(y)=\sum_xp(x,y)$.

(b) $(X,Y)$ is jointly continuous with joint density $f\ge0$ if $$P((X,Y)\in B)=\iint_Bf(x,y)\,dx\,dy$$ for every Borel set $B\subseteq\mathbb{R}^2$ (it suffices to require this for $B=(-\infty,x]\times(-\infty,y]$). Then $f=\frac{\partial^2F}{\partial x\,\partial y}$ wherever $f$ is continuous, and the marginal densities are $$f_X(x)=\int_{-\infty}^\infty f(x,y)\,dy,\qquad f_Y(y)=\int_{-\infty}^\infty f(x,y)\,dx.$$

The marginal formula is Fubini's theorem: $P(X\le x)=\int_{-\infty}^x\left(\int f(s,y)\,dy\right)ds$. Note that two continuous random variables need not be jointly continuous: $(X,X)$ lives on the diagonal, a set of area zero, so it has no joint density although $X$ has one.

📘 Definition 1.26 — Independent random variables
Random variables $X_1,\dots,X_n$ are independent if $$P(X_1\in B_1,\dots,X_n\in B_n)=\prod_{i=1}^nP(X_i\in B_i)$$ for all Borel sets $B_1,\dots,B_n$. An infinite family is independent if every finite subfamily is. iid means independent and identically distributed.
📐 Theorem 1.34 — Criteria for independence
  1. $X$ and $Y$ are independent iff $F(x,y)=F_X(x)F_Y(y)$ for all $x,y$.
  2. Jointly discrete $X,Y$ are independent iff $p(x,y)=p_X(x)p_Y(y)$ for all $x,y$.
  3. Jointly continuous $X,Y$ are independent iff $f(x,y)=f_X(x)f_Y(y)$ for (almost) all $x,y$. (Factorisation) This holds iff $f(x,y)=g(x)h(y)$ for some functions $g,h\ge0$, for all $(x,y)\in\mathbb{R}^2$. In particular, the set where $f\gt0$ must then be a product set $A\times B$.
  4. If $X$ and $Y$ are independent, so are $\phi(X)$ and $\psi(Y)$ for Borel functions $\phi,\psi$, and $$E[\phi(X)\psi(Y)]=E[\phi(X)]\,E[\psi(Y)]$$ whenever both expectations on the right are finite.

Proof. (1) "Only if": take $B_1=(-\infty,x]$, $B_2=(-\infty,y]$. "If" (sketch): by the rectangle rule, a product formula for $F$ gives a product formula for $P(X\in I,\ Y\in J)$ for half-open intervals $I,J$, hence for finite disjoint unions of them. The two probability measures $B\mapsto P((X,Y)\in B)$ and $B\mapsto(P_X\otimes P_Y)(B)$ therefore agree on a class of sets that is closed under intersection and generates $\mathcal{B}(\mathbb{R}^2)$, and by the uniqueness theorem for measures (Dynkin's π–λ theorem) they agree everywhere.

(2) "Only if" is the definition with singletons. "If": $P(X\in A,\ Y\in B)=\sum_{x\in A}\sum_{y\in B}p_X(x)p_Y(y)$ $=P(X\in A)P(Y\in B)$.

(3) If $f=f_Xf_Y$, then $F(x,y)=\int_{-\infty}^x\int_{-\infty}^yf_X(s)f_Y(t)\,dt\,ds=F_X(x)F_Y(y)$, and (1) applies. Conversely, independence gives $F=F_XF_Y=\int\int f_X(s)f_Y(t)$, so $f_Xf_Y$ is a joint density, and densities are unique up to sets of area zero. For the factorisation criterion, suppose $f=g(x)h(y)$ everywhere. Let $G=\int g$ and $H=\int h$. Then $1=\iint f=GH$, $f_X(x)=g(x)H$ and $f_Y(y)=h(y)G$, so $$f_X(x)f_Y(y)=g(x)h(y)\,GH=f(x,y).$$ If $f=g(x)h(y)$, then $\{f\gt0\}=\{g\gt0\}\times\{h\gt0\}$, a product set.

(4) $\{\phi(X)\in A\}=\{X\in\phi^{-1}(A)\}$, so the defining product rule for $X,Y$ gives it for $\phi(X),\psi(Y)$. For the expectation, in the jointly continuous case and for $\phi,\psi\ge0$, Tonelli's theorem gives $$\iint\phi(x)\psi(y)f_X(x)f_Y(y)\,dx\,dy=\int\phi f_X\cdot\int\psi f_Y.$$ The general case follows by splitting into positive and negative parts, and the discrete case is the same computation with double sums. ∎

✏️ Example 1 — Three joint densities on the unit square
(a) $f(x,y)=x+y$ on $(0,1)^2$. Check that $f$ is a density, find the marginals, decide independence, and find $P(X+Y\le1)$. (b) $f(x,y)=2$ on the triangle $0\lt x\lt y\lt1$. Find the marginals and decide independence. (c) $f(x,y)=c\,xy^2$ on $(0,1)^2$. Find $c$, decide independence, and find $P(X\lt Y)$.
(a) $\iint(x+y)=\tfrac12+\tfrac12=1$. $f_X(x)=\int_0^1(x+y)\,dy=x+\tfrac12$ and similarly $f_Y(y)=y+\tfrac12$. Since $(x+\tfrac12)(y+\tfrac12)\ne x+y$ (try $x=y=0$: $\tfrac14\ne0$), $X$ and $Y$ are dependent. Finally $$P(X+Y\le1)=\int_0^1\!\int_0^{1-x}(x+y)\,dy\,dx=\int_0^1\left[x(1-x)+\tfrac{(1-x)^2}{2}\right]dx=\tfrac16+\tfrac16=\tfrac13.$$
(b) $f_X(x)=\int_x^12\,dy=2(1-x)$ and $f_Y(y)=\int_0^y2\,dx=2y$. Then $f_X(x)f_Y(y)=4y(1-x)$, which is not $f$. They are dependent although $f$ is constant: the support is a triangle, not a product set. Knowing $Y=0.1$ forces $X\lt0.1$.
(c) $\iint xy^2=\tfrac12\cdot\tfrac13=\tfrac16$, so $c=6$. Here $f=(2x)(3y^2)$ on the product set $(0,1)^2$ (and $0$ outside, which is also a product: $f=g(x)h(y)$ with $g=2x\mathbf{1}_{(0,1)}$, $h=3y^2\mathbf{1}_{(0,1)}$). So $X$ and $Y$ are independent with densities $2x$ and $3y^2$, and $$P(X\lt Y)=\int_0^13y^2\left(\int_0^y2x\,dx\right)dy=\int_0^13y^4\,dy=\tfrac35.$$

Expectations, Covariance and Correlation

📐 Theorem 1.35 — Expectation of a function of a random vector
For a Borel function $g:\mathbb{R}^2\to\mathbb{R}$, $$E[g(X,Y)]=\sum_{x,y}g(x,y)\,p(x,y)\quad\text{or}\quad E[g(X,Y)]=\iint g(x,y)f(x,y)\,dx\,dy,$$ whenever the sum or integral converges absolutely. In particular $E[X+Y]=E[X]+E[Y]$ for integrable $X,Y$, whether or not they are independent.

Proof. This is the two-dimensional law of the unconscious statistician, and the proof of Theorem 1.28(1) applies word for word: for $g\ge0$, $$E[g(X,Y)]=\int_0^\infty P(g(X,Y)\gt t)\,dt=\iint g\,f,$$ by Tonelli. Linearity follows by taking $g(x,y)=x+y$ and splitting the integral. ∎

📘 Definition 1.27 — Covariance and correlation
For $X,Y$ with finite variances and means $\mu_X,\mu_Y$, the covariance is $$\operatorname{Cov}(X,Y)=E[(X-\mu_X)(Y-\mu_Y)],$$ and if $\sigma_X,\sigma_Y\gt0$ the correlation coefficient is $\rho(X,Y)=\dfrac{\operatorname{Cov}(X,Y)}{\sigma_X\sigma_Y}$. $X$ and $Y$ are uncorrelated if $\operatorname{Cov}(X,Y)=0$. (The product $(X-\mu_X)(Y-\mu_Y)$ is integrable because $|ab|\le\tfrac12(a^2+b^2)$.)
📐 Theorem 1.36 — Properties of covariance and correlation

Let all variables have finite variances.

  1. $\operatorname{Cov}(X,Y)=E[XY]-E[X]E[Y]$; covariance is symmetric and bilinear, $\operatorname{Cov}(X,X)=\operatorname{Var}(X)$, and $\operatorname{Cov}(aX+b,\ cY+d)=ac\operatorname{Cov}(X,Y)$. Hence $\rho(aX+b,\ cY+d)=\operatorname{sign}(ac)\,\rho(X,Y)$ for $ac\ne0$.
  2. $\operatorname{Var}\left(\sum_ia_iX_i\right)=\sum_ia_i^2\operatorname{Var}(X_i)+2\sum_{i\lt j}a_ia_j\operatorname{Cov}(X_i,X_j)$. Independent variables are uncorrelated, so for them the variance of a sum is the sum of the variances.
  3. $|\rho(X,Y)|\le1$, with $|\rho|=1$ iff $Y=a+bX$ almost surely for constants with $b\ne0$; then $\operatorname{sign}(b)=\rho$.
  4. (Best linear predictor) $E[(Y-a-bX)^2]$ is minimised by $b^*=\frac{\operatorname{Cov}(X,Y)}{\operatorname{Var}(X)}$ and $a^*=\mu_Y-b^*\mu_X$, and the minimum is $\sigma_Y^2(1-\rho^2)$.

Proof. (1) Expand $(X-\mu_X)(Y-\mu_Y)$ and use linearity. Bilinearity is linearity of $E$; shifts cancel in $X-\mu_X$.

(2) $\operatorname{Var}(\sum a_iX_i)=\operatorname{Cov}(\sum_ia_iX_i,\sum_ja_jX_j)$; expand by bilinearity. If $X,Y$ are independent, Theorem 1.34(4) gives $E[XY]=E[X]E[Y]$.

(3) Let $U=(X-\mu_X)/\sigma_X$ and $V=(Y-\mu_Y)/\sigma_Y$, so $E[U^2]=E[V^2]=1$ and $E[UV]=\rho$. Then $$0\le E[(V-\rho U)^2]=1-2\rho^2+\rho^2=1-\rho^2.$$ So $\rho^2\le1$, and $\rho^2=1$ iff $E[(V-\rho U)^2]=0$, i.e. $V=\rho U$ almost surely (Theorem 1.28(4)), which says that $Y$ is an affine function of $X$ with slope of the sign of $\rho$. Conversely, if $Y=a+bX$ then $V=\operatorname{sign}(b)U$ and $\rho=\operatorname{sign}(b)$.

(4) For fixed $b$, the best $a$ is $E[Y-bX]$ (Theorem 1.29(2)), and then the error is $\operatorname{Var}(Y-bX)=\sigma_Y^2-2b\operatorname{Cov}+b^2\sigma_X^2$. This quadratic in $b$ is minimised at $b^*=\operatorname{Cov}/\sigma_X^2$, with value $\sigma_Y^2-\operatorname{Cov}^2/\sigma_X^2=\sigma_Y^2(1-\rho^2)$. ∎

These are the population versions of the results on $r$ and the least-squares line in §1.1. Correlation measures only linear dependence: it is invariant under changes of origin and scale (up to sign), it vanishes for independent variables, but it can vanish for strongly dependent ones too.

✏️ Example 2 — Covariances, and zero correlation with full dependence
(a) For $f(x,y)=x+y$ on $(0,1)^2$ (Example 1(a)), find $\operatorname{Cov}(X,Y)$ and $\rho$. (b) $X$ is uniform on $(-1,1)$ and $Y=X^2$. Show that $X$ and $Y$ are uncorrelated but not independent. (c) $\operatorname{Var}(X)=4$, $\operatorname{Var}(Y)=9$ and $\rho(X,Y)=-\tfrac12$. Find $\operatorname{Var}(2X-3Y)$ and $\operatorname{Cov}(X+Y,\ X-Y)$.
(a) Moments From the marginal $x+\tfrac12$: $E[X]=\tfrac13+\tfrac14=\tfrac7{12}$ and $E[X^2]=\tfrac14+\tfrac16=\tfrac5{12}$, so $\operatorname{Var}(X)=\tfrac5{12}-\tfrac{49}{144}=\tfrac{11}{144}$; by symmetry the same holds for $Y$. Next $E[XY]=\iint xy(x+y)=2\cdot\tfrac13\cdot\tfrac12=\tfrac13$.
(a) Answer $\operatorname{Cov}(X,Y)=\tfrac13-\tfrac{49}{144}=-\tfrac1{144}$ and $\rho=\frac{-1/144}{11/144}=-\tfrac1{11}\approx-0.091$: a weak negative association.
(b) $E[X]=0$ and $E[X^3]=0$ by symmetry, so $\operatorname{Cov}(X,X^2)=E[X^3]-E[X]E[X^2]=0$. But $P(|X|\le\tfrac12,\ Y\gt\tfrac14)=0$, while $P(|X|\le\tfrac12)\,P(Y\gt\tfrac14)=\tfrac12\cdot\tfrac12=\tfrac14$. $Y$ is a function of $X$, as dependent as possible, yet $\rho=0$.
(c) $\operatorname{Cov}(X,Y)=-\tfrac12\cdot2\cdot3=-3$. So $$\operatorname{Var}(2X-3Y)=4\cdot4+9\cdot9-2\cdot2\cdot3\cdot(-3)=16+81+36=133,$$ and $\operatorname{Cov}(X+Y,X-Y)=\operatorname{Var}X-\operatorname{Var}Y=-5$.

Conditional Distributions and Conditional Expectation

📘 Definition 1.28 — Conditional distributions and conditional expectation

(a) Discrete. For $p_Y(y)\gt0$, the conditional pmf of $X$ given $Y=y$ is $p_{X\mid Y}(x\mid y)=\dfrac{p(x,y)}{p_Y(y)}$.

(b) Continuous. For $f_Y(y)\gt0$, the conditional density of $X$ given $Y=y$ is $f_{X\mid Y}(x\mid y)=\dfrac{f(x,y)}{f_Y(y)}$.

(c) The conditional expectation of $X$ given $Y=y$ is the mean of this conditional law, $$m(y)=E[X\mid Y=y]=\sum_xx\,p_{X\mid Y}(x\mid y)\quad\text{or}\quad\int x\,f_{X\mid Y}(x\mid y)\,dx.$$ The random variable $E[X\mid Y]=m(Y)$ is the conditional expectation of $X$ given $Y$. Similarly $\operatorname{Var}(X\mid Y)=E[X^2\mid Y]-\left(E[X\mid Y]\right)^2$.

The conditional density is a density in $x$ for each fixed $y$: it is nonnegative and integrates to $f_Y(y)/f_Y(y)=1$. It is the natural limit of $P(X\le x\mid y\le Y\le y+h)$ as $h\downarrow0$, but beware: conditioning on an event of probability zero depends on how that event is approached (pitfall 9 below). Conditional expectation is the right object for prediction: the best guess of $X$ when $Y$ is observed.

📐 Theorem 1.37 — Properties of conditional expectation

Let $E|X|\lt\infty$, and let $(X,Y)$ be jointly discrete or jointly continuous.

  1. (Tower property, or law of total expectation) $E\left[E[X\mid Y]\right]=E[X]$.
  2. (Taking out what is known) $E[\psi(Y)X\mid Y]=\psi(Y)E[X\mid Y]$, and hence $E[\psi(Y)X]=E[\psi(Y)E[X\mid Y]]$, when $\psi(Y)X$ is integrable.
  3. If $X$ and $Y$ are independent, $E[X\mid Y]=E[X]$.
  4. (Best predictor) If $E[X^2]\lt\infty$, then for every function $h$ with $E[h(Y)^2]\lt\infty$, $$E[(X-h(Y))^2]=E[(X-m(Y))^2]+E[(m(Y)-h(Y))^2].$$ So $m(Y)=E[X\mid Y]$ minimises the mean squared prediction error among all functions of $Y$.

Proof. We give the continuous case; the discrete one replaces integrals by sums. (1) By Fubini (justified by $E|X|\lt\infty$), $$E[m(Y)]=\int\left(\int x\frac{f(x,y)}{f_Y(y)}\,dx\right)f_Y(y)\,dy=\iint x\,f(x,y)\,dx\,dy=E[X].$$ (2) Given $Y=y$, $\psi(Y)=\psi(y)$ is a constant: $\int\psi(y)x\,f_{X\mid Y}(x\mid y)\,dx=\psi(y)m(y)$. Taking expectations and using (1) gives the second statement. (3) Independence gives $f(x,y)=f_X(x)f_Y(y)$, so $f_{X\mid Y}(x\mid y)=f_X(x)$ and $m(y)=E[X]$. (4) Write $X-h(Y)=(X-m(Y))+(m(Y)-h(Y))$ and expand the square. The cross term is $$2E\left[(m(Y)-h(Y))(X-m(Y))\right]=2E\left[(m(Y)-h(Y))\,E[X-m(Y)\mid Y]\right]=0,$$ by (2) with $\psi=m-h$, because $E[X-m(Y)\mid Y]=m(Y)-m(Y)=0$. ∎

📐 Theorem 1.38 — Law of total variance
If $E[X^2]\lt\infty$, then $$\operatorname{Var}(X)=E\left[\operatorname{Var}(X\mid Y)\right]+\operatorname{Var}\left(E[X\mid Y]\right).$$ In words: total variance = average within-group variance + variance of the group means. Consequently $\operatorname{Var}(E[X\mid Y])\le\operatorname{Var}(X)$.

Proof. Put $m=m(Y)=E[X\mid Y]$. By the tower property applied to $X^2$, $E[\operatorname{Var}(X\mid Y)]=E[E[X^2\mid Y]]-E[m^2]$ $=E[X^2]-E[m^2]$. Also $E[m]=E[X]$, so $\operatorname{Var}(m)=E[m^2]-(E[X])^2$. Adding the two lines gives $E[X^2]-(E[X])^2=\operatorname{Var}(X)$. ∎

This is the probabilistic form of the "within plus between" decomposition for grouped data (Theorem 1.6 of §1.1), and the seed of analysis of variance (§6.2). The inequality $\operatorname{Var}(E[X\mid Y])\le\operatorname{Var}(X)$ drives the Rao–Blackwell theorem (§4.2): averaging over what you do not need never increases variance.

✏️ Example 3 — Conditioning in coin tosses
A fair coin is tossed three times. Let $X$ be the number of heads in the first two tosses and $Y$ the number of heads in all three. (a) Tabulate the joint pmf and the marginals. (b) Find the conditional pmf of $X$ given $Y=1$, and $E[X\mid Y]$ as a function of $Y$. (c) Find $E[Y\mid X]$. (d) Verify the tower property and the law of total variance, and find $\rho(X,Y)$.
(a) Each of the $8$ outcomes has probability $\tfrac18$. The nonzero entries are $p(0,0)=\tfrac18$, $p(0,1)=\tfrac18$, $p(1,1)=\tfrac14$, $p(1,2)=\tfrac14$, $p(2,2)=\tfrac18$, $p(2,3)=\tfrac18$. Marginals: $X$ takes $0,1,2$ with probabilities $\tfrac14,\tfrac12,\tfrac14$; $Y$ takes $0,1,2,3$ with $\tfrac18,\tfrac38,\tfrac38,\tfrac18$.
(b) $P(X=0\mid Y=1)=\frac{1/8}{3/8}=\tfrac13$ and $P(X=1\mid Y=1)=\tfrac23$, so $E[X\mid Y=1]=\tfrac23$. In the same way $E[X\mid Y=y]=0,\tfrac23,\tfrac43,2$ for $y=0,1,2,3$, i.e. $E[X\mid Y]=\tfrac23Y$. This matches intuition: given $Y$ heads, each toss is a head with probability $Y/3$ by symmetry, and $X$ counts two of the three tosses.
(c) $Y=X+T_3$ where $T_3$, the third toss, is independent of $X$. By Theorem 1.37(2),(3), $E[Y\mid X]=X+E[T_3]=X+\tfrac12$.
(d) Tower: $E[\tfrac23Y]=\tfrac23\cdot\tfrac32=1=E[X]$. Total variance: $\operatorname{Var}(X\mid Y=y)$ is $0,\tfrac29,\tfrac29,0$, so $E[\operatorname{Var}(X\mid Y)]=\tfrac38\cdot\tfrac29\cdot2=\tfrac16$, while $\operatorname{Var}(\tfrac23Y)=\tfrac49\cdot\tfrac34=\tfrac13$. The sum is $\tfrac12=\operatorname{Var}(X)$. Finally $\operatorname{Cov}(X,Y)=\operatorname{Cov}(X,X+T_3)=\operatorname{Var}(X)=\tfrac12$, so $\rho=\frac{1/2}{\sqrt{(1/2)(3/4)}}=\sqrt{2/3}\approx0.816$.
✏️ Example 4 — A hierarchical model: the Bayes–Laplace urn
The unknown success probability $Y$ of a coin is uniform on $(0,1)$. Given $Y=y$, the coin is tossed $n$ times independently, and $X$ is the number of heads, so $P(X=k\mid Y=y)=\binom nky^k(1-y)^{n-k}$. (a) Find $E[X]$ and $\operatorname{Var}(X)$. (b) Find the marginal pmf of $X$. (c) Find the conditional density of $Y$ given $X=k$, and $E[Y\mid X=k]$.
(a) Given $Y$, $X$ is a sum of $n$ independent indicators with mean $Y$, so $E[X\mid Y]=nY$ and $\operatorname{Var}(X\mid Y)=nY(1-Y)$. Tower: $E[X]=\tfrac n2$. Total variance: $$E[nY(1-Y)]+\operatorname{Var}(nY)=n\left(\tfrac12-\tfrac13\right)+\frac{n^2}{12}=\frac{n(n+2)}{12}.$$ This is much more than the $\tfrac n4$ of a fair coin: uncertainty about $Y$ adds the $\tfrac{n^2}{12}$ term.
(b) By total probability with a density, $$P(X=k)=\int_0^1\binom nky^k(1-y)^{n-k}\,dy=\binom nk\frac{k!\,(n-k)!}{(n+1)!}=\frac{1}{n+1},$$ using the beta integral $\int_0^1y^a(1-y)^b\,dy=\frac{a!\,b!}{(a+b+1)!}$. So $X$ is uniform on $\{0,1,\dots,n\}$! Check: that law has variance $\frac{(n+1)^2-1}{12}=\frac{n(n+2)}{12}$, as in (a).
(c) Bayes' theorem with densities: $$f_{Y\mid X}(y\mid k)=\frac{P(X=k\mid Y=y)\cdot1}{P(X=k)}=(n+1)\binom nky^k(1-y)^{n-k},\quad0\lt y\lt1,$$ a beta density (§2.1). Its mean is $(n+1)\binom nk\frac{(k+1)!\,(n-k)!}{(n+2)!}=\frac{k+1}{n+2}$. This is Laplace's rule of succession: after $k$ successes in $n$ trials, the next trial succeeds with probability $\frac{k+1}{n+2}$. It is our first Bayesian posterior (§4.3).

Transformations: Jacobians, Sums, Maxima and Minima

📐 Theorem 1.39 — The Jacobian formula

Let $(X,Y)$ have joint density $f$ vanishing outside an open set $D\subseteq\mathbb{R}^2$. Let $g=(g_1,g_2)$ be a one-to-one map of $D$ onto an open set $R$ whose inverse $(u,v)\mapsto(x(u,v),y(u,v))$ is continuously differentiable on $R$, with Jacobian $$J(u,v)=\det\begin{pmatrix}\partial x/\partial u&\partial x/\partial v\\ \partial y/\partial u&\partial y/\partial v\end{pmatrix}\ne0.$$ Then $(U,V)=g(X,Y)$ has joint density $$f_{U,V}(u,v)=f\big(x(u,v),y(u,v)\big)\,|J(u,v)|\quad\text{on }R,$$ and $0$ outside $R$. If $g$ is not one-to-one but $D$ splits into pieces on which it is, add one such term per piece.

Proof. For a Borel set $B\subseteq R$, $P((U,V)\in B)=P((X,Y)\in g^{-1}(B))$ $=\iint_{g^{-1}(B)}f(x,y)\,dx\,dy$. The change-of-variables theorem for double integrals, with $(x,y)=(x(u,v),y(u,v))$, turns this into $\iint_Bf(x(u,v),y(u,v))\,|J(u,v)|\,du\,dv$. Since this holds for every $B$, the integrand is a density of $(U,V)$. ∎

To find the law of a single function $U=g_1(X,Y)$, invent a convenient companion $V$ (often $V=X$ or $V=Y$), apply the theorem, and integrate $v$ out. This is how the next theorem is proved.

📐 Theorem 1.40 — Sums of independent random variables (convolution)

Let $X$ and $Y$ be independent.

  1. If they take integer values, $P(X+Y=n)=\sum_kP(X=k)\,P(Y=n-k)$.
  2. If they have densities, $X+Y$ has density $$f_{X+Y}(z)=\int_{-\infty}^\infty f_X(x)\,f_Y(z-x)\,dx.$$
  3. $G_{X+Y}=G_XG_Y$ for nonnegative integer variables, and $M_{X+Y}(t)=M_X(t)M_Y(t)$ wherever both mgfs are finite.

Proof. (1) Split $\{X+Y=n\}$ according to the value $k$ of $X$, and use independence. (2) Take $U=X+Y$, $V=X$, a linear bijection of $\mathbb{R}^2$ with inverse $x=v$, $y=u-v$ and $|J|=1$. Theorem 1.39 gives $f_{U,V}(u,v)=f_X(v)f_Y(u-v)$; integrate out $v$. (3) $s^{X+Y}=s^Xs^Y$ and $e^{t(X+Y)}=e^{tX}e^{tY}$ are products of functions of $X$ and of $Y$, so Theorem 1.34(4) applies. ∎

Together with the uniqueness theorems for pgfs and mgfs, (3) is the quickest way to identify the law of a sum. For example, if $X$ and $Y$ are independent Poisson variables with means $\lambda$ and $\mu$, then $G_{X+Y}(s)=e^{\lambda(s-1)}e^{\mu(s-1)}=e^{(\lambda+\mu)(s-1)}$, so $X+Y$ is Poisson with mean $\lambda+\mu$ (Exercise IV.3).

📐 Theorem 1.41 — Maxima and minima of independent random variables

Let $X_1,\dots,X_n$ be independent with cdfs $F_1,\dots,F_n$, and let $M=\max_iX_i$, $L=\min_iX_i$. Then $$F_M(x)=\prod_{i=1}^nF_i(x),\qquad P(L\gt x)=\prod_{i=1}^n\left(1-F_i(x)\right).$$ If the $X_i$ are iid with cdf $F$ and density $f$, then $f_M=nF^{n-1}f$ and $f_L=n(1-F)^{n-1}f$.

(Competing exponentials) If $X_i$ is exponential with rate $\lambda_i$, then $L$ is exponential with rate $\lambda=\sum_i\lambda_i$, and $P(L=X_j)=\lambda_j/\lambda$.

Proof. $\{M\le x\}=\bigcap_i\{X_i\le x\}$ and $\{L\gt x\}=\bigcap_i\{X_i\gt x\}$; apply independence. Differentiate for the iid densities. For exponentials, $P(L\gt x)=\prod e^{-\lambda_ix}=e^{-\lambda x}$. Finally $X_j$ is the minimum iff the others exceed it, so $$P(L=X_j)=\int_0^\infty\lambda_je^{-\lambda_jx}\prod_{i\ne j}e^{-\lambda_ix}\,dx=\int_0^\infty\lambda_je^{-\lambda x}\,dx=\frac{\lambda_j}{\lambda}.$$ (Ties have probability $0$.) ∎

✏️ Example 5 — A Jacobian computation: gamma and uniform from two exponentials
$X$ and $Y$ are iid exponential with rate $\lambda$. Let $U=X+Y$ and $V=\dfrac{X}{X+Y}$. Find the joint density of $(U,V)$, show that $U$ and $V$ are independent, and identify their laws. Deduce $P(X\lt2Y)$.
Inverse map The map sends $D=(0,\infty)^2$ one-to-one onto $R=(0,\infty)\times(0,1)$, with inverse $x=uv$, $y=u(1-v)$. The Jacobian is $$J=\det\begin{pmatrix}v&u\\1-v&-u\end{pmatrix}=-uv-u(1-v)=-u,$$ so $|J|=u$.
Density $f(x,y)=\lambda^2e^{-\lambda(x+y)}=\lambda^2e^{-\lambda u}$, so $$f_{U,V}(u,v)=\lambda^2u\,e^{-\lambda u}\cdot\mathbf{1}_{(0,1)}(v),\quad u\gt0.$$
Independence This is a product of a function of $u$ and a function of $v$ on the product set $R$, so $U$ and $V$ are independent (Theorem 1.34(3)). $U$ has density $\lambda^2ue^{-\lambda u}$ (the gamma law with shape $2$, the waiting time to the second event of a Poisson process, §3.3), and $V$ is uniform on $(0,1)$.
Consequence $X\lt2Y\iff X\lt2(U-X)\iff V\lt\tfrac23$, so $P(X\lt2Y)=\tfrac23$, whatever the value of $\lambda$. Check directly: $\int_0^\infty\lambda e^{-\lambda y}(1-e^{-2\lambda y})\,dy=1-\tfrac13=\tfrac23$.
✏️ Example 6 — Convolution, series and parallel systems
(a) $X$ and $Y$ are iid uniform on $(0,1)$. Find the density of $S=X+Y$ and $P(S\le1.5)$. (b) A device has three independent components with exponential lifetimes of rates $1$, $2$ and $3$ per year. Find the mean lifetime of the device if it works only while all three work (series system), the probability that component 3 is the first to fail, and the mean lifetime if it works while at least one works (parallel system).
(a) $f_S(z)=\int_0^1\mathbf{1}\{0\lt z-x\lt1\}\,dx$ is the length of $(0,1)\cap(z-1,z)$, which is $z$ for $0\lt z\le1$ and $2-z$ for $1\lt z\lt2$: the triangular density. $P(S\gt1.5)$ is the area of the small triangle $\tfrac12\cdot0.5\cdot0.5=\tfrac18$, so $P(S\le1.5)=\tfrac78$. Convolution has smoothed two flat densities into a tent.
(b) Series The device lifetime is $L=\min$, exponential with rate $1+2+3=6$, so $E[L]=\tfrac16$ year, i.e. $2$ months. Component 3 fails first with probability $\tfrac36=\tfrac12$.
(b) Parallel $M=\max$ has $F_M(t)=(1-e^{-t})(1-e^{-2t})(1-e^{-3t})$. Expanding $1-F_M$ and using $\int_0^\infty e^{-ct}dt=\tfrac1c$ (inclusion–exclusion), $$E[M]=\left(1+\tfrac12+\tfrac13\right)-\left(\tfrac13+\tfrac14+\tfrac15\right)+\tfrac16=\frac{73}{60}\approx1.217\text{ years}.$$ Redundancy multiplies the mean life by about $7.3$ compared with the series design.
Interactive 1 — Sums, Maxima and Minima of n Independent Variables

Choose a base law, a statistic and the number $n$ of iid variables, then press Play. Each tick simulates more groups of $n$ draws (seeded) and adds the statistic to the blue histogram. The red curve is the exact density: the $n$-fold convolution for sums (Theorem 1.40) and $nF^{n-1}f$ or $n(1-F)^{n-1}f$ for the maximum and minimum (Theorem 1.41). Watch convolution smooth the uniform into a tent and then a bell, while the minimum of exponentials stays exponential.

2
groups simulated—
sample mean—
exact mean—
sample SD / exact SD—

The Bivariate Normal Distribution

Recall that $Z$ is standard normal if it has density $\varphi(z)=\frac{1}{\sqrt{2\pi}}e^{-z^2/2}$, and that $X=\mu+\sigma Z$ ($\sigma\gt0$) is normal $N(\mu,\sigma^2)$, with density $\frac{1}{\sigma}\varphi\left(\frac{x-\mu}{\sigma}\right)$, mean $\mu$ and variance $\sigma^2$. Completing the square, $\int e^{tz}\varphi(z)\,dz=e^{t^2/2}\int\varphi(z-t)\,dz$, so $M_Z(t)=e^{t^2/2}$ and $M_X(t)=e^{\mu t+\sigma^2t^2/2}$. (The full theory of the normal law is in §2.1.)

📘 Definition 1.29 — Bivariate normal distribution
Let $\mu_1,\mu_2\in\mathbb{R}$, $\sigma_1,\sigma_2\gt0$ and $-1\lt\rho\lt1$. $(X,Y)$ has the bivariate normal distribution $\mathrm{BVN}(\mu_1,\mu_2,\sigma_1^2,\sigma_2^2,\rho)$ if it has joint density $$f(x,y)=\frac{1}{2\pi\sigma_1\sigma_2\sqrt{1-\rho^2}}\exp\left(-\frac{z_1^2-2\rho z_1z_2+z_2^2}{2(1-\rho^2)}\right),$$ where $z_1=\frac{x-\mu_1}{\sigma_1}$ and $z_2=\frac{y-\mu_2}{\sigma_2}$. The contours of $f$ are ellipses centred at $(\mu_1,\mu_2)$. (The degenerate cases $\rho=\pm1$, where $Y$ is an affine function of $X$, have no density.)
📐 Theorem 1.42 — Properties of the bivariate normal distribution

Let $(X,Y)\sim\mathrm{BVN}(\mu_1,\mu_2,\sigma_1^2,\sigma_2^2,\rho)$.

  1. (Marginals) $X\sim N(\mu_1,\sigma_1^2)$ and $Y\sim N(\mu_2,\sigma_2^2)$.
  2. (Conditionals) Given $X=x$, $$Y\sim N\left(\mu_2+\rho\frac{\sigma_2}{\sigma_1}(x-\mu_1),\ \sigma_2^2(1-\rho^2)\right).$$ The regression of $Y$ on $X$ is linear, and the conditional variance does not depend on $x$.
  3. (Correlation) $\rho(X,Y)=\rho$, and $X,Y$ are independent iff $\rho=0$.
  4. (Construction) If $Z_1,Z_2$ are iid $N(0,1)$, then $X=\mu_1+\sigma_1Z_1$, $Y=\mu_2+\sigma_2\left(\rho Z_1+\sqrt{1-\rho^2}\,Z_2\right)$ has this distribution.
  5. (Linear combinations) For $(a,b)\ne(0,0)$, $aX+bY$ is normal with mean $a\mu_1+b\mu_2$ and variance $a^2\sigma_1^2+2ab\rho\sigma_1\sigma_2+b^2\sigma_2^2$.

Proof. (1), (2) Complete the square in $z_2$: $$z_1^2-2\rho z_1z_2+z_2^2=(z_2-\rho z_1)^2+(1-\rho^2)z_1^2.$$ Hence $f(x,y)$ is the product of $$\frac{1}{\sigma_1\sqrt{2\pi}}e^{-z_1^2/2}$$ and $$\frac{1}{\sigma_2\sqrt{1-\rho^2}\sqrt{2\pi}}\exp\left(-\frac{(z_2-\rho z_1)^2}{2(1-\rho^2)}\right).$$ The first factor is the $N(\mu_1,\sigma_1^2)$ density in $x$. For fixed $x$, since $z_2-\rho z_1=\frac{y-\mu_2-\rho\sigma_2z_1}{\sigma_2}$, the second factor is the density in $y$ of $N(\mu_2+\rho\sigma_2z_1,\ \sigma_2^2(1-\rho^2))$, so it integrates to $1$ over $y$. Integrating $f$ over $y$ leaves the first factor, which is (1) for $X$ (and $Y$ follows by symmetry). Dividing $f$ by $f_X$ leaves the second factor, which is (2).

(3) Using (2) and the tower property, $$\operatorname{Cov}(X,Y)=E\left[(X-\mu_1)\,E[Y-\mu_2\mid X]\right]=E\left[(X-\mu_1)\,\rho\tfrac{\sigma_2}{\sigma_1}(X-\mu_1)\right]=\rho\sigma_1\sigma_2.$$ If $\rho=0$ the exponent splits as $-\tfrac12(z_1^2+z_2^2)$ and $f=f_Xf_Y$; conversely independent variables are uncorrelated.

(4) The map $(z_1,z_2)\mapsto(x,y)$ is linear with inverse $z_1=\frac{x-\mu_1}{\sigma_1}$, $z_2=\frac{(y-\mu_2)/\sigma_2-\rho z_1}{\sqrt{1-\rho^2}}$ and Jacobian $\frac{1}{\sigma_1\sigma_2\sqrt{1-\rho^2}}$. The joint density of $(Z_1,Z_2)$ is $\frac{1}{2\pi}e^{-(z_1^2+z_2^2)/2}$, and substituting gives exactly the exponent $z_1^2+\frac{(z_2-\rho z_1)^2}{1-\rho^2}$ of (1)–(2) (with $z_2$ there meaning $\frac{y-\mu_2}{\sigma_2}$). Theorem 1.39 completes the proof.

(5) By (4), $aX+bY=c+\alpha Z_1+\beta Z_2$ with $c=a\mu_1+b\mu_2$, $\alpha=a\sigma_1+b\rho\sigma_2$, $\beta=b\sigma_2\sqrt{1-\rho^2}$, not both zero. By Theorem 1.40(3), its mgf is $$e^{ct}e^{\alpha^2t^2/2}e^{\beta^2t^2/2}=e^{ct+(\alpha^2+\beta^2)t^2/2},$$ the mgf of $N(c,\alpha^2+\beta^2)$. By uniqueness (Theorem 1.32(5)), $aX+bY$ has this law, and $\alpha^2+\beta^2$ expands to the stated variance. ∎

The slope $\rho\sigma_2/\sigma_1$ in (2) is the population regression coefficient, and $\sigma_2^2(1-\rho^2)$ is the minimum mean squared error of Theorem 1.36(4). For the bivariate normal the best predictor (Theorem 1.37(4)) is linear, so the best linear predictor is the best predictor of all. Because $|\rho|\lt1$, the predicted $Y$ lies fewer standard deviations from its mean than the observed $X$: this is Galton's regression to the mean.

✏️ Example 7 — Heights of fathers and sons
In a population, the heights (in cm) of a father $X$ and his adult son $Y$ are bivariate normal with $\mu_1=172$, $\sigma_1=7$, $\mu_2=174$, $\sigma_2=8$ and $\rho=0.5$. (a) A father is $186$ cm tall. Find the conditional mean and standard deviation of his son's height, and the probability that the son is taller than $186$ cm. (b) Find the probability that a randomly chosen son is taller than his father.
(a) Conditional law $z_1=\frac{186-172}{7}=2$. By Theorem 1.42(2), $$E[Y\mid X=186]=174+0.5\cdot\tfrac87\cdot14=182\text{ cm},$$ and the conditional SD is $8\sqrt{1-0.25}=\sqrt{48}\approx6.93$ cm.
(a) Probability $P(Y\gt186\mid X=186)=1-\Phi\left(\frac{186-182}{6.93}\right)=1-\Phi(0.577)\approx0.282$. The father is $2$ SDs above the mean; the son is expected to be only $\frac{182-174}{8}=1$ SD above: regression to the mean. (Unconditionally $P(Y\gt186)=1-\Phi(1.5)\approx0.067$, so knowing the father's height still helps a lot.)
(b) By Theorem 1.42(5), $Y-X$ is normal with mean $2$ and variance $64+49-2\cdot0.5\cdot7\cdot8=57$. So $P(Y\gt X)=P(Y-X\gt0)=\Phi\left(\tfrac{2}{\sqrt{57}}\right)=\Phi(0.265)\approx0.604$.
Interactive 2 — The Bivariate Normal: Scatter, Ellipses and a Conditional Slice

Seven hundred points $(X,Y)=(Z_1,\ \rho Z_1+\sqrt{1-\rho^2}\,Z_2)$ from a seeded generator, with the $50\%$ and $90\%$ probability ellipses. The red line is the regression line $E[Y\mid X=x]=\rho x$; the dashed green line is $E[X\mid Y=y]=\rho y$; the grey dashed line is the major axis of the ellipses. Points in the vertical slice at $x_0$ are highlighted, and the right panel compares their $Y$-values with the conditional density $N(\rho x_0,1-\rho^2)$ (red) and the marginal $N(0,1)$ (grey). Press Play to sweep $\rho$ and watch the cloud, the ellipses and the two regression lines move together.

0.601.5
ρ—
sample correlation—
E[Y | X = x₀] = ρx₀—
SD(Y | X) = √(1 − ρ²)—
⚠️ Counterexamples and pitfalls
  1. Marginals do not determine the joint law. With $U$ uniform on $(0,1)$, the pairs $(U,U)$, $(U,1-U)$ and $(U,U')$ with $U'$ an independent copy all have uniform marginals. They attain the upper Fréchet bound, the lower one, and the product $xy$ respectively (Exercise I.7).
  2. Each variable continuous, but no joint density. $(X,X)$ is concentrated on a line of area $0$.
  3. A product formula on a non-product support is not independence. $f(x,y)=8xy$ on $0\lt x\lt y\lt1$ "factorises", but the support is a triangle, so $X$ and $Y$ are dependent (Exercise I.8). The factorisation must hold on all of $\mathbb{R}^2$, indicator functions included.
  4. Uncorrelated does not imply independent. $X$ uniform on $(-1,1)$ and $Y=X^2$ (Example 2(b)); or $(X,Y)$ uniform on the unit disc (Exercise I.6). Likewise $E[XY]=E[X]E[Y]$ does not imply independence.
  5. Normal marginals and zero correlation do not make an independent pair. Let $X\sim N(0,1)$, $W=\pm1$ with probability $\tfrac12$ each, independent of $X$, and $Y=WX$. Then $Y\sim N(0,1)$ and $\operatorname{Cov}(X,Y)=E[W]E[X^2]=0$, but $|Y|=|X|$. And $(X,Y)$ is not bivariate normal: $P(X+Y=0)=\tfrac12$ (Exercise V.6). Theorem 1.42(3) needs joint normality.
  6. A function that is monotone in each variable need not be a joint cdf. $F(x,y)=\mathbf{1}\{x+y\ge0\}$ violates the rectangle rule (after Theorem 1.33).
  7. $E[X\mid Y]$ is a random variable, the function $m(Y)$, not a number. Only $E[X\mid Y=y]$ is a number, and only $E[E[X\mid Y]]$ equals $E[X]$.
  8. The variance of a difference. $\operatorname{Var}(X-Y)=\operatorname{Var}X+\operatorname{Var}Y-2\operatorname{Cov}(X,Y)$. For independent variables it is the sum of the variances, never their difference.
  9. Conditioning on a null event is ambiguous (Borel–Kolmogorov). For $X,Y$ iid exponential(1), "given $X=Y$" computed via $D=X-Y=0$ gives the conditional density $2e^{-2x}$ (mean $\tfrac12$), but via $R=X/Y=1$ it gives $4xe^{-2x}$ (mean $1$). A conditional density is attached to a conditioning variable, not to an event of probability zero.
  10. The maximum and minimum of iid variables are dependent. Always $L\le M$; for two uniforms $P(L\gt\tfrac12,\ M\lt\tfrac12)=0$ though both factors are positive.
  11. The two regression lines are different. $E[Y\mid X=x]$ has slope $\rho\sigma_2/\sigma_1$, while solving $E[X\mid Y=y]$ for $y$ gives slope $\sigma_2/(\rho\sigma_1)$. They coincide only when $|\rho|=1$, and they are perpendicular (the two axes) when $\rho=0$ (Interactive 2).
  12. Pairwise independence of random variables is not mutual independence, exactly as for events (§1.2): two fair $\pm1$ coins $W_1,W_2$ and their product $W_1W_2$ are pairwise but not mutually independent.

Exercises

I
Joint laws, marginals and independence.
  • 1 $f(x,y)=c(x^2+y)$ on $(0,1)^2$. Find $c$, both marginal densities and $P(X\gt Y)$.
  • 2 $f(x,y)=e^{-y}$ for $0\lt x\lt y$. Find the marginal densities, decide independence and find $P(Y\gt2X)$.
  • 3 $p(x,y)=c(x+y)$ for $x,y\in\{0,1,2\}$. Find $c$ and the marginal pmfs, and decide whether $X$ and $Y$ are independent.
  • 4 Show that $F(x,y)=1-e^{-x-y}$ for $x,y\ge0$ (and $0$ otherwise) is not a joint distribution function.
  • 5 Prove that a random variable independent of itself is almost surely constant.
  • 6 $(X,Y)$ is uniform on the unit disc. Find $f_X$, show that $\operatorname{Cov}(X,Y)=0$, and show that $X$ and $Y$ are not independent.
  • 7 $U$ is uniform on $(0,1)$. Compute the joint cdfs of $(U,U)$ and of $(U,1-U)$, and show that they attain the upper and lower Fréchet bounds.
  • 8 $f(x,y)=8xy$ for $0\lt x\lt y\lt1$. Check that $f$ is a density, find the marginals, and explain why $X$ and $Y$ are dependent although $f$ is a product $g(x)h(y)$ on its support.
1 $\iint(x^2+y)=\tfrac13+\tfrac12=\tfrac56$, so $c=\tfrac65$. $f_X(x)=\tfrac65\left(x^2+\tfrac12\right)$ and $f_Y(y)=\tfrac65\left(\tfrac13+y\right)$. $P(X\gt Y)=\tfrac65\int_0^1\int_0^x(x^2+y)\,dy\,dx=\tfrac65\int_0^1\left(x^3+\tfrac{x^2}{2}\right)dx=\tfrac65\cdot\tfrac5{12}=\tfrac12$.
2 $f_X(x)=\int_x^\infty e^{-y}dy=e^{-x}$ ($x\gt0$) and $f_Y(y)=\int_0^ye^{-y}dx=ye^{-y}$ ($y\gt0$). The support $\{0\lt x\lt y\}$ is not a product set, so they are dependent. $P(Y\gt2X)=\int_0^\infty\int_{2x}^\infty e^{-y}\,dy\,dx=\int_0^\infty e^{-2x}dx=\tfrac12$.
3 $\sum_{x,y}(x+y)=2\cdot3\cdot3=18$, so $c=\tfrac1{18}$. $p_X(x)=\frac{3x+3}{18}=\frac{x+1}6$, and $p_Y$ is the same. $p(0,0)=0\ne p_X(0)p_Y(0)=\tfrac1{36}$, so they are dependent.
4 For $0\le a\lt b$ and $0\le c\lt d$ the rectangle rule gives $$F(b,d)-F(a,d)-F(b,c)+F(a,c)=(e^{-a}-e^{-b})(e^{-d}-e^{-c})\lt0,$$ a negative probability.
5 Independence of $X$ from itself gives $F(x)=P(X\le x,X\le x)=F(x)^2$, so $F(x)\in\{0,1\}$ for every $x$. A cdf with values in $\{0,1\}$ jumps from $0$ to $1$ at a single point $c$, so $P(X=c)=1$.
6 $f=\tfrac1\pi$ on the disc, so $f_X(x)=\tfrac1\pi\cdot2\sqrt{1-x^2}$ on $(-1,1)$. $E[XY]=0$ and $E[X]=0$ by the symmetry $(x,y)\mapsto(-x,y)$, so $\operatorname{Cov}=0$. But $P(X\gt0.8,\ Y\gt0.8)=0$ (the point $(0.8,0.8)$ is outside the disc), while each factor $P(X\gt0.8)$, $P(Y\gt0.8)$ is positive.
7 $P(U\le x,\ U\le y)=\min(x,y)$, the upper bound $\min(F_X,F_Y)$. $P(U\le x,\ 1-U\le y)=P(1-y\le U\le x)=\max(x+y-1,0)$, the lower bound. (Both for $x,y\in[0,1]$.)
8 $\int_0^1\int_0^y8xy\,dx\,dy=\int_0^14y^3dy=1$. $f_X(x)=\int_x^18xy\,dy=4x(1-x^2)$ and $f_Y(y)=4y^3$. Their product is not $8xy$. The function $f$ equals $8xy\cdot\mathbf{1}\{x\lt y\}$, and the indicator does not split into a function of $x$ times a function of $y$, so Theorem 1.34(3) does not apply.
II
Covariance and correlation.
  • 1 For $f(x,y)=2$ on $0\lt x\lt y\lt1$, find $E[X]$, $E[Y]$, the variances, $\operatorname{Cov}(X,Y)$ and $\rho$.
  • 2 Show that if $\operatorname{Var}(X)=\operatorname{Var}(Y)$, then $X+Y$ and $X-Y$ are uncorrelated.
  • 3 $X_1,\dots,X_n$ have common variance $\sigma^2\gt0$ and common pairwise correlation $\rho$. Find $\operatorname{Var}(X_1+\cdots+X_n)$ and deduce that $\rho\ge-\frac{1}{n-1}$.
  • 4 For the distribution of II.1, find the best linear predictor $a+bX$ of $Y$, and compare it with $E[Y\mid X]$.
  • 5 Give two random variables with $\rho=0$ such that one is a function of the other, other than Example 2(b).
  • 6 Show that two indicator variables $\mathbf{1}_A$ and $\mathbf{1}_B$ are uncorrelated iff the events $A$ and $B$ are independent.
1 From $f_X=2(1-x)$ and $f_Y=2y$: $E[X]=\tfrac13$, $E[Y]=\tfrac23$, $E[X^2]=\tfrac16$, $E[Y^2]=\tfrac12$, so both variances are $\tfrac1{18}$. $E[XY]=\int_0^1\int_0^y2xy\,dx\,dy=\int_0^1y^3dy=\tfrac14$, so $\operatorname{Cov}=\tfrac14-\tfrac29=\tfrac1{36}$ and $\rho=\tfrac12$.
2 $\operatorname{Cov}(X+Y,X-Y)=\operatorname{Var}X-\operatorname{Cov}(X,Y)+\operatorname{Cov}(Y,X)-\operatorname{Var}Y=0$.
3 $\operatorname{Var}(\sum X_i)=n\sigma^2+n(n-1)\rho\sigma^2=n\sigma^2(1+(n-1)\rho)$. A variance is nonnegative, so $1+(n-1)\rho\ge0$. Equality $\rho=-\tfrac1{n-1}$ occurs, e.g., for $X_i=Y_i-\bar Y$ with $Y_i$ iid, whose sum is $0$.
4 $b=\operatorname{Cov}/\operatorname{Var}X=\frac{1/36}{1/18}=\tfrac12$ and $a=\tfrac23-\tfrac12\cdot\tfrac13=\tfrac12$: the predictor is $\tfrac12+\tfrac12X$. Given $X=x$, $Y$ is uniform on $(x,1)$, so $E[Y\mid X]=\frac{1+X}{2}$: the same line. Here the best predictor happens to be linear.
5 $X$ uniform on $(-1,1)$ and $Y=|X|$: $E[X|X|]=0=E[X]E|X|$. Or $X$ uniform on $\{-1,0,1\}$ and $Y=\mathbf{1}\{X=0\}$: $E[XY]=0=E[X]E[Y]$.
6 $\operatorname{Cov}(\mathbf{1}_A,\mathbf{1}_B)=E[\mathbf{1}_{A\cap B}]-P(A)P(B)=P(A\cap B)-P(A)P(B)$. It vanishes iff $A,B$ are independent, and then $\mathbf{1}_A,\mathbf{1}_B$ are independent random variables (complements of independent events are independent, §1.2). So for $\{0,1\}$-valued variables, uncorrelated and independent are equivalent.
III
Conditional distributions and conditional expectation.
  • 1 For $f(x,y)=e^{-y}$, $0\lt x\lt y$, find $f_{Y\mid X}(y\mid x)$, $E[Y\mid X]$, $f_{X\mid Y}(x\mid y)$ and $E[X\mid Y]$, and check the tower property both ways.
  • 2 $Y$ is uniform on $(0,1)$, and given $Y=y$, $X$ is uniform on $(0,y)$. Find $f_X$, $E[X]$ and $\operatorname{Var}(X)$, the last both directly and by the law of total variance.
  • 3 (Insurance) The number $N$ of claims in a year has mean $50$ and variance $50$. Claim sizes are iid with mean $\text{₹}\,4000$ and SD $\text{₹}\,3000$, independent of $N$. Find the mean and SD of the total $S=X_1+\cdots+X_N$.
  • 4 Give an example where $E[X\mid Y]=E[X]$ but $X$ and $Y$ are not independent.
  • 5 A fair die is rolled, and a fair coin is then tossed as many times as the die shows. Let $X$ be the number of heads. Find $E[X]$, $\operatorname{Var}(X)$ and $P(X=0)$.
  • 6 Prove that $\operatorname{Var}(E[X\mid Y])=\operatorname{Var}(X)$ iff $X$ is almost surely a function of $Y$ (in the discrete case: $\operatorname{Var}(X\mid Y=y)=0$ for every $y$ with $P(Y=y)\gt0$).
  • 7 Given $\Lambda=\lambda$, $X$ is Poisson with mean $\lambda$, and $\Lambda$ is exponential with rate $1$. Find the pmf, mean and variance of $X$.
  • 8 In Example 4 with $n=10$, all ten tosses are heads. Find the posterior mean of $Y$ and the probability that the eleventh toss is a head.
1 $f_{Y\mid X}(y\mid x)=e^{-y}/e^{-x}=e^{-(y-x)}$ for $y\gt x$: $Y-X$ given $X$ is exponential(1), so $E[Y\mid X]=X+1$. $f_{X\mid Y}(x\mid y)=e^{-y}/(ye^{-y})=\tfrac1y$ on $(0,y)$, so $E[X\mid Y]=\tfrac Y2$. Tower: $E[X+1]=2=E[Y]$ and $E[Y/2]=1=E[X]$.
2 $f_X(x)=\int_x^1\tfrac1y\,dy=-\log x$ on $(0,1)$. $E[X]=E[Y/2]=\tfrac14$. Total variance: $E[Y^2/12]+\operatorname{Var}(Y/2)=\tfrac1{36}+\tfrac1{48}=\tfrac7{144}$. Directly: $E[X^2]=\int_0^1x^2(-\log x)\,dx=\tfrac19$, and $\tfrac19-\tfrac1{16}=\tfrac7{144}$.
3 $E[S\mid N]=4000N$ and $\operatorname{Var}(S\mid N)=3000^2N$. So $E[S]=4000\cdot50=\text{₹}\,2{,}00{,}000$ and $$\operatorname{Var}(S)=9\times10^6\cdot50+(4000)^2\cdot50=1.25\times10^9,$$ an SD of about $\text{₹}\,35{,}355$. The general rule is $\operatorname{Var}(S)=E[N]\sigma^2+\mu^2\operatorname{Var}(N)$.
4 $(X,Y)$ uniform on the unit disc: given $Y=y$, $X$ is uniform on $(-\sqrt{1-y^2},\sqrt{1-y^2})$, so $E[X\mid Y]=0=E[X]$, yet the conditional spread depends on $y$ (Exercise I.6).
5 Let $N$ be the die. $E[X\mid N]=\tfrac N2$ and $\operatorname{Var}(X\mid N)=\tfrac N4$. So $E[X]=\tfrac74$ and $\operatorname{Var}(X)=\tfrac{3.5}{4}+\tfrac{35/12}{4}=\tfrac{77}{48}\approx1.604$. $P(X=0)=\tfrac16\sum_{n=1}^62^{-n}=\tfrac16\cdot\tfrac{63}{64}=\tfrac{21}{128}$.
6 By Theorem 1.38, equality holds iff $E[\operatorname{Var}(X\mid Y)]=0$. Since $\operatorname{Var}(X\mid Y)\ge0$, this means $\operatorname{Var}(X\mid Y)=0$ almost surely, i.e. given $Y$ the law of $X$ is a point mass at $m(Y)$, so $X=m(Y)$ almost surely.
7 $P(X=k)=\int_0^\infty e^{-\lambda}\frac{\lambda^k}{k!}e^{-\lambda}d\lambda=\frac{1}{k!}\cdot\frac{k!}{2^{k+1}}=2^{-(k+1)}$, $k\ge0$. $E[X]=E[\Lambda]=1$ and $\operatorname{Var}(X)=E[\Lambda]+\operatorname{Var}(\Lambda)=2$, matching the geometric law of Exercise IV.3 of §1.3.
8 The posterior is $11y^{10}$ on $(0,1)$, with mean $\frac{k+1}{n+2}=\frac{11}{12}$. Given $Y$, the eleventh toss is a head with probability $Y$, so by the tower property the answer is $E[Y\mid X=10]=\tfrac{11}{12}\approx0.917$.
IV
Transformations, sums and extremes.
  • 1 $X,Y$ iid uniform on $(0,1)$. Find $P(X+Y\le\tfrac12)$ and $P(\tfrac12\le X+Y\le\tfrac32)$.
  • 2 $X,Y$ iid exponential with rate $\lambda$. Show that $X-Y$ has the Laplace density $\tfrac\lambda2e^{-\lambda|z|}$.
  • 3 $X,Y$ independent Poisson with means $\lambda,\mu$. Show that $X+Y$ is Poisson with mean $\lambda+\mu$ and that, given $X+Y=n$, $X$ is binomial with $n$ trials and success probability $\frac{\lambda}{\lambda+\mu}$.
  • 4 $X,Y$ iid $N(0,1)$. Use the Jacobian formula with $V=Y$ to show that $X/Y$ has the Cauchy density $\frac{1}{\pi(1+z^2)}$.
  • 5 $X,Y$ iid $N(0,1)$, and $(R,\Theta)$ are their polar coordinates. Show that $R$ and $\Theta$ are independent, $\Theta$ uniform on $(0,2\pi)$ and $R$ with density $re^{-r^2/2}$. Deduce the Box–Muller method: if $U_1,U_2$ are iid uniform, then $\sqrt{-2\log U_1}\cos(2\pi U_2)$ is $N(0,1)$.
  • 6 Five bulbs have iid exponential lifetimes with mean $1000$ hours. Find the expected time to the first failure and to the last failure.
  • 7 $X_1,\dots,X_n$ are iid uniform on $(0,\theta)$ and $M=\max X_i$. Find $E[M]$ and the smallest $n$ with $P(M\gt0.9\theta)\ge0.95$.
  • 8 $X,Y$ iid exponential(1), $L=\min(X,Y)$, $M=\max(X,Y)$. Show that $L$ and $M-L$ are independent, with $L$ exponential(2) and $M-L$ exponential(1).
1 From the triangular density: $P(S\le\tfrac12)=\int_0^{1/2}z\,dz=\tfrac18$, and by symmetry $P(S\ge\tfrac32)=\tfrac18$, so $P(\tfrac12\le S\le\tfrac32)=\tfrac34$.
2 $f_{X-Y}(z)=\int f_X(x)f_Y(x-z)\,dx$ (convolution of $X$ with $-Y$). For $z\ge0$ the integrand is positive for $x\gt z$: $\int_z^\infty\lambda^2e^{-\lambda x}e^{-\lambda(x-z)}dx=\tfrac\lambda2e^{-\lambda z}$. By symmetry ($X-Y$ and $Y-X$ have the same law) the density is $\tfrac\lambda2e^{-\lambda|z|}$.
3 pgfs: $e^{\lambda(s-1)}e^{\mu(s-1)}=e^{(\lambda+\mu)(s-1)}$, so the sum is Poisson$(\lambda+\mu)$ by uniqueness. Then $$P(X=k\mid X+Y=n)=\frac{e^{-\lambda}\frac{\lambda^k}{k!}e^{-\mu}\frac{\mu^{n-k}}{(n-k)!}}{e^{-(\lambda+\mu)}\frac{(\lambda+\mu)^n}{n!}}=\binom nk\left(\tfrac{\lambda}{\lambda+\mu}\right)^k\left(\tfrac{\mu}{\lambda+\mu}\right)^{n-k}.$$
4 $Z=X/Y$, $V=Y$: inverse $x=zv$, $y=v$, $|J|=|v|$. So $f_Z(z)=\int|v|\varphi(zv)\varphi(v)\,dv=\frac{2}{2\pi}\int_0^\infty ve^{-(1+z^2)v^2/2}dv=\frac{1}{\pi(1+z^2)}$.
5 $x=r\cos\theta$, $y=r\sin\theta$, $|J|=r$. So $f(r,\theta)=\frac{1}{2\pi}e^{-r^2/2}r$ on $(0,\infty)\times(0,2\pi)$, a product: $\Theta$ uniform and $R$ with density $re^{-r^2/2}$, independent. $P(R\gt r)=e^{-r^2/2}$, so $R$ has the law of $\sqrt{-2\log U_1}$ (quantile transform), $\Theta$ that of $2\pi U_2$, and $X=R\cos\Theta$. This is the generator used by the widgets on these pages.
6 The minimum is exponential with rate $5/1000$: mean $200$ h. For the maximum, after each failure the remaining bulbs restart afresh (memorylessness, §2.1), so $E[M]=1000\left(\tfrac15+\tfrac14+\tfrac13+\tfrac12+1\right)=1000\cdot\tfrac{137}{60}\approx2283$ h.
7 $F_M(x)=(x/\theta)^n$, $f_M=nx^{n-1}/\theta^n$, so $E[M]=\frac{n}{n+1}\theta$. $P(M\gt0.9\theta)=1-0.9^n\ge0.95$ iff $n\ge\frac{\log0.05}{\log0.9}\approx28.4$, so $n=29$.
8 For $a,b\ge0$, by symmetry between $X\lt Y$ and $Y\lt X$, $$P(L\gt a,\ M-L\gt b)=2P(a\lt X,\ Y\gt X+b)=2\int_a^\infty e^{-x}e^{-(x+b)}dx=e^{-2a}\,e^{-b}.$$ The joint survival function factorises into an exponential(2) and an exponential(1) survival function, which gives independence and both laws.
V
The bivariate normal distribution.
  • 1 $(X,Y)$ is bivariate normal with $\mu_1=1$, $\mu_2=-1$, $\sigma_1=2$, $\sigma_2=3$, $\rho=0.4$. Find $E[Y\mid X=3]$, $\operatorname{Var}(Y\mid X=3)$ and $P(Y\gt0\mid X=3)$.
  • 2 For the same pair, find the laws of $X+Y$ and $X-Y$, and decide whether they are independent.
  • 3 For a standard bivariate normal pair ($\mu_i=0$, $\sigma_i=1$) with correlation $\rho$, show that $X+Y$ and $X-Y$ are independent.
  • 4 For a standard bivariate normal pair with correlation $\rho$, prove that $P(X\gt0,\ Y\gt0)=\tfrac14+\frac{\arcsin\rho}{2\pi}$, and evaluate it for $\rho=\tfrac12$.
  • 5 For $(X,Y)$ bivariate normal, find the constant $c$ such that $X-cY$ is independent of $Y$.
  • 6 $X\sim N(0,1)$, $W=\pm1$ with probability $\tfrac12$ each independently of $X$, and $Y=WX$. Show that $Y\sim N(0,1)$ and $\operatorname{Cov}(X,Y)=0$, that $X$ and $Y$ are dependent, and that $(X,Y)$ is not bivariate normal.
1 $E[Y\mid X=3]=-1+0.4\cdot\tfrac32\cdot(3-1)=0.2$ and $\operatorname{Var}(Y\mid X)=9(1-0.16)=7.56$ (SD $2.750$). $P(Y\gt0\mid X=3)=1-\Phi\left(\frac{-0.2}{2.750}\right)=\Phi(0.0727)\approx0.529$.
2 $\operatorname{Cov}(X,Y)=0.4\cdot6=2.4$. $X+Y\sim N(0,\ 4+9+4.8)=N(0,17.8)$ and $X-Y\sim N(2,\ 13-4.8)=N(2,8.2)$. $\operatorname{Cov}(X+Y,X-Y)=4-9=-5\ne0$, so they are dependent.
3 $(X+Y,X-Y)$ is a linear image of $(Z_1,Z_2)$ and hence jointly normal (every linear combination is normal, by Theorem 1.42(5); this is the definition of joint normality used in §7.1). Their covariance is $\operatorname{Var}X-\operatorname{Var}Y=0$, so they are independent by Theorem 1.42(3).
4 Write $X=Z_1$ and $Y=\rho Z_1+\sqrt{1-\rho^2}Z_2$. In the $(z_1,z_2)$-plane the two events are half-planes through the origin with unit normals $(1,0)$ and $(\rho,\sqrt{1-\rho^2})$, at angle $\theta=\arccos\rho$. Their intersection is a wedge of angle $\pi-\theta$, and the law of $(Z_1,Z_2)$ is rotation invariant, so the probability is $\frac{\pi-\arccos\rho}{2\pi}=\tfrac14+\frac{\arcsin\rho}{2\pi}$. For $\rho=\tfrac12$: $\tfrac14+\tfrac1{12}=\tfrac13$.
5 $(X-cY,Y)$ is jointly normal, so independence is equivalent to $\operatorname{Cov}(X-cY,Y)=\rho\sigma_1\sigma_2-c\sigma_2^2=0$, i.e. $c=\rho\sigma_1/\sigma_2$. Then $X-cY$ is the "residual" of $X$ after regression on $Y$.
6 $P(Y\le y)=\tfrac12P(X\le y)+\tfrac12P(-X\le y)=\Phi(y)$ by symmetry of $X$. $\operatorname{Cov}(X,Y)=E[WX^2]=E[W]E[X^2]=0$. But $|Y|=|X|$, so $P(|X|\le1,\ |Y|\gt1)=0\ne P(|X|\le1)P(|Y|\gt1)$. Finally $X+Y=0$ whenever $W=-1$, so $P(X+Y=0)=\tfrac12$: $X+Y$ is not normal, contradicting Theorem 1.42(5).
🎯 Exam corner CSIR Part BCSIR Part CGATE ST

E1. CSIR Part B $(X,Y)$ has joint density $f(x,y)=6x$ for $0\lt x\lt y\lt1$. Then $E\left[X\mid Y=\tfrac34\right]$ equals (A) $\tfrac12$; (B) $\tfrac38$; (C) $\tfrac23$; (D) $\tfrac14$.

Idea Find $f_Y$, divide, and integrate $x$ against the conditional density.
Compute $f_Y(y)=\int_0^y6x\,dx=3y^2$, so $f_{X\mid Y}(x\mid y)=\frac{2x}{y^2}$ on $(0,y)$ and $E[X\mid Y=y]=\int_0^y\frac{2x^2}{y^2}dx=\frac{2y}{3}$. At $y=\tfrac34$ this is $\tfrac12$. Option (B) is $y/2$, the answer for a uniform conditional law, and (C) is $2/3$ without the factor $y$.
Answer (A).

E2. CSIR Part B GATE ST $X$ and $Y$ are iid exponential with mean $1$. Then $E[\max(X,Y)]$ equals (A) $1$; (B) $\tfrac32$; (C) $2$; (D) $\tfrac12$.

Idea $\max+\min=X+Y$, and $\min$ is exponential with rate $2$ (Theorem 1.41).
Compute $E[\max]=E[X+Y]-E[\min]=2-\tfrac12=\tfrac32$. Alternatively $\int_0^\infty\left(1-(1-e^{-t})^2\right)dt=2-\tfrac12$. Option (D) is $E[\min]$.
Answer (B).

E3. CSIR Part B $(X,Y)$ is bivariate normal with $\sigma_X=2$, $\sigma_Y=4$ and $\rho=0.6$. Then $\operatorname{Var}(Y\mid X=x)$ is (A) $5.76$; (B) $9.6$; (C) $10.24$; (D) $16$.

Idea Theorem 1.42(2): $\operatorname{Var}(Y\mid X=x)=\sigma_Y^2(1-\rho^2)$, the same for every $x$.
Compute $16(1-0.36)=10.24$. Option (A) is $\sigma_Y^2\rho^2$, the part of the variance that is explained by $X$; (D) ignores the conditioning.
Answer (C).

E4. CSIR Part C Let $X$ and $Y$ have finite positive variances. Which of the following are true? (One or more options may be correct.) (A) If $\operatorname{Cov}(X,Y)=0$, then $X$ and $Y$ are independent. (B) If $X$ and $Y$ are independent, then $\operatorname{Var}(X-Y)=\operatorname{Var}(X)+\operatorname{Var}(Y)$. (C) If $(X,Y)$ is bivariate normal and $\operatorname{Cov}(X,Y)=0$, then $X$ and $Y$ are independent. (D) If $X$ and $Y$ are each normally distributed and uncorrelated, then $X+Y$ is normally distributed.

(A) False $X$ uniform on $(-1,1)$ and $Y=X^2$ (Example 2(b)).
(B) True $\operatorname{Var}(X-Y)=\operatorname{Var}X+\operatorname{Var}Y-2\operatorname{Cov}(X,Y)$ and the covariance is $0$.
(C) True Theorem 1.42(3): with $\rho=0$ the joint density factorises.
(D) False $Y=WX$ with an independent random sign $W$: both are $N(0,1)$ and uncorrelated, but $P(X+Y=0)=\tfrac12$ (Exercise V.6).
Answer (B) and (C).

E5. CSIR Part C $Y$ is uniform on $(0,2)$ and, given $Y=y$, $X$ is uniform on $(0,y)$. Which of the following are true? (A) $E[X]=\tfrac12$. (B) $\operatorname{Var}(X)=\tfrac19$. (C) $X$ has density $\tfrac12\log\frac2x$ on $(0,2)$. (D) $X$ and $Y$ are independent.

(A) True $E[X]=E[E[X\mid Y]]=E[Y/2]=\tfrac12$.
(B) False $\tfrac19=E[Y^2/12]=E[\operatorname{Var}(X\mid Y)]$ is only the first term. Adding $\operatorname{Var}(Y/2)=\tfrac14\cdot\tfrac13=\tfrac1{12}$ gives $\operatorname{Var}(X)=\tfrac{7}{36}$.
(C) True $f_X(x)=\int_x^2\tfrac12\cdot\tfrac1y\,dy=\tfrac12\log\frac2x$ for $0\lt x\lt2$. (It integrates to $1$: $\int_0^2\tfrac12\log\frac2x\,dx=\tfrac12\cdot2=1$.)
(D) False The conditional law of $X$ depends on $y$; e.g. $P(X\gt1\mid Y=1)=0$ while $P(X\gt1)\gt0$.
Answer (A) and (C).

E6. CSIR Part C GATE ST $X_1,X_2,X_3$ are iid uniform on $(0,1)$, $M=\max X_i$ and $L=\min X_i$. Which of the following are true? (A) $E[M]=\tfrac34$. (B) $P(L\gt\tfrac12)=\tfrac18$. (C) $M$ and $L$ are independent. (D) $X_1+X_2$ has density $\min(z,2-z)$ on $(0,2)$.

(A) True $f_M=3x^2$, so $E[M]=\int_0^13x^3dx=\tfrac34=\tfrac{n}{n+1}$.
(B) True $P(L\gt\tfrac12)=\left(\tfrac12\right)^3=\tfrac18$.
(C) False $P(L\gt\tfrac12,\ M\lt\tfrac12)=0$, but $P(L\gt\tfrac12)P(M\lt\tfrac12)=\tfrac18\cdot\tfrac18\gt0$.
(D) True The triangular density of Example 6(a): $z$ on $(0,1]$ and $2-z$ on $(1,2)$, i.e. $\min(z,2-z)$.
Answer (A), (B) and (D).
🧑‍🏫 For the teacher

Suggested plan: 3 lectures × 60 min.

  1. Joint laws and covariance. Definition 1.24, Theorem 1.33 with the rectangle rule drawn as four overlapping quadrants, and the $\mathbf{1}\{x+y\ge0\}$ counterexample. Joint densities and marginals; independence and the factorisation criterion (Theorem 1.34), stressing the support. Example 1 on the board (let students guess (b) first). Covariance and correlation (Theorems 1.35–1.36) with the Cauchy–Schwarz proof of $|\rho|\le1$; Example 2.
  2. Conditioning. Definition 1.28, Example 3 as a table on the board. Tower property, taking out what is known, the best-predictor theorem (Theorem 1.37), and the law of total variance (Theorem 1.38). Example 4 (the uniform marginal is a lovely surprise; ask for predictions). Mention the random-sum formula of Exercise III.3, a frequent exam item.
  3. Transformations and the bivariate normal. The Jacobian theorem, Example 5, convolution and maxima/minima (Theorems 1.39–1.41), Example 6. Project Interactive 1: uniform sums with $n=1,2,3,6$, then exponential maxima. Definition 1.29 and Theorem 1.42 by completing the square; Example 7; Interactive 2, sweeping $\rho$ and moving the slice. Finish with E4 and E6 live.

Discussion questions. (1) A study finds that sons of very tall fathers are, on average, shorter than their fathers. Does this mean the population is becoming more uniform in height over generations? (2) Why must the factorisation criterion be applied to the density on all of $\mathbb{R}^2$, indicator functions included? (3) Is there a pair of random variables with $E[X\mid Y]=E[X]$ and $E[Y\mid X]=E[Y]$ that is not independent?

Misconceptions to watch for. Reading independence off a formula and ignoring the support; equating "uncorrelated" with "independent"; believing normal marginals imply joint normality; writing $\operatorname{Var}(X-Y)=\operatorname{Var}X-\operatorname{Var}Y$; forgetting $|J|$ or using the Jacobian of the forward map instead of the inverse; treating $E[X\mid Y]$ as a number; assuming the maximum and minimum of an iid sample are independent.

Suggested homework. I(1, 2, 6, 8), II(1, 3, 4), III(1, 2, 3, 5), IV(2, 4, 6, 8), V(1, 4, 6); Exam corner E1, E5, E6.

🔬 Research–Teaching NexusResearch-tutored · optional, about 60 minutes

Research question. Patients enrolled in a trial because their blood pressure was high often improve at the next visit even without treatment. How much of an observed "improvement" is just regression to the mean?

Why it matters. Clinical trials of hypertension, cholesterol and diabetes drugs, sports "Sophomore slumps", speed cameras installed at accident black spots, and school remedial programmes for low scorers all select units because a first measurement was extreme. If repeated measurements are positively but imperfectly correlated, the second measurement drifts back towards the mean by itself. Regulators such as the WHO and national drug agencies insist on randomised control groups largely for this reason.

What to do.

  1. Model two systolic readings $(X_1,X_2)$ of the same untreated patient as bivariate normal with common mean $\mu=140$ mmHg, common SD $\sigma=15$ mmHg and correlation $\rho=0.7$.
  2. Patients are enrolled if $X_1\gt160$. Using Theorem 1.42(2) and the tower property, prove that $E[X_2\mid X_1\gt c]=\mu+\rho\,(E[X_1\mid X_1\gt c]-\mu)$, and that $E[X_1\mid X_1\gt c]=\mu+\sigma\lambda(z)$ with $z=\frac{c-\mu}{\sigma}$ and $\lambda(z)=\frac{\varphi(z)}{1-\Phi(z)}$. Deduce the expected "improvement" $(1-\rho)\sigma\lambda(z)$.
  3. Compute it for $\rho=0.5,0.7,0.9$, and check one case by simulating $10^5$ patients with a seeded generator (use the construction of Theorem 1.42(4); Interactive 2 shows the same geometry).
  4. Explain why a randomised control group, selected by the same rule, removes this bias from the estimated treatment effect.

Tools: Python (scipy for $\Phi$ and $\varphi$) or a spreadsheet; Interactive 2 on this page.

Hand in: A one-page memo for a trial committee: the formula, a small table of the expected spurious improvement for the three correlations, the simulation check, and a recommendation on trial design.

Going further. Read about Galton's 1886 study of the heights of parents and children, where the word "regression" was born, and about how regression to the mean is estimated and corrected in observational studies of road safety.

Instructor note: with $\mu=140$, $\sigma=15$, $c=160$, about $9.1\%$ of patients are enrolled ($z=\tfrac43$, $\lambda(z)\approx1.798$). Their mean first reading is about $167.0$ mmHg, the expected second reading is about $158.9$ mmHg for $\rho=0.7$, and the spurious improvement is about $8.1$ mmHg. For $\rho=0.5$ and $0.9$ it is about $13.5$ and $2.7$ mmHg. Simulations with $10^5$ patients typically agree to within about $0.2$ mmHg.

🧪 Self-check quiz

0/5 correct

Q1. $(X,Y)$ has joint density $4xy$ on $(0,1)^2$. Then $X$ and $Y$ are

Q2. $\operatorname{Var}(X)=1$, $\operatorname{Var}(Y)=4$ and $\operatorname{Cov}(X,Y)=1$. Then $\rho(X,Y)$ is

Q3. For integrable $X$, the random variable $E[X\mid Y]$ always satisfies

Q4. $X$ and $Y$ are iid exponential with rate $\lambda$. Then $\min(X,Y)$ is

Q5. $(X,Y)$ is standard bivariate normal with $\rho=0.8$. Given $X=1$, $Y$ has