1.3 Random Variables, Expectation and Moments
Random variables turn outcomes into numbers. Distribution functions and their jumps, discrete, continuous and mixed laws, quantiles, change of variables, expectation through the tail formula, moments, and the probability and moment generating functions that store them all.
A probability space describes chance, but we rarely care about the outcome $\omega$ itself. A clinic records the waiting time of a patient, an insurer the size of a claim, a quality engineer the number of defective items in a lot. Each is a numerical function of the outcome, a random variable. Its whole probabilistic behaviour is captured by one real function of a real variable, the distribution function $F(x)=P(X\le x)$. Once we have $F$, the sample space can be forgotten. Expectations, variances, quantiles and probabilities of intervals are all read off from $F$, and two random variables with the same $F$ cannot be told apart by any probability calculation.
This section develops that toolkit with full proofs. We characterise distribution functions and classify them into discrete, continuous and mixed types. We introduce the quantile function and use it to simulate any law from a uniform random number, and we derive the change-of-variables formula for the density of $g(X)$. We define expectation carefully, including when it fails to exist, and prove the tail formula $E[X]=\int_0^\infty P(X\gt x)\,dx$, which turns many hard expectations into easy integrals. Finally we study moments and the probability and moment generating functions. CSIR Part B questions often ask for one of these numbers (a mean from a mixed cdf, a variance from an mgf, a density after a transformation), and Part C questions test the boundary cases: functions that are almost but not quite distribution functions, symmetric laws with no mean, and moments that exist without an mgf.
Random Variables and Distribution Functions
Let $(\Omega,\mathcal{F},P)$ be a probability space. A random variable is a function $X:\Omega\to\mathbb{R}$ such that $\{X\le x\}=\{\omega:X(\omega)\le x\}\in\mathcal{F}$ for every $x\in\mathbb{R}$. Its distribution (or law) is the probability measure $P_X(B)=P(X\in B)$ on the Borel sets $B\in\mathcal{B}(\mathbb{R})$.
The condition guarantees that $\{X\in B\}$ is an event for every Borel set $B$. Indeed, the collection of sets $B\subseteq\mathbb{R}$ with $X^{-1}(B)\in\mathcal{F}$ is a σ-field, since inverse images commute with complements and countable unions. It contains every half-line $(-\infty,x]$, and these generate $\mathcal{B}(\mathbb{R})$. For the same reason, $g(X)$ is a random variable for every Borel function $g$ (for example every continuous or monotone $g$).
Let $F$ be the cdf of $X$. Then:
- $F$ is nondecreasing;
- $F$ is right-continuous: $F(x+)=F(x)$ for every $x$;
- $\lim_{x\to-\infty}F(x)=0$ and $\lim_{x\to\infty}F(x)=1$;
- $P(X\lt x)=F(x-)$ and $P(X=x)=F(x)-F(x-)$, the size of the jump at $x$;
- $F$ has at most countably many discontinuities.
Conversely, every function $F:\mathbb{R}\to\mathbb{R}$ with properties 1–3 is the distribution function of some random variable (proved in Theorem 1.25 below).
Proof. (1) If $x\lt y$ then $\{X\le x\}\subseteq\{X\le y\}$, so $F(x)\le F(y)$ by monotonicity of $P$.
(2) Because $F$ is monotone it suffices to take $x_n=x+\tfrac1n$. The events $\{X\le x+\tfrac1n\}$ decrease to $\{X\le x\}$, so continuity from above (Theorem 1.14(5)) gives $F(x+\tfrac1n)\to F(x)$.
(3) The events $\{X\le -n\}$ decrease to $\varnothing$ and $\{X\le n\}$ increase to $\Omega$. Continuity gives the limits $0$ and $1$ along integers, and monotonicity extends them to real $x\to\pm\infty$.
(4) $\{X\le x-\tfrac1n\}$ increases to $\{X\lt x\}$, so $F(x-\tfrac1n)\to P(X\lt x)$ by continuity from below. Then $P(X=x)=P(X\le x)-P(X\lt x)$ $=F(x)-F(x-)$.
(5) Let $D_k$ be the set of points where the jump exceeds $\tfrac1k$. If $D_k$ contained $k$ distinct points $x_1,\dots,x_k$, the disjoint events $\{X=x_i\}$ would have total probability greater than $1$. So $|D_k|\lt k$, and the set of all discontinuities, $\bigcup_kD_k$, is a countable union of finite sets. ∎
(a) $X$ is discrete if $P(X\in S)=1$ for some countable set $S$. Its probability mass function (pmf) is $p(x)=P(X=x)$, and then $F(x)=\sum_{s\in S,\,s\le x}p(s)$ is a step function with jumps $p(s)$ at the points $s\in S$.
(b) $X$ is (absolutely) continuous with probability density function (pdf) $f$ if $f\ge0$ is integrable and $$F(x)=\int_{-\infty}^xf(t)\,dt\quad\text{for all }x.$$ Then $F$ is continuous, $\int_{-\infty}^\infty f=1$, $P(X\in B)=\int_Bf$ for Borel sets $B$, and $F'(x)=f(x)$ at every point where $f$ is continuous. The density is determined only up to changes on a set of length zero.
(c) $X$ is mixed if $F=\alpha F_d+(1-\alpha)F_c$ with $0\lt\alpha\lt1$, $F_d$ a discrete cdf and $F_c$ a cdf with a density.
Let $F$ be a cdf with jump points $x_1,x_2,\dots$ and jumps $p_i=F(x_i)-F(x_i-)$, and let $\alpha=\sum_ip_i\le1$. Then $F_J(x)=\sum_{i:\,x_i\le x}p_i$ and $F_C=F-F_J$ are both nondecreasing and right-continuous, and $F_C$ is continuous. Hence, when $0\lt\alpha\lt1$, $F=\alpha F_d+(1-\alpha)G$ with $F_d=F_J/\alpha$ a discrete cdf and $G=F_C/(1-\alpha)$ a continuous cdf.
(Statement only.) A continuous cdf $G$ splits further as $G=\beta G_{ac}+(1-\beta)G_s$, where $G_{ac}$ has a density and $G_s$ is singular: continuous, but with $G_s'=0$ except on a set of length zero. The Cantor function is the classic singular cdf.
Proof. $F_J$ is nondecreasing, and it is right-continuous because $F_J(x+h)-F_J(x)=\sum_{x\lt x_i\le x+h}p_i\to0$ as $h\downarrow0$ (the tail of a convergent series of nonnegative terms). For $x\lt y$, the jumps of $F$ in $(x,y]$ add up to at most the total increase: $$F_J(y)-F_J(x)=\sum_{x\lt x_i\le y}p_i\le F(y)-F(x).$$ (Finitely many disjoint events $\{X=x_i\}$ lie inside $\{x\lt X\le y\}$; let their number grow.) So $F_C=F-F_J$ is nondecreasing, and it is right-continuous as a difference of right-continuous functions. At each point, the jump of $F_C$ is the jump of $F$ minus the jump of $F_J$, which is $p_i-p_i=0$ at $x_i$ and $0-0$ elsewhere. So $F_C$ is continuous. The final claim is a rescaling, with $F_J(\infty)=\alpha$ and $F_C(\infty)=1-\alpha$. ∎
Quantiles and Functions of a Random Variable
A cdf answers "how likely is $X$ to be at most $x$?". The reverse question, "below which value does $X$ fall with probability $u$?", is answered by the quantile function. When $F$ is continuous and strictly increasing, it is simply $F^{-1}$. In general, $F$ can be flat (no value in an interval) or can jump (an atom), and we need a careful definition.
The quantile function of a cdf $F$ is $$Q(u)=F^{-1}(u)=\inf\{x\in\mathbb{R}:F(x)\ge u\},\qquad 0\lt u\lt1.$$
A number $x_p$ is a $p$-th quantile of $X$ if $P(X\lt x_p)\le p\le P(X\le x_p)$, i.e. $F(x_p-)\le p\le F(x_p)$. A median is a $\tfrac12$-quantile, and $Q(\tfrac14)$, $Q(\tfrac34)$ are the lower and upper quartiles. The set of $p$-th quantiles is a closed interval whose left endpoint is $Q(p)$. It is a single point unless $F$ is flat at level $p$.
Let $F$ satisfy properties 1–3 of Theorem 1.23, and let $Q$ be its quantile function.
- For $0\lt u\lt1$ and $x\in\mathbb{R}$: $\ Q(u)\le x\iff u\le F(x)$.
- (Quantile transform) If $U$ is uniform on $(0,1)$, then $Q(U)$ has cdf $F$. In particular every such $F$ is a distribution function.
- (Probability integral transform) If $X$ has a continuous cdf $F$, then $F(X)$ is uniform on $(0,1)$.
Proof. (1) The set $A_u=\{x:F(x)\ge u\}$ is nonempty (since $F(x)\to1\gt u$) and bounded below (since $F(x)\to0\lt u$), so $Q(u)=\inf A_u$ is a real number. Because $F$ is nondecreasing, $A_u$ is an interval unbounded to the right. Because $F$ is right-continuous, $A_u$ contains its infimum: take $x_n\in A_u$ with $x_n\downarrow Q(u)$; then $F(Q(u))=\lim F(x_n)\ge u$. So $A_u=[Q(u),\infty)$, and $x\in A_u\iff x\ge Q(u)$, which is the claim.
(2) By (1), $P(Q(U)\le x)=P(U\le F(x))=F(x)$, because $P(U\le c)=c$ for $c\in[0,1]$. To produce a random variable with cdf $F$, take $\Omega=(0,1)$ with length as the probability (the uniform law) and $X(\omega)=Q(\omega)$. $Q$ is nondecreasing, so $\{Q\le x\}$ is an interval and $X$ is a random variable.
(3) Fix $u\in(0,1)$. By (1), $F(X)\ge u\iff X\ge Q(u)$, so $P(F(X)\ge u)=1-F(Q(u)-)$. Since $F$ is continuous, $F(Q(u)-)=F(Q(u))$. Moreover $F(Q(u))\ge u$ by (1), while $F(x)\lt u$ for all $x\lt Q(u)$, and continuity gives $F(Q(u))\le u$. So $F(Q(u))=u$ and $P(F(X)\lt u)=u$ for all $u\in(0,1)$. Taking right limits in $u$, $P(F(X)\le u)=u$: $F(X)$ is uniform. ∎
Part (2) is how computers sample from any distribution: generate $U$ and return $Q(U)$. For the exponential law, $F(x)=1-e^{-\lambda x}$ gives $Q(u)=-\tfrac1\lambda\log(1-u)$. Part (3) is the basis of $p$-values (§5.1) and of the Kolmogorov–Smirnov test (§5.3). It fails when $F$ has jumps: if $X$ is a fair coin with values $0,1$, then $F(X)$ takes only the values $\tfrac12$ and $1$.
(a) Let $X$ have density $f_X$ that vanishes outside an open interval $I$. Let $g:I\to J$ be a bijection onto an open interval $J$ that is strictly monotone, with inverse $h=g^{-1}$ continuously differentiable on $J$. Then $Y=g(X)$ has density $$f_Y(y)=f_X(h(y))\,|h'(y)|\quad(y\in J),\qquad f_Y(y)=0\ \ (y\notin J).$$
(b) Suppose instead that $I$ is the union of disjoint open intervals $I_1,\dots,I_k$ (up to finitely many points) and $g$ is strictly monotone on each $I_j$, with continuously differentiable inverse $h_j$ on $g(I_j)$. Then $$f_Y(y)=\sum_{j:\ y\in g(I_j)}f_X(h_j(y))\,|h_j'(y)|.$$
(c) If $X$ is discrete, $P(g(X)=y)=\sum_{x:\,g(x)=y}P(X=x)$.
Proof. (a) Suppose $g$ is increasing, so $h$ is increasing and $h'\ge0$. For $y\in J$, $\{Y\le y\}=\{X\le h(y)\}$ up to an event of probability $0$ (the event $X\notin I$). Substituting $x=h(t)$, $$F_Y(y)=\int_{\inf I}^{h(y)}f_X(x)\,dx=\int_{\inf J}^{y}f_X(h(t))\,h'(t)\,dt.$$ So $F_Y$ is the integral of $f_X(h)h'$, which is the claim. If $g$ is decreasing, $\{Y\le y\}=\{X\ge h(y)\}$ and $$F_Y(y)=\int_{h(y)}^{\sup I}f_X(x)\,dx=\int_{y}^{\sup J}f_X(h(t))\,(-h'(t))\,dt,$$ so $F_Y(y)=1-\int_y^{\sup J}f_X(h)|h'|$, whose derivative is $f_X(h(y))|h'(y)|$; again $F_Y$ is the integral of this function. (b) For a Borel set $B$, $P(Y\in B)=\sum_jP(X\in I_j,\ g(X)\in B)$. Apply (a) to the restriction of $X$ to each $I_j$ and add. (c) Group the atoms of $X$ by their image. ∎
- Find the range of $Y$. Outside it, $f_Y=0$.
- If $g$ is monotone on the support, use $f_X(h(y))|h'(y)|$. Do not forget the absolute value when $g$ decreases.
- If $g$ is not monotone, split the support into monotone pieces and add one term per preimage. Points $y$ with two preimages get two terms, points with one get one term.
- When in doubt, work with the cdf: write $P(Y\le y)$ as a probability about $X$ and differentiate. This always works, and it is the only safe method for mixed variables (e.g. $\max(X,0)$).
- Check that $\int f_Y=1$.
Expectation, Variance and Moments
The expectation of $X$ is its probability-weighted average value, the centre of mass of its distribution. The definition must say what happens when the weighted sum or integral does not converge absolutely, because rearranging a conditionally convergent series can change its sum, and an average should not depend on the order in which we list the outcomes.
(a) If $X$ is discrete with pmf $p$, then $E[X]=\sum_xx\,p(x)$, provided $\sum_x|x|\,p(x)\lt\infty$.
(b) If $X$ has density $f$, then $E[X]=\int_{-\infty}^\infty x\,f(x)\,dx$, provided $\int|x|f(x)\,dx\lt\infty$.
(c) For a mixed law, $E[X]$ is the sum over the atoms plus the integral against the density part. In general $E[X]=\int_\Omega X\,dP=\int_{\mathbb{R}}x\,dF(x)$ is a Lebesgue (or Lebesgue–Stieltjes) integral.
$X$ is integrable if $E|X|\lt\infty$. For $X\ge0$ we always allow $E[X]=\infty$. In general, with $X^+=\max(X,0)$ and $X^-=\max(-X,0)$, $E[X]=E[X^+]-E[X^-]$ is defined when at least one of the two is finite, and does not exist when both are infinite.
(a) If $X\ge0$, then $$E[X]=\int_0^\infty P(X\gt x)\,dx=\int_0^\infty\left(1-F(x)\right)dx,$$ where both sides may be $+\infty$. More generally $E[X^r]=\int_0^\infty r\,x^{r-1}P(X\gt x)\,dx$ for $r\gt0$.
(b) If $X$ takes values in $\{0,1,2,\dots\}$, then $E[X]=\sum_{n=1}^\infty P(X\ge n)=\sum_{n=0}^\infty P(X\gt n)$.
(c) For a general integrable $X$, $$E[X]=\int_0^\infty\left(1-F(x)\right)dx-\int_{-\infty}^0F(x)\,dx.$$
(d) If $X\ge0$ and $E[X]\lt\infty$, then $x\,P(X\gt x)\to0$ as $x\to\infty$. The converse is false.
Proof. (a) For every outcome, $X(\omega)=\int_0^\infty\mathbf{1}\{X(\omega)\gt x\}\,dx$, since the integrand is $1$ exactly on $[0,X(\omega))$. Take expectations and interchange the expectation with the $dx$-integral. This is legitimate by Tonelli's theorem, because the integrand is nonnegative: $$E[X]=E\int_0^\infty\mathbf{1}\{X\gt x\}\,dx=\int_0^\infty E\left[\mathbf{1}\{X\gt x\}\right]dx=\int_0^\infty P(X\gt x)\,dx.$$ For a density the interchange is the familiar swap of the order of integration, $\int_0^\infty\int_x^\infty f(t)\,dt\,dx=\int_0^\infty f(t)\int_0^tdx\,dt=\int_0^\infty tf(t)\,dt$. For $X^r$, use $X^r=\int_0^\infty rx^{r-1}\mathbf{1}\{X\gt x\}\,dx$ in the same way.
(b) For integer-valued $X\ge0$, $P(X\gt x)=P(X\ge n)$ for $x\in[n-1,n)$, so the integral in (a) is the sum of the $P(X\ge n)$. Directly: $\sum_{n\ge1}P(X\ge n)=\sum_{n\ge1}\sum_{k\ge n}p(k)$ $=\sum_{k\ge1}k\,p(k)$, swapping the order of a double series of nonnegative terms.
(c) Apply (a) to $X^+$ and $X^-$. $P(X^+\gt x)=1-F(x)$ for $x\ge0$, and $P(X^-\gt x)=P(X\lt-x)=F((-x)-)$. Since $F(t-)=F(t)$ except at countably many $t$, $\int_0^\infty F((-x)-)\,dx=\int_{-\infty}^0F(t)\,dt$.
(d) For $x\gt0$, $x\,\mathbf{1}\{X\gt x\}\le X\,\mathbf{1}\{X\gt x\}$, so $x\,P(X\gt x)\le E\left[X\mathbf{1}\{X\gt x\}\right]$. The right side tends to $0$ by dominated convergence (it is bounded by $X$, which is integrable, and tends to $0$ pointwise). A counterexample to the converse is in Example 6(d). ∎
- (Law of the unconscious statistician) For a Borel function $g$, $$E[g(X)]=\sum_xg(x)\,p(x)\quad\text{or}\quad E[g(X)]=\int g(x)f(x)\,dx,$$ whenever the sum or integral converges absolutely. We never need the distribution of $g(X)$ itself.
- (Linearity) $E[aX+bY]=aE[X]+bE[Y]$ for integrable $X,Y$ defined on the same space, and $E[c]=c$.
- (Monotonicity) If $X\le Y$ then $E[X]\le E[Y]$; in particular $|E[X]|\le E|X|$.
- If $X\ge0$ and $E[X]=0$, then $P(X=0)=1$.
Proof. (1) Discrete case. Group the atoms of $X$ by the value of $g$: $$\sum_yy\,P(g(X)=y)=\sum_yy\sum_{x:\,g(x)=y}p(x)=\sum_xg(x)\,p(x).$$ The regrouping is valid because the series converges absolutely. Density case, $g\ge0$. Apply the tail formula to $g(X)$ and swap integrals (Tonelli): $$E[g(X)]=\int_0^\infty P(g(X)\gt t)\,dt=\int_0^\infty\int\mathbf{1}\{g(x)\gt t\}f(x)\,dx\,dt=\int f(x)\int_0^{g(x)}dt\,dx=\int g(x)f(x)\,dx.$$ For general $g$, write $g=g^+-g^-$. (2) and (3) are linearity and monotonicity of the Lebesgue integral; for a discrete space $E[X]=\sum_\omega X(\omega)P(\{\omega\})$, and both properties are visible. The second statement in (3) follows from $-|X|\le X\le|X|$. (4) For each $n$, $X\ge\tfrac1n\mathbf{1}\{X\gt\tfrac1n\}$, so $0=E[X]\ge\tfrac1nP(X\gt\tfrac1n)$. Hence $P(X\gt\tfrac1n)=0$ for all $n$, and $P(X\gt0)=0$ by continuity from below. ∎
For $r\gt0$, when the expectations exist: the $r$-th raw moment is $\mu_r'=E[X^r]$, the $r$-th absolute moment is $E|X|^r$, and with $\mu=E[X]$ the $r$-th central moment is $\mu_r=E[(X-\mu)^r]$. The variance is $\operatorname{Var}(X)=\sigma^2=\mu_2$ and $\sigma$ is the standard deviation. For integer-valued $X$, the $r$-th factorial moment is $E[X(X-1)\cdots(X-r+1)]$.
The coefficients of skewness and kurtosis are $\gamma_1=\mu_3/\sigma^3$ and $\beta_2=\mu_4/\sigma^4$, exactly as for data in §1.1; for a normal law $\gamma_1=0$ and $\beta_2=3$.
Let $E[X^2]\lt\infty$ and $\mu=E[X]$.
- $\operatorname{Var}(X)=E[X^2]-\mu^2$, and $\operatorname{Var}(aX+b)=a^2\operatorname{Var}(X)$.
- $E[(X-c)^2]=\operatorname{Var}(X)+(\mu-c)^2$, so $c=\mu$ uniquely minimises the mean squared deviation.
- If $E|X|\lt\infty$, then $c\mapsto E|X-c|$ is minimised exactly at the medians of $X$.
- $\operatorname{Var}(X)=0$ if and only if $P(X=\mu)=1$.
- (Lower moments exist) If $E|X|^s\lt\infty$ and $0\lt r\lt s$, then $E|X|^r\lt\infty$.
Proof. (1) Expand $(X-\mu)^2=X^2-2\mu X+\mu^2$ and use linearity: $E[X^2]-2\mu^2+\mu^2$. Also $aX+b-(a\mu+b)=a(X-\mu)$.
(2) $E[(X-c)^2]=E[(X-\mu)^2]$ $+2(\mu-c)E[X-\mu]+(\mu-c)^2$, and the middle term is $0$.
(3) Write $|X-c|=(X-c)^++(c-X)^+$ and apply the tail formula to each part: $$E(X-c)^+=\int_0^\infty P(X\gt c+t)\,dt=\int_c^\infty\left(1-F(s)\right)ds,$$ $$E(c-X)^+=\int_0^\infty P(X\lt c-t)\,dt=\int_{-\infty}^cF(s)\,ds,$$ using again that $F(s-)=F(s)$ except at countably many points. Let $m$ be a median and $\phi(c)=E|X-c|$. Subtracting the two expressions, for $c\gt m$, $$\phi(c)-\phi(m)=\int_m^c\left(2F(s)-1\right)ds\ge0,$$ since $F(s)\ge F(m)\ge\tfrac12$ for $s\ge m$; for $c\lt m$, $$\phi(c)-\phi(m)=\int_c^m\left(1-2F(s)\right)ds\ge0,$$ since $F(s)\le F(m-)\le\tfrac12$ for $s\lt m$. So every median is a minimiser. Conversely, if $c$ is not a median, then either $F(c)\lt\tfrac12$ (and $\phi$ strictly decreases just to the right of $c$, because $2F-1\lt0$ on some $[c,c+\delta)$ by right-continuity) or $F(c-)\gt\tfrac12$ (and $\phi$ strictly decreases as we move left from $c$). This is the population version of Theorem 1.2 of §1.1.
(4) If $\operatorname{Var}(X)=E[(X-\mu)^2]=0$, Theorem 1.28(4) applied to $(X-\mu)^2\ge0$ gives $X=\mu$ almost surely. The converse is immediate.
(5) $|x|^r\le1+|x|^s$ for all $x$ (consider $|x|\le1$ and $|x|\gt1$ separately), so $E|X|^r\le1+E|X|^s$. ∎
In particular, a finite variance forces a finite mean, and a finite fourth moment forces finite moments of every lower order. The sharper inequality $\left(E|X|^r\right)^{1/r}\le\left(E|X|^s\right)^{1/s}$ (Lyapunov) follows from Jensen's inequality in §2.2.
Let $X$ be symmetric about $c$. Then:
- $c$ is a median of $X$;
- if $E|X|\lt\infty$ then $E[X]=c$, and every odd central moment $E[(X-c)^{2k+1}]$ that exists is $0$;
- if $c=0$ and $F$ is continuous, then $F(-x)=1-F(x)$, and $P(|X|\le x)=2F(x)-1$ for $x\ge0$.
Proof. Put $W=X-c$, so $W$ and $-W$ have the same law. (1) $P(W\lt0)=P(-W\lt0)=P(W\gt0)$, and these two numbers add up to at most $1$, so each is $\le\tfrac12$. Hence $P(X\lt c)\le\tfrac12$ and $P(X\le c)=1-P(X\gt c)\ge\tfrac12$: $c$ is a median. (2) $E[W]=E[-W]=-E[W]$, so $E[W]=0$; likewise $E[W^{2k+1}]=E[(-W)^{2k+1}]=-E[W^{2k+1}]$ when the moment exists. (3) $F(-x)=P(X\le-x)=P(-X\ge x)$ $=P(X\ge x)=1-F(x)$, using continuity at the last step; then $P(-x\le X\le x)=F(x)-F(-x)=2F(x)-1$. ∎
The hypothesis $E|X|\lt\infty$ in (2) cannot be dropped. The Cauchy law is symmetric about $0$, yet it has no mean at all (Example 6). Conversely, a vanishing third central moment does not imply symmetry (Exercise III.8).
Generating Functions
A generating function packs a whole sequence of probabilities or moments into one function. Differentiating it recovers them, multiplying generating functions corresponds to adding independent random variables (§1.4), and their limits detect convergence in distribution (§2.2). Two kinds are used throughout Unit 4.
(a) If $X$ takes values in $\{0,1,2,\dots\}$ with $p_k=P(X=k)$, its probability generating function (pgf) is $$G_X(s)=E\left[s^X\right]=\sum_{k=0}^\infty p_ks^k,\qquad |s|\le1.$$
(b) The moment generating function (mgf) of $X$ is $M_X(t)=E\left[e^{tX}\right]$ for those real $t$ where it is finite. We say $X$ has an mgf if $M_X(t)\lt\infty$ for all $t$ in some interval $(-h,h)$, $h\gt0$. The cumulant generating function is $K_X(t)=\log M_X(t)$. For integer-valued $X$, $M_X(t)=G_X(e^t)$.
- The power series converges for $|s|\le1$ and $G_X(1)=1$. Moreover $p_k=\dfrac{G_X^{(k)}(0)}{k!}$, so $G_X$ determines the distribution.
- For $r\in\mathbb{N}$, the factorial moment is $E[X(X-1)\cdots(X-r+1)]=\lim_{s\uparrow1}G_X^{(r)}(s)$, finite or $+\infty$. In particular $$E[X]=G_X'(1),\qquad\operatorname{Var}(X)=G_X''(1)+G_X'(1)-G_X'(1)^2.$$
- For constants $a,b\in\{0,1,2,\dots\}$, $G_{aX+b}(s)=s^bG_X(s^a)$.
Proof. (1) $\sum p_k|s|^k\le\sum p_k=1$ for $|s|\le1$, so the radius of convergence is at least $1$. Inside the disc a power series can be differentiated term by term, and $G^{(k)}(0)=k!\,p_k$. (2) For $0\le s\lt1$, $$G^{(r)}(s)=\sum_{k\ge r}k(k-1)\cdots(k-r+1)\,p_k\,s^{k-r}.$$ The terms are nonnegative and increase with $s$, so by monotone convergence the limit as $s\uparrow1$ is $\sum_kk(k-1)\cdots(k-r+1)p_k$. For the variance, $E[X^2]=E[X(X-1)]+E[X]$. (3) $E[s^{aX+b}]=s^bE[(s^a)^X]$. ∎
Suppose $M_X(t)\lt\infty$ for $|t|\lt h$, where $h\gt0$. Then:
- $E|X|^r\lt\infty$ for every $r\gt0$;
- $M_X(t)=\sum_{r=0}^\infty\dfrac{t^r}{r!}E[X^r]$ for $|t|\lt h$, and hence $E[X^r]=M_X^{(r)}(0)$;
- $M_{aX+b}(t)=e^{bt}M_X(at)$;
- $K_X'(0)=E[X]$ and $K_X''(0)=\operatorname{Var}(X)$;
- (Uniqueness, statement) if $M_X(t)=M_Y(t)$ for all $t$ in a neighbourhood of $0$, then $X$ and $Y$ have the same distribution.
Proof. Fix $0\lt t_0\lt h$. Then $e^{t_0|x|}\le e^{t_0x}+e^{-t_0x}$, so $E\left[e^{t_0|X|}\right]\le M(t_0)+M(-t_0)\lt\infty$. (1) $|x|^r/e^{t_0|x|}$ is bounded on $\mathbb{R}$ (it tends to $0$ at $\pm\infty$), so $E|X|^r\le C_r\,E\left[e^{t_0|X|}\right]\lt\infty$. (2) For $|t|\le t_0$, the partial sums of $e^{tX}=\sum_r\frac{(tX)^r}{r!}$ are dominated by $\sum_r\frac{|tX|^r}{r!}=e^{|t||X|}\le e^{t_0|X|}$, which is integrable. By dominated convergence we may take expectations term by term: $M(t)=\sum_r\frac{t^r}{r!}E[X^r]$. Since $t_0\lt h$ was arbitrary, this power series converges on $(-h,h)$, and differentiating it $r$ times at $0$ gives $E[X^r]$. (3) $E[e^{t(aX+b)}]=e^{bt}E[e^{(at)X}]$. (4) $K'=M'/M$ and $K''=(M''M-M'^2)/M^2$. At $t=0$, $M(0)=1$, $M'(0)=E[X]$ and $M''(0)=E[X^2]$. (5) The proof uses the characteristic function and its inversion formula (§2.2). ∎
- A continuous cdf need not have a density. The Cantor function is a continuous cdf whose derivative is $0$ except on the Cantor set, a set of length $0$. So $\int F'=0\ne1$. In exam language, "continuous random variable" means "has a density".
- Right-continuous, not left-continuous. $P(X\lt x)=F(x-)$, which differs from $F(x)$ at an atom. A function such as $\mathbf{1}\{x\gt0\}$ is not a cdf (Example 2(c)).
- A density is not a probability. $f(x)=\frac{1}{2\sqrt x}$ on $(0,1)$ is a density (of $U^2$) that is unbounded, and $f(x)=10$ on $(0,0.1)$ exceeds $1$. Only $\int_Bf$ is a probability.
- Symmetric does not mean "mean equals centre". The Cauchy law is symmetric about $0$ and has no mean (Example 6(b)). Symmetry gives $E[X]=c$ only when $E|X|\lt\infty$.
- Finite values, infinite mean. $P(X=k)=\frac1{k(k+1)}$ has $E[X]=\infty$. A Pareto law with $\alpha=2$ has a mean but no variance.
- $x\,P(X\gt x)\to0$ does not give a finite mean (Example 6(d)). It is necessary, not sufficient.
- $E[g(X)]\ne g(E[X])$ in general. For $X$ uniform on $\{1,2\}$, $E[1/X]=\tfrac34$ but $1/E[X]=\tfrac23$. Equality for all $X$ holds only for affine $g$.
- All moments finite, but no mgf. For the lognormal $X=e^Z$ ($Z$ standard normal), $E[X^r]=e^{r^2/2}$ for every $r$, but $E[e^{tX}]=\infty$ for every $t\gt0$. Moreover the lognormal moments are shared by other distributions, so moments need not determine a law. Moments do determine it when an mgf exists (Theorem 1.32(5)).
- The one-branch change-of-variables formula. For non-monotone $g$, every preimage contributes. For $Y=X^2$ with $X$ uniform on $(-1,2)$, using only $x=\sqrt y$ gives "density" of total mass $\tfrac23$ (Example 3(c)).
- $F(X)$ is uniform only for continuous $F$. For a fair coin with values $0,1$, $F(X)\in\{\tfrac12,1\}$. The quantile transform $Q(U)\sim F$, by contrast, works for every $F$.
- Medians need not be unique, and $Q(\tfrac12)$ is only the smallest one: for three fair coin tosses every $m\in[1,2]$ is a median (Example 4(b)).
- $\operatorname{Var}(aX+b)=a^2\operatorname{Var}(X)$, not $a\operatorname{Var}(X)+b$. Shifting does not change spread, and the scale factor is squared.
Exercises
- 1 $F(x)=0$ for $x\lt0$, $\tfrac x4$ for $0\le x\lt1$, $\tfrac12$ for $1\le x\lt2$, and $1$ for $x\ge2$. Find $P(X=1)$, $P(X=2)$, $P(\tfrac12\lt X\le\tfrac32)$, $P(X\lt1)$, $P(1\le X\lt2)$ and $E[X]$.
- 2 Find all constants $a,b$ for which $F(x)=a+b\arctan x$ is a distribution function.
- 3 Show that if $F$ is a cdf and $n\in\mathbb{N}$, then $F^n$ and $1-(1-F)^n$ are cdfs. (§1.4 interprets them as the laws of a maximum and a minimum.)
- 4 $X$ has a continuous cdf $F$. Find the cdf of $Y=\max(X,0)$ and decide whether $Y$ is discrete, continuous or mixed.
- 5 Let $q_1,q_2,\dots$ enumerate the rationals. Show that $F(x)=\sum_{n:\,q_n\le x}2^{-n}$ is a cdf that is discontinuous at every rational number.
- 6 Prove that $P(X=x)=0$ for every $x$ if and only if $F$ is continuous.
- 7 Find $c$ so that $f(x)=cx(1-x)$ on $(0,1)$ is a density. Find $F$, $P(X\gt\tfrac12)$ and $P(\tfrac14\lt X\lt\tfrac34)$.
- 8 Find $c$ so that $p(k)=\dfrac{c}{k(k+1)(k+2)}$, $k\in\mathbb{N}$, is a pmf. Find $E[X]$, and decide whether $E[X^2]$ is finite.
- 1 $X$ is uniform on $(0,1)$. Find the density of $Y=1/X$, and show that $E[Y]=\infty$.
- 2 $X$ is exponential with rate $1$. Find the densities of $\sqrt X$ and of $e^{-X}$.
- 3 $X$ has density $2x$ on $(0,1)$. Show that $-\log X$ is exponential and find its rate.
- 4 $Z$ is standard normal. Find the density of $|Z|$ and compute $E|Z|$.
- 5 $X$ is uniform on $(-\tfrac\pi2,\tfrac\pi2)$. Find the density of $\sin X$.
- 6 Find the quantile function and the median of the Pareto law $F(x)=1-x^{-\alpha}$, $x\ge1$ ($\alpha\gt0$), and explain how to simulate it from a uniform random number.
- 7 $P(X=0)=0.2$, $P(X=1)=0.3$, $P(X=3)=0.5$. Find $Q(0.2)$, $Q(0.25)$, $Q(0.5)$, $Q(0.9)$ and the set of all medians.
- 8 $X$ has a continuous, strictly increasing cdf $F$. Show that $-2\log F(X)$ has density $\tfrac12e^{-y/2}$, $y\gt0$. (This is Fisher's device for combining $p$-values.)
- 1 $X$ has density $6x(1-x)$ on $(0,1)$. Find $E[X]$ and $\operatorname{Var}(X)$.
- 2 Using the tail formula, find $E[X]$ for $X$ uniform on $(0,\theta)$, and $E[\min(X,c)]$ for $X$ exponential with rate $\lambda$ and a constant $c\gt0$.
- 3 For the Pareto density $f(x)=\alpha x^{-\alpha-1}$, $x\gt1$, show that $E[X^r]\lt\infty$ iff $r\lt\alpha$, and find $E[X]$ ($\alpha\gt1$) and $\operatorname{Var}(X)$ ($\alpha\gt2$).
- 4 Three fair dice are rolled. Use $E[X]=\sum_{n\ge1}P(X\ge n)$ to find the expected value of the smallest score.
- 5 For $X$ exponential with rate $1$, find $\min_cE|X-c|$ and the minimising $c$.
- 6 $X$ takes values in $[0,1]$ and has mean $\mu$. Prove that $\operatorname{Var}(X)\le\mu(1-\mu)\le\tfrac14$, and find when equality holds in the first inequality.
- 7 (Insurance) A claim $X$ is exponential with mean $\text{₹}\,2000$. The policy has a deductible of $\text{₹}\,500$, so it pays $Y=\max(X-500,0)$. Find $P(Y=0)$ and $E[Y]$.
- 8 Show that if $X$ is symmetric about $0$ and $E|X|^3\lt\infty$, then $E[X^3]=0$. Then give an $X$ with $E[X]=0$ and $E[X^3]\ne0$, and an $X$ with $E[X]=E[X^3]=0$ that is not symmetric.
- 1 Find the pgf of $X$ uniform on $\{0,1,\dots,n\}$ and deduce $E[X]$.
- 2 $P(X=k)=e^{-\lambda}\lambda^k/k!$, $k\ge0$ (Poisson). Find $G_X$, all factorial moments, $E[X]$ and $\operatorname{Var}(X)$.
- 3 $X$ has pgf $G(s)=\dfrac{1}{2-s}$. Find $P(X=k)$, $E[X]$ and $\operatorname{Var}(X)$.
- 4 Which of the following are pgfs? (a) $\tfrac12(s+s^2)$; (b) $\frac{1-s}{1+s}$; (c) $s\,e^{s-1}$; (d) $\frac{2}{3-s}$.
- 5 $X$ is exponential with rate $\lambda$. Find $M_X$ and its domain, and deduce $E[X^r]=r!/\lambda^r$.
- 6 $X$ is uniform on $(a,b)$. Find $M_X$, and from it the mean and variance.
- 7 Identify the distribution with mgf $M(t)=\dfrac{e^t}{2-e^t}$, $t\lt\log2$, and find its mean and variance.
- 8 Show that the Cauchy law has $M(t)=\infty$ for every $t\ne0$. Show that the Laplace density $\tfrac12e^{-|x|}$ has mgf $\frac{1}{1-t^2}$ for $|t|\lt1$, and find its variance and kurtosis $\beta_2$.
E1. CSIR Part B A random variable has cdf $F(x)=0$ for $x\lt0$, $F(x)=\frac{1+x}{3}$ for $0\le x\lt1$, and $F(x)=1$ for $x\ge1$. Then $E[X]$ equals (A) $\tfrac13$; (B) $\tfrac12$; (C) $\tfrac23$; (D) $\tfrac56$.
E2. CSIR Part B GATE ST $U$ is uniform on $(0,1)$ and $Y=-2\log U$. Then $P(Y\gt2)$ equals (A) $e^{-2}$; (B) $e^{-1}$; (C) $1-e^{-1}$; (D) $\tfrac12$.
E3. CSIR Part B $X$ has mgf $M(t)=(1-3t)^{-2}$ for $t\lt\tfrac13$. Then $\operatorname{Var}(X)$ is (A) $6$; (B) $9$; (C) $18$; (D) $54$.
E4. CSIR Part C Which of the following functions are distribution functions? (One or more options may be correct.) (A) $F(x)=e^{-e^{-x}}$, $x\in\mathbb{R}$. (B) $F(x)=0$ for $x\lt0$ and $F(x)=1-\tfrac12e^{-x}$ for $x\ge0$. (C) $F(x)=0$ for $x\le0$ and $F(x)=1$ for $x\gt0$. (D) $F(x)=0$ for $x\lt0$ and $F(x)=1-e^{-x}(1+\sin x)$ for $x\ge0$.
E5. CSIR Part C Let $X$ be a nonnegative random variable with cdf $F$. Which of the following are true? (A) $E[X]=\int_0^\infty(1-F(x))\,dx$, both sides possibly infinite. (B) If $E[X]\lt\infty$ then $x\,P(X\gt x)\to0$ as $x\to\infty$. (C) If $x\,P(X\gt x)\to0$ then $E[X]\lt\infty$. (D) If $X$ is integer-valued, then $E[X]=\sum_{n=0}^\infty P(X\gt n)$.
E6. CSIR Part C Which of the following are true? (A) If $X$ is symmetric about $0$, then $E[X]=0$. (B) If $E[e^{tX}]\lt\infty$ for all $|t|\lt h$, for some $h\gt0$, then $E|X|^r\lt\infty$ for every $r\gt0$. (C) If $E|X|^r\lt\infty$ for every $r\gt0$, then $E[e^{tX}]\lt\infty$ for all $t$ in some neighbourhood of $0$. (D) If $X$ has a continuous cdf $F$, then $F(X)$ is uniform on $(0,1)$.
Suggested plan: 3 lectures × 60 min.
- Distribution functions. Definition 1.16 with the "inverse images form a σ-field" argument. Theorem 1.23 with full proof (each property is one use of continuity of $P$). The interval rule cards. Types of distributions, Theorem 1.24, Example 1 drawn on the board. Example 2 as a class poll: students vote on each function before you reveal the answer. Project Interactive 1 in the "mixed" and "binomial" modes.
- Quantiles and transformations. Definition 1.19 and Theorem 1.25 (prove (1) carefully: right-continuity is exactly what makes the infimum attained). Simulation via $Q(U)$. Theorem 1.26 and the strategy box. Example 3, then Interactive 2 in the $X^2$ mode with the wrong formula visible. Example 4 on the board.
- Expectation and generating functions. Definition 1.20, the tail formula with the Tonelli proof (draw the region $\{(\omega,x):0\le x\lt X(\omega)\}$). Theorems 1.28 and 1.29 (prove the median property through the tail formula). Examples 5 and 6; Interactive 2 in the Cauchy mode to show a running mean that never settles. Theorems 1.31 and 1.32 with Example 7. Finish with E4 and E6 live.
Discussion questions. (1) Why does the definition of $E[X]$ insist on absolute convergence? Give a series whose sum changes when the outcomes are listed in a different order. (2) Two random variables on different probability spaces have the same cdf. In what sense are they "the same", and which questions can still tell them apart? (3) Is it possible for $E[X]$ to exist while $E[X^2]$ does not? For $E[X^2]$ to exist while $E[X]$ does not?
Misconceptions to watch for. Reading $f(x)$ as a probability; using $F(b)-F(a)$ for a closed interval when $F$ jumps at $a$; forgetting $|h'|$ or a second branch in change of variables; assuming every symmetric law has mean equal to its centre; assuming finite values imply a finite mean; confusing the pgf derivative at $1$ (moments) with the derivative at $0$ (probabilities); writing $\operatorname{Var}(aX+b)=a\operatorname{Var}(X)$.
Suggested homework. I(1, 2, 4, 7, 8), II(1, 3, 5, 7), III(2, 3, 5, 7), IV(2, 3, 4, 8); Exam corner E1, E4, E5.
Research question. When the largest losses dominate, as with insurance claims after floods and cyclones or with crashes in financial returns, does the average loss even exist, and how would we know from data?
Why it matters. Reinsurers, catastrophe-bond markets and national disaster funds price risk from historical losses. Losses from natural catastrophes, city sizes and file sizes on the internet are often modelled by Pareto tails $P(X\gt x)\approx Cx^{-\alpha}$. Example 6 and Exercise III.3 show that the mean exists only if $\alpha\gt1$ and the variance only if $\alpha\gt2$. If the true $\alpha$ is near $1$, a sample mean can be dominated by a single event, and a premium set from "the average" is meaningless.
What to do.
- Using the quantile transform $X=U^{-1/\alpha}$ (Exercise II.6) and a seeded random number generator, simulate $10^5$ Pareto losses for $\alpha=0.8$, $1.5$ and $3$.
- Plot the running mean $\bar X_n$ against $n$ for each $\alpha$, together with the true mean $\frac{\alpha}{\alpha-1}$ where it exists. Describe what you see in each case. Repeat with three different seeds.
- Plot $\log\hat P(X\gt x)$ against $\log x$ from the empirical survival function. Explain why the plot is close to a straight line of slope $-\alpha$, and estimate $\alpha$ by least squares on the largest $10\%$ of the data.
- Find a published data set of insured catastrophe losses (or use the daily returns of a stock index) and repeat step 3. Is $\alpha\gt2$ plausible?
Tools: Python or a spreadsheet; Interactive 2 in its Cauchy mode as a reference picture of a law with no mean.
Hand in: A two-page note with the three running-mean plots, your tail-index estimates, and a recommendation on whether a sample mean is a safe basis for pricing in your real data set.
Going further. Read about the Hill estimator of the tail index and about the "mean excess function" $E[X-u\mid X\gt u]$, which is linear in $u$ for a Pareto tail. Both are standard tools of extreme-value theory.
🧪 Self-check quiz
Q1. For any random variable with cdf $F$, $P(X=a)$ equals
Q2. $X$ takes values in $\{0,1,2,\dots\}$ and $P(X\ge n)=2^{-n}$ for $n\ge1$. Then $E[X]$ is
Q3. $X$ is uniform on $(0,1)$. The density of $Y=X^2$ on $(0,1)$ is
Q4. $X$ has pgf $G(s)=\tfrac14(1+s)^2$. Then $\operatorname{Var}(X)$ is
Q5. Which of these distributions has no mean?