Chapter 2 · Distributions and Limit Theorems

2.1 Standard Discrete and Continuous Distributions

The catalogue every statistician carries: Bernoulli trials and the binomial, geometric, negative binomial and hypergeometric laws; Poisson as the law of rare events; exponential, gamma and beta waiting times; the normal family and its heavy-tailed relatives — with their moments, generating functions and the relationships that tie them together.

Chapter 1 built the general machinery: distribution functions, densities, expectations, generating functions, joint laws and transformations. In practice, though, a small number of distributions do almost all the work. The number of defective chips in a sample is binomial or hypergeometric. The number of insurance claims in a month, of radioactive decays in a second, or of typing errors on a page is close to Poisson. Waiting times between such events are exponential, and the waiting time for the $r$-th event is gamma. Proportions are often given a beta law, measurement errors a normal law, incomes and particle sizes a lognormal law, and lifetimes of components a Weibull law. These families are not an arbitrary zoo. Each arises from a simple mechanism, such as independent trials, rare events, waiting or adding many small effects, and the mechanisms connect the families to one another.

This section develops each standard law from its mechanism and proves its basic properties: the pmf or density, mean, variance and generating function, and the characterising properties (memorylessness of the geometric and exponential laws, the Poisson limit of the binomial). Above all, it proves the relationships: sums of exponentials are gamma, Poisson counts and gamma waiting times describe the same process, gamma ratios are beta, beta integrals are binomial tails, and many families are closed under independent sums (the reproductive property). CSIR Part B questions regularly ask for one number from these facts, such as a Poisson probability, a gamma moment or a beta integral. Part C questions test the boundaries: is a sum of exponentials with different rates still gamma? Is the uniform law memoryless? Which laws have a mean at all? A summary table of all the laws is in the last teaching subsection.

Bernoulli Trials: Binomial, Geometric, Negative Binomial and Hypergeometric Laws

The simplest random experiment has two outcomes, conventionally "success" and "failure". Repeating it independently under identical conditions gives a sequence of Bernoulli trials. Four questions about such a sequence lead to four distributions: how many successes in $n$ trials (binomial), how many trials until the first success (geometric), how many failures before the $r$-th success (negative binomial), and, when the trials are draws without replacement from a finite population, how many successes in the sample (hypergeometric).

📘 Definition 2.1 — Bernoulli trials and the laws built from them

Let $0\lt p\lt1$ and $q=1-p$. A sequence $I_1,I_2,\dots$ of independent random variables with $P(I_j=1)=p$, $P(I_j=0)=q$ is a sequence of Bernoulli trials with success probability $p$. A single $I_j$ has the Bernoulli law $\operatorname{Ber}(p)$.

  1. The number of successes $X=I_1+\dots+I_n$ in $n$ trials has the binomial law $\operatorname{Bin}(n,p)$: $$P(X=k)=\binom nkp^kq^{n-k},\qquad k=0,1,\dots,n.$$
  2. The number $T$ of trials up to and including the first success has the geometric law $\operatorname{Geo}(p)$ on $\mathbb{N}$: $P(T=k)=q^{k-1}p$, $k=1,2,\dots$ The number of failures before the first success, $T-1$, is geometric on $\{0,1,2,\dots\}$ with $P(T-1=k)=q^kp$.
  3. For $r\in\mathbb{N}$, the number $Y$ of failures before the $r$-th success has the negative binomial law $\operatorname{NB}(r,p)$: $$P(Y=k)=\binom{k+r-1}{k}p^rq^k,\qquad k=0,1,2,\dots$$ The number of trials needed for $r$ successes is $Y+r$. More generally, for real $r\gt0$, $\operatorname{NB}(r,p)$ is the law with $P(Y=k)=\dfrac{\Gamma(k+r)}{k!\,\Gamma(r)}p^rq^k$.

The discrete uniform law on $\{1,\dots,n\}$ gives each value probability $\tfrac1n$. It has mean $\tfrac{n+1}{2}$ and variance $\tfrac{n^2-1}{12}$ (Exercise IV.1 of §1.3 finds the mean of the uniform law on $\{0,\dots,n\}$ with a pgf). Watch the conventions: some books count trials and some count failures for the geometric and negative binomial laws. Always check the support before using a formula.

📐 Theorem 2.1 — The binomial law

Let $X\sim\operatorname{Bin}(n,p)$. Then:

  1. $E[X]=np$, $\operatorname{Var}(X)=npq$, $G_X(s)=(q+ps)^n$ and $M_X(t)=(q+pe^t)^n$;
  2. (Reproductive property) if $X\sim\operatorname{Bin}(n,p)$ and $Y\sim\operatorname{Bin}(m,p)$ are independent, then $X+Y\sim\operatorname{Bin}(n+m,p)$;
  3. (Mode) with $m=(n+1)p$, the pmf increases strictly for $k\lt m$ and decreases strictly for $k\gt m$. So the mode is $\lfloor m\rfloor$ when $m$ is not an integer, and both $m-1$ and $m$ are modes when $m$ is an integer.

Proof. (1) By the binomial theorem the probabilities add up to $(q+p)^n=1$. Write $X=\sum_{j=1}^nI_j$ with independent Bernoulli trials. Each $I_j$ has mean $p$, variance $E[I_j^2]-p^2=p-p^2=pq$ and pgf $q+ps$. Expectations add, variances of independent variables add, and pgfs of independent variables multiply (Theorem 1.40(3) of §1.4), which gives $np$, $npq$ and $(q+ps)^n$. Put $s=e^t$ for the mgf.

(2) $G_{X+Y}(s)=(q+ps)^n(q+ps)^m=(q+ps)^{n+m}$, the pgf of $\operatorname{Bin}(n+m,p)$, and a pgf determines the law (Theorem 1.31 of §1.3). More directly: $X+Y$ counts the successes in $n+m$ Bernoulli trials.

(3) Write $b(k)=P(X=k)$. For $1\le k\le n$, $$\frac{b(k)}{b(k-1)}=\frac{n-k+1}{k}\cdot\frac pq.$$ This is greater than $1$ iff $(n-k+1)p\gt kq$, i.e. iff $(n+1)p\gt k(p+q)=k$. Likewise it equals $1$ iff $k=m$ and is less than $1$ iff $k\gt m$. So $b$ increases up to the largest integer below $m$ and decreases after it, with a tie $b(m-1)=b(m)$ when $m$ is an integer. ∎

📐 Theorem 2.2 — The geometric law and its lack of memory

(a) If $T\sim\operatorname{Geo}(p)$ on $\mathbb{N}$, then $P(T\gt n)=q^n$ for $n\ge0$, $E[T]=\tfrac1p$, $\operatorname{Var}(T)=\tfrac{q}{p^2}$, $G_T(s)=\dfrac{ps}{1-qs}$ for $|s|\lt\tfrac1q$, and $T$ is memoryless: $$P(T\gt m+n\mid T\gt m)=P(T\gt n)\qquad\text{for all integers }m,n\ge0.$$

(b) Conversely, let $T$ take values in $\mathbb{N}$ with $P(T\gt m+n)=P(T\gt m)\,P(T\gt n)$ for all integers $m,n\ge0$. Then $T$ is geometric with $p=P(T=1)$ (where $p=1$ means $T\equiv1$).

Proof. (a) $T\gt n$ iff the first $n$ trials are failures, so $P(T\gt n)=q^n$ by independence. Then $P(T\gt m+n\mid T\gt m)=q^{m+n}/q^m=q^n$. The pgf is $\sum_{k\ge1}q^{k-1}ps^k=ps\sum_{j\ge0}(qs)^j=\frac{ps}{1-qs}$. Differentiating, $G'(s)=\frac{p}{(1-qs)^2}$ and $G''(s)=\frac{2pq}{(1-qs)^3}$, so $E[T]=G'(1)=\tfrac1p$ and $$\operatorname{Var}(T)=G''(1)+G'(1)-G'(1)^2=\frac{2q}{p^2}+\frac1p-\frac1{p^2}=\frac{q}{p^2}.$$

(b) Let $g(n)=P(T\gt n)$, so $g(0)=1$ because $T\ge1$. The hypothesis with $m=1$ gives $g(n+1)=g(1)g(n)$, hence $g(n)=g(1)^n$ by induction. Put $q=g(1)=P(T\gt1)=1-P(T=1)$. Then $q\lt1$, because otherwise $g(n)=1$ for all $n$, which contradicts $g(n)\to0$. So $P(T=k)=g(k-1)-g(k)=q^{k-1}(1-q)$: $T$ is geometric with $p=1-q$. ∎

Memorylessness says that past failures carry no information about the future. After $m$ failed trials, the number of further trials needed is again $\operatorname{Geo}(p)$. This is the discrete twin of the exponential law (Theorem 2.9), and Exercise III.2 shows how the two are linked: the integer part of an exponential variable is geometric.

📐 Theorem 2.3 — The negative binomial law

Let $Y\sim\operatorname{NB}(r,p)$ with $r\gt0$.

  1. The probabilities add up to $1$ and $G_Y(s)=\left(\dfrac{p}{1-qs}\right)^r$ for $|s|\lt\tfrac1q$. Hence $E[Y]=\dfrac{rq}{p}$ and $\operatorname{Var}(Y)=\dfrac{rq}{p^2}$.
  2. For $r\in\mathbb{N}$, $Y$ has the law of $G_1+\dots+G_r$, where the $G_i$ are independent geometric variables on $\{0,1,\dots\}$ (failures before the first success). In particular, the pmf in Definition 2.1(3) is the law of the number of failures before the $r$-th success.
  3. (Reproductive property) If $Y_1\sim\operatorname{NB}(r_1,p)$ and $Y_2\sim\operatorname{NB}(r_2,p)$ are independent, then $Y_1+Y_2\sim\operatorname{NB}(r_1+r_2,p)$.

Proof. (1) The negative binomial series is $(1-x)^{-r}=\sum_{k\ge0}\frac{\Gamma(k+r)}{k!\,\Gamma(r)}x^k$ for $|x|\lt1$ and real $r\gt0$. It is the Taylor series of $(1-x)^{-r}$, whose $k$-th derivative at $0$ is $r(r+1)\cdots(r+k-1)=\frac{\Gamma(k+r)}{\Gamma(r)}$, and it converges for $|x|\lt1$ by the ratio test. Hence $$G_Y(s)=p^r\sum_k\frac{\Gamma(k+r)}{k!\,\Gamma(r)}(qs)^k=p^r(1-qs)^{-r},$$ and $s=1$ gives total mass $p^rp^{-r}=1$. With $K(s)=\log G_Y(s)=r\log p-r\log(1-qs)$, we get $G'=G\,K'$ and $G''=G(K''+K'^2)$, where $K'(s)=\frac{rq}{1-qs}$ and $K''(s)=\frac{rq^2}{(1-qs)^2}$. At $s=1$: $E[Y]=\frac{rq}{p}$, and $$\operatorname{Var}(Y)=G''(1)+G'(1)-G'(1)^2=\frac{rq^2}{p^2}+\frac{rq}{p}=\frac{rq(q+p)}{p^2}=\frac{rq}{p^2}.$$

(2) A geometric variable on $\{0,1,\dots\}$ has pgf $\sum_kq^kps^k=\frac{p}{1-qs}$. The sum of $r$ independent copies has pgf $\left(\frac{p}{1-qs}\right)^r=G_Y(s)$, so it has the law of $Y$. Directly: the $r$-th success happens on trial $k+r$ iff trial $k+r$ is a success and the first $k+r-1$ trials contain exactly $r-1$ successes and $k$ failures. That has probability $\binom{k+r-1}{k}p^{r-1}q^k\cdot p$. Also $\binom{k+r-1}{k}=\frac{\Gamma(k+r)}{k!\,\Gamma(r)}$ for integer $r$, so the two formulas in Definition 2.1(3) agree.

(3) The pgfs multiply to $\left(\frac{p}{1-qs}\right)^{r_1+r_2}$. ∎

Note that $\operatorname{Var}(Y)=\frac{E[Y]}{p}\gt E[Y]$: the negative binomial is overdispersed compared with a Poisson law of the same mean. This is why it is the standard model for counts that vary more than a Poisson model allows (the Research–Teaching Nexus below).

📘 Definition 2.2 — Hypergeometric law
A population of $N$ items contains $K$ "successes" and $N-K$ "failures" ($0\le K\le N$). A sample of $n\le N$ items is drawn at random without replacement, every subset of size $n$ being equally likely. The number $X$ of successes in the sample has the hypergeometric law $\operatorname{Hyp}(N,K,n)$: $$P(X=k)=\frac{\binom Kk\binom{N-K}{n-k}}{\binom Nn},\qquad \max(0,\,n-N+K)\le k\le\min(n,K).$$
📐 Theorem 2.4 — Moments of the hypergeometric law and its binomial limit

Let $X\sim\operatorname{Hyp}(N,K,n)$ and $p=K/N$. Then $$E[X]=np,\qquad \operatorname{Var}(X)=np(1-p)\,\frac{N-n}{N-1}.$$ The factor $\frac{N-n}{N-1}$ is the finite population correction. If $N\to\infty$ and $K=K_N$ with $K_N/N\to p\in(0,1)$, while $n$ and $k$ stay fixed, then $P(X=k)\to\binom nkp^k(1-p)^{n-k}$.

Proof. Imagine the sample drawn one item at a time and let $I_j=1$ if the $j$-th draw is a success. Every ordering of the population is equally likely, so the $j$-th item drawn is equally likely to be any of the $N$ items. Hence $P(I_j=1)=\frac KN=p$ for each $j$. Likewise, the ordered pair (draw $i$, draw $j$) for $i\ne j$ is equally likely to be any ordered pair of distinct items, so $$P(I_i=I_j=1)=\frac{K(K-1)}{N(N-1)},\qquad \operatorname{Cov}(I_i,I_j)=\frac{K(K-1)}{N(N-1)}-\frac{K^2}{N^2}=-\frac{p(1-p)}{N-1}.$$ Now $X=\sum_{j=1}^nI_j$, so $E[X]=np$ and $$\operatorname{Var}(X)=\sum_j\operatorname{Var}(I_j)+\sum_{i\ne j}\operatorname{Cov}(I_i,I_j)=np(1-p)-n(n-1)\frac{p(1-p)}{N-1}=np(1-p)\frac{N-n}{N-1}.$$ For the limit, write the binomial coefficients with falling factorials $[a]_j=a(a-1)\cdots(a-j+1)$: $$P(X=k)=\binom nk\frac{[K]_k\,[N-K]_{n-k}}{[N]_n}=\binom nk\cdot\frac{[K]_k}{N^k}\cdot\frac{[N-K]_{n-k}}{N^{n-k}}\cdot\frac{N^n}{[N]_n}.$$ Since $[K]_k/N^k=\prod_{i\lt k}\left(\frac KN-\frac iN\right)\to p^k$, and similarly the third factor tends to $(1-p)^{n-k}$ and the fourth to $1$, the limit is the binomial probability. ∎

The indicators $I_j$ are exchangeable but not independent: drawing a success makes later successes less likely, which is why the covariance is negative and the variance is smaller than the binomial $np(1-p)$. When the sampling fraction $n/N$ is small, the correction is close to $1$, and sampling without replacement behaves like sampling with replacement. This is the basis of the "finite population correction" in survey sampling (§8.1).

✏️ Example 1 — Counting successes and waiting for them
Data packets are sent over a noisy channel; each packet arrives intact with probability $0.8$, independently of the others. (a) Of $10$ packets, let $X$ be the number that arrive intact. Find $P(X=8)$, $P(X\ge9)$ and the most likely value of $X$. (b) A single packet is resent until it arrives intact. Let $T$ be the number of transmissions. Find $P(T=3)$, $E[T]$, and the probability that more than $4$ transmissions are needed given that the first two failed. (c) Let $T_3$ be the number of transmissions needed to deliver $3$ packets intact (each transmission carrying one packet). Find $P(T_3=5)$, $E[T_3]$ and $\operatorname{Var}(T_3)$.
(a) Binomial $X\sim\operatorname{Bin}(10,0.8)$. $P(X=8)=\binom{10}{8}0.8^8\,0.2^2=45\times0.16777\times0.04\approx0.3020$. $P(X\ge9)=10\times0.8^9\times0.2+0.8^{10}\approx0.2684+0.1074=0.3758$. Since $(n+1)p=8.8$ is not an integer, Theorem 2.1(3) gives the unique mode $\lfloor8.8\rfloor=8$.
(b) Geometric $T\sim\operatorname{Geo}(0.8)$. $P(T=3)=0.2^2\times0.8=0.032$ and $E[T]=\tfrac1{0.8}=1.25$. By memorylessness, $P(T\gt4\mid T\gt2)=P(T\gt2)=0.2^2=0.04$: two failures so far change nothing about the future.
(c) Negative binomial $T_3=Y+3$ with $Y\sim\operatorname{NB}(3,0.8)$ the number of failures. $T_3=5$ means $Y=2$: $P=\binom42\,0.8^3\,0.2^2=6\times0.512\times0.04=0.12288$. By Theorem 2.3, $E[T_3]=3+\frac{3\times0.2}{0.8}=3.75$ and $\operatorname{Var}(T_3)=\operatorname{Var}(Y)=\frac{3\times0.2}{0.64}=0.9375$.
Check $T_3$ is a sum of three independent $\operatorname{Geo}(0.8)$ variables (Theorem 2.3(2)), so $E[T_3]=3\times1.25=3.75$ and $\operatorname{Var}(T_3)=3\times\frac{0.2}{0.64}=0.9375$.
✏️ Example 2 — Acceptance sampling: hypergeometric, binomial or Poisson?
A lot of $50$ items contains $5$ defectives. An inspector draws $10$ items without replacement and accepts the lot if at most one is defective. Let $X$ be the number of defectives in the sample. (a) Find $E[X]$ and $\operatorname{Var}(X)$. (b) Find $P(X=0)$ and the acceptance probability $P(X\le1)$ exactly. (c) Compare with the binomial and Poisson approximations, and explain the differences.
(a) $X\sim\operatorname{Hyp}(50,5,10)$ with $p=0.1$. By Theorem 2.4, $E[X]=1$ and $\operatorname{Var}(X)=10\times0.1\times0.9\times\tfrac{40}{49}\approx0.7347$, compared with $0.9$ for sampling with replacement.
(b) $P(X=0)=\binom{45}{10}\big/\binom{50}{10}=\frac{45\cdot44\cdots36}{50\cdot49\cdots41}\approx0.3106$, and $P(X=1)=\frac{5\binom{45}{9}}{\binom{50}{10}}\approx0.4313$. So $P(X\le1)\approx0.7419$.
(c) With replacement, $X$ would be $\operatorname{Bin}(10,0.1)$: $P(X=0)=0.9^{10}\approx0.3487$ and $P(X\le1)\approx0.7361$. The Poisson law with mean $1$ gives $P(X=0)=e^{-1}\approx0.3679$ and $P(X\le1)=2e^{-1}\approx0.7358$. Here the sampling fraction $n/N=0.2$ is not small, so the hypergeometric law is visibly more concentrated than the binomial (variance $0.73$ against $0.9$). The Poisson law (variance $1$) is the most spread out. All three acceptance probabilities happen to be close, because the smaller mass at $0$ under the hypergeometric law ($0.3106$ against $0.3487$) is offset by more mass at $1$ ($0.4313$ against $0.3874$).

The Poisson Distribution and the Law of Rare Events

When the number of trials is large and the success probability small, the binomial law depends, to a good approximation, only on the mean number of successes $\lambda=np$. The limiting law is the Poisson distribution, the universal model for counts of rare events: accidents at a junction, calls to a helpline, mutations in a DNA strand, claims on an insurance portfolio.

📘 Definition 2.3 — Poisson law
For $\lambda\gt0$, $X$ has the Poisson law $\operatorname{Poi}(\lambda)$ if $$P(X=k)=e^{-\lambda}\frac{\lambda^k}{k!},\qquad k=0,1,2,\dots$$
📐 Theorem 2.5 — Properties of the Poisson law

Let $X\sim\operatorname{Poi}(\lambda)$.

  1. $G_X(s)=e^{\lambda(s-1)}$ and $M_X(t)=\exp\left(\lambda(e^t-1)\right)$ for all $s,t$. The $r$-th factorial moment is $\lambda^r$, so $E[X]=\operatorname{Var}(X)=\lambda$.
  2. $P(X=k)=\dfrac{\lambda}{k}P(X=k-1)$ for $k\ge1$. Hence the mode is $\lfloor\lambda\rfloor$ if $\lambda\notin\mathbb{N}$, and both $\lambda-1$ and $\lambda$ are modes if $\lambda\in\mathbb{N}$.
  3. (Reproductive property) If $X_i\sim\operatorname{Poi}(\lambda_i)$, $i=1,\dots,m$, are independent, then $\sum_iX_i\sim\operatorname{Poi}\left(\sum_i\lambda_i\right)$.

Proof. (1) $G_X(s)=e^{-\lambda}\sum_k\frac{(\lambda s)^k}{k!}=e^{\lambda(s-1)}$, finite for every $s$; putting $s=e^t$ gives $M_X$. The $r$-th derivative of $G_X$ at $1$ is $\lambda^r$, which is the factorial moment by Theorem 1.31 of §1.3. So $E[X]=\lambda$ and $\operatorname{Var}(X)=\lambda^2+\lambda-\lambda^2=\lambda$. (2) Divide consecutive probabilities: the ratio $\lambda/k$ exceeds $1$ iff $k\lt\lambda$ and equals $1$ iff $k=\lambda$. (3) The pgfs multiply to $e^{(\sum\lambda_i)(s-1)}$. ∎

📐 Theorem 2.6 — Poisson limit theorem (law of rare events) and Le Cam's inequality

(a) If $p_n\in(0,1)$ and $np_n\to\lambda\gt0$ as $n\to\infty$, then for every fixed $k\ge0$, $$\binom nkp_n^k(1-p_n)^{n-k}\to e^{-\lambda}\frac{\lambda^k}{k!}.$$

(b) (Le Cam) Let $S=I_1+\dots+I_n$ with independent $I_i\sim\operatorname{Ber}(p_i)$, and let $\lambda=\sum_ip_i$. Then for every set $A\subseteq\{0,1,2,\dots\}$, $$\left|P(S\in A)-\operatorname{Poi}(\lambda)(A)\right|\le\sum_{i=1}^np_i^2.$$ In particular, the total variation distance between $\operatorname{Bin}(n,\lambda/n)$ and $\operatorname{Poi}(\lambda)$ is at most $\lambda^2/n$.

Proof. (a) Write $\lambda_n=np_n$ and split the binomial probability into four factors: $$\binom nkp_n^k(1-p_n)^{n-k}=\frac{[n]_k}{n^k}\cdot\frac{\lambda_n^k}{k!}\cdot\left(1-\frac{\lambda_n}{n}\right)^n\cdot(1-p_n)^{-k}.$$ The first factor is $\prod_{i\lt k}(1-\tfrac in)\to1$; the second tends to $\frac{\lambda^k}{k!}$; the fourth tends to $1$ because $p_n=\lambda_n/n\to0$. For the third, $\log(1-x)=-x+O(x^2)$ as $x\to0$, so $n\log(1-\lambda_n/n)=-\lambda_n+O(\lambda_n^2/n)\to-\lambda$, and the third factor tends to $e^{-\lambda}$.

(b) We couple each Bernoulli variable with a Poisson variable on the same probability space. Let $U_1,\dots,U_n$ be independent uniform $(0,1)$ variables. Put $I_i=\mathbf{1}\{U_i\gt1-p_i\}$, so $I_i\sim\operatorname{Ber}(p_i)$. Put $Y_i=0$ if $U_i\le e^{-p_i}$, and otherwise $Y_i=k$ when $U_i$ lies in the $k$-th of the consecutive intervals of lengths $e^{-p_i}p_i^k/k!$, $k=1,2,\dots$, that fill $(e^{-p_i},1)$. Then $Y_i\sim\operatorname{Poi}(p_i)$, and the $Y_i$ are independent. Because $e^{-p}\ge1-p$, the event $\{I_i=0\}=\{U_i\le1-p_i\}$ lies inside $\{Y_i=0\}$, and the event $\{Y_i=1\}$ lies inside $\{U_i\gt e^{-p_i}\}\subseteq\{I_i=1\}$. So $I_i$ and $Y_i$ differ only on $\{I_i=1,\ Y_i\ne1\}$, and $$P(I_i\ne Y_i)=P(I_i=1)-P(Y_i=1)=p_i-p_ie^{-p_i}=p_i(1-e^{-p_i})\le p_i^2.$$ Let $T=\sum Y_i$, which is $\operatorname{Poi}(\lambda)$ by Theorem 2.5(3). For any $A$, the events $\{S\in A\}$ and $\{T\in A\}$ can differ only where $S\ne T$, so $$|P(S\in A)-P(T\in A)|\le P(S\ne T)\le\sum_iP(I_i\ne Y_i)\le\sum_ip_i^2.$$ For $\operatorname{Bin}(n,\lambda/n)$, all $p_i=\lambda/n$ and the bound is $n\cdot\lambda^2/n^2=\lambda^2/n$. ∎

Part (b) is stronger than (a) in two ways: it gives an explicit error, and it does not need equal success probabilities. The number of rare events among many different independent sources (households filing claims, each with its own small probability) is approximately Poisson with mean $\sum p_i$. The bound also shows what can go wrong: with $p=\tfrac12$ the error bound $n/4$ is useless, and indeed $\operatorname{Bin}(n,\tfrac12)$ (variance $\tfrac n4$) is not close to $\operatorname{Poi}(\tfrac n2)$ (variance $\tfrac n2$). Interactive 1 shows the total variation distance shrinking like $1/n$.

📐 Theorem 2.7 — Splitting a Poisson count, and conditioning on a total

(a) (Thinning) Let $N\sim\operatorname{Poi}(\lambda)$, and suppose that, given $N=n$, each of the $n$ events is independently of type 1 with probability $p$ and of type 2 with probability $q=1-p$. Let $N_1$, $N_2$ be the numbers of events of each type. Then $N_1\sim\operatorname{Poi}(\lambda p)$ and $N_2\sim\operatorname{Poi}(\lambda q)$, and $N_1$, $N_2$ are independent.

(b) (Conditioning) If $X\sim\operatorname{Poi}(\lambda)$ and $Y\sim\operatorname{Poi}(\mu)$ are independent, then given $X+Y=n$, $X\sim\operatorname{Bin}\left(n,\tfrac{\lambda}{\lambda+\mu}\right)$.

Proof. (a) For $i,j\ge0$, the event $\{N_1=i,N_2=j\}$ requires $N=i+j$ and then $i$ type-1 events among the $i+j$: $$P(N_1=i,N_2=j)=e^{-\lambda}\frac{\lambda^{i+j}}{(i+j)!}\binom{i+j}{i}p^iq^j=e^{-\lambda p}\frac{(\lambda p)^i}{i!}\cdot e^{-\lambda q}\frac{(\lambda q)^j}{j!},$$ using $e^{-\lambda}=e^{-\lambda p}e^{-\lambda q}$. The joint pmf factorises into the two Poisson pmfs, which proves both claims. (b) $X+Y\sim\operatorname{Poi}(\lambda+\mu)$ by Theorem 2.5(3), so for $0\le k\le n$, $$P(X=k\mid X+Y=n)=\frac{e^{-\lambda}\frac{\lambda^k}{k!}\,e^{-\mu}\frac{\mu^{n-k}}{(n-k)!}}{e^{-(\lambda+\mu)}\frac{(\lambda+\mu)^n}{n!}}=\binom nk\left(\frac{\lambda}{\lambda+\mu}\right)^k\left(\frac{\mu}{\lambda+\mu}\right)^{n-k}.\ ∎$$

Part (a) is surprising at first: for a fixed number $n$ of events, the counts $N_1$ and $n-N_1$ are perfectly negatively dependent. It is the Poisson randomness of the total that makes them independent. Independence of the two parts holds for no other law of $N$: it forces the pgf identity $G(ps+qt)=G(ps+q)\,G(p+qt)$, whose only solutions are $G(s)=e^{\lambda(s-1)}$ (statement only). The result is the key to thinning and superposing Poisson processes (§3.3).

✏️ Example 3 — Typing errors and a proof-reader
A typist mistypes each character independently with probability $0.0005$, and a page has $3000$ characters. (a) Find the probability of at most one error on a page, exactly and by the Poisson approximation, and bound the error of the approximation. (b) A proof-reader catches each error independently with probability $0.8$. Find the probability that a page has no uncaught error, and show that the numbers of caught and uncaught errors are independent. (c) Given that a page has $4$ errors in total, find the probability that exactly $3$ of them are caught.
(a) The exact law is $\operatorname{Bin}(3000,0.0005)$, with $\lambda=np=1.5$. Exactly, $P(X\le1)=0.9995^{3000}+3000\times0.0005\times0.9995^{2999}\approx0.557784$. The Poisson approximation gives $e^{-1.5}(1+1.5)=2.5e^{-1.5}\approx0.557825$. Le Cam's inequality bounds the error by $np^2=3000\times(0.0005)^2=0.00075$; the actual error is about $0.00004$.
(b) Treat the error count as $N\sim\operatorname{Poi}(1.5)$. By Theorem 2.7(a), caught errors are $\operatorname{Poi}(1.2)$ and uncaught errors are $\operatorname{Poi}(0.3)$, independent. So $P(\text{no uncaught error})=e^{-0.3}\approx0.7408$.
(c) Given $N=4$, the number caught is $\operatorname{Bin}(4,0.8)$ by construction (equivalently by Theorem 2.7(b) with $\lambda=1.2$, $\mu=0.3$, $\frac{\lambda}{\lambda+\mu}=0.8$). So the probability is $\binom43\,0.8^3\times0.2=0.4096$.
Interactive 1 — Limits Between Discrete Laws

Choose a limit theorem and press Play (or drag the size slider). The blue bars are the finite law, the red dots its limit. Watch the total variation distance $d_{TV}=\tfrac12\sum_k|p_k-q_k|$ shrink. For the binomial it stays below Le Cam's bound $\lambda^2/n$ (Theorem 2.6). For the hypergeometric (sample size $10$) the variance carries the factor $\frac{N-10}{N-1}$ (Theorem 2.4). For the negative binomial with mean $\lambda$ the variance $\lambda+\lambda^2/r$ exceeds the Poisson variance until $r$ is large (Exercise II.8).

3—
n—
total variation distance—
variance: finite law / limit—
Le Cam bound λ²/n—

Waiting Times: Exponential, Gamma and Beta Laws

Continuous waiting times are modelled by the exponential law and its sums, the gamma laws. Proportions and probabilities themselves are modelled by the beta laws on $(0,1)$. Both families are built on two classical integrals.

📘 Definition 2.4 — The gamma and beta functions
For $\alpha\gt0$ and $a,b\gt0$, $$\Gamma(\alpha)=\int_0^\infty x^{\alpha-1}e^{-x}\,dx,\qquad B(a,b)=\int_0^1x^{a-1}(1-x)^{b-1}\,dx.$$ Both integrals converge: near $0$ the integrands behave like $x^{\alpha-1}$ and $x^{a-1}$, which are integrable since $\alpha,a\gt0$; near $\infty$ the factor $e^{-x}$ wins; near $1$ the beta integrand behaves like $(1-x)^{b-1}$.
📐 Theorem 2.8 — Properties of $\Gamma$ and $B$
  1. $\Gamma(\alpha+1)=\alpha\,\Gamma(\alpha)$ and $\Gamma(1)=1$; hence $\Gamma(n)=(n-1)!$ for $n\in\mathbb{N}$.
  2. $B(a,b)=\dfrac{\Gamma(a)\Gamma(b)}{\Gamma(a+b)}$, and $B(a,b)=B(b,a)$.
  3. $\Gamma(\tfrac12)=\sqrt\pi$, and consequently $\displaystyle\int_{-\infty}^\infty e^{-z^2/2}\,dz=\sqrt{2\pi}$.

Proof. (1) Integrate by parts: $\int_0^Rx^\alpha e^{-x}dx=\left[-x^\alpha e^{-x}\right]_0^R+\alpha\int_0^Rx^{\alpha-1}e^{-x}dx$. The boundary term is $-R^\alpha e^{-R}\to0$ (and it vanishes at $0$ because $\alpha\gt0$). Also $\Gamma(1)=\int_0^\infty e^{-x}dx=1$, and induction gives $\Gamma(n)=(n-1)!$.

(2) By Tonelli's theorem, $\Gamma(a)\Gamma(b)=\iint_{(0,\infty)^2}x^{a-1}y^{b-1}e^{-(x+y)}\,dx\,dy$. Substitute $x=uv$, $y=u(1-v)$ with $u\gt0$, $0\lt v\lt1$. This is the bijection of Example 5 of §1.4, with Jacobian of absolute value $u$. So $$\Gamma(a)\Gamma(b)=\int_0^\infty\!\!\int_0^1(uv)^{a-1}(u(1-v))^{b-1}e^{-u}\,u\,dv\,du=\int_0^\infty u^{a+b-1}e^{-u}du\int_0^1v^{a-1}(1-v)^{b-1}dv,$$ which is $\Gamma(a+b)B(a,b)$. Symmetry follows from the substitution $x\mapsto1-x$.

(3) By (1) and (2), $\Gamma(\tfrac12)^2=\Gamma(1)B(\tfrac12,\tfrac12)=\int_0^1\frac{dv}{\sqrt{v(1-v)}}$. With $v=\sin^2\theta$, $dv=2\sin\theta\cos\theta\,d\theta$, the integral becomes $\int_0^{\pi/2}2\,d\theta=\pi$. Since $\Gamma(\tfrac12)\gt0$, $\Gamma(\tfrac12)=\sqrt\pi$. Finally, with $z=\sqrt{2u}$ on $(0,\infty)$, $dz=\frac{du}{\sqrt{2u}}$ and $$\int_{-\infty}^\infty e^{-z^2/2}dz=2\int_0^\infty e^{-u}\frac{du}{\sqrt{2u}}=\sqrt2\,\Gamma(\tfrac12)=\sqrt{2\pi}.\ ∎$$

📘 Definition 2.5 — Uniform, exponential, gamma and beta laws
  1. $U(a,b)$, $a\lt b$: density $\frac{1}{b-a}$ on $(a,b)$. Mean $\frac{a+b}{2}$, variance $\frac{(b-a)^2}{12}$, mgf $\frac{e^{tb}-e^{ta}}{t(b-a)}$ (§1.3, Exercise IV.6).
  2. $\operatorname{Exp}(\lambda)$, rate $\lambda\gt0$: density $\lambda e^{-\lambda x}$, $x\gt0$, and $P(X\gt x)=e^{-\lambda x}$. Its mean is $\theta=1/\lambda$ (the scale); many texts parametrise by $\theta$ instead.
  3. $\operatorname{Gamma}(\alpha,\lambda)$, shape $\alpha\gt0$ and rate $\lambda\gt0$: density $$f(x)=\frac{\lambda^\alpha}{\Gamma(\alpha)}x^{\alpha-1}e^{-\lambda x},\qquad x\gt0.$$ $\operatorname{Gamma}(1,\lambda)=\operatorname{Exp}(\lambda)$; for integer $\alpha$ it is also called the Erlang law. The chi-square law with $n$ degrees of freedom is $\chi^2_n=\operatorname{Gamma}(\tfrac n2,\tfrac12)$.
  4. $\operatorname{Beta}(a,b)$, $a,b\gt0$: density $\dfrac{x^{a-1}(1-x)^{b-1}}{B(a,b)}$, $0\lt x\lt1$. $\operatorname{Beta}(1,1)=U(0,1)$.
📐 Theorem 2.9 — The exponential law is the only memoryless continuous law

(a) If $X\sim\operatorname{Exp}(\lambda)$, then $P(X\gt s+t\mid X\gt s)=P(X\gt t)$ for all $s,t\ge0$. The conditional density of $X$ at $s$ given survival to $s$, $f(s)/P(X\gt s)$, is the constant $\lambda$.

(b) Conversely, let $X$ be a random variable with $P(X\gt0)=1$ and $P(X\gt s+t)=P(X\gt s)P(X\gt t)$ for all $s,t\ge0$. Then $X\sim\operatorname{Exp}(\lambda)$ for some $\lambda\gt0$.

Proof. (a) $e^{-\lambda(s+t)}/e^{-\lambda s}=e^{-\lambda t}$, and $\lambda e^{-\lambda s}/e^{-\lambda s}=\lambda$.

(b) Let $G(t)=P(X\gt t)$. It is nonincreasing and right-continuous, with $G(0)=1$ and $G(t)\to0$ as $t\to\infty$, and it satisfies $G(s+t)=G(s)G(t)$. Put $a=G(1)$. Applying the equation $n$ times gives $G(\tfrac1n)^n=G(1)=a$ and $G(\tfrac mn)=G(\tfrac1n)^m$. If $a=0$, then $G(\tfrac1n)=0$ for all $n$, contradicting $G(\tfrac1n)\to G(0)=1$ by right-continuity. If $a=1$, then $G(m)=1$ for all $m$, contradicting $G(t)\to0$. So $0\lt a\lt1$; put $\lambda=-\log a\gt0$. Then $G(\tfrac1n)=a^{1/n}$ (the unique nonnegative $n$-th root) and $G(r)=a^r=e^{-\lambda r}$ for every positive rational $r$. For real $t\ge0$, choose rationals $r_k\downarrow t$; right-continuity gives $G(t)=\lim e^{-\lambda r_k}=e^{-\lambda t}$. ∎

In reliability language (§9.1), the ratio $h(s)=f(s)/P(X\gt s)$ is the hazard rate: the rate of failure of a unit that has survived to age $s$. The exponential law is the law of a unit that does not age. Interactive 2 shows how the gamma and Weibull hazards rise or fall as the shape parameter crosses $1$.

📐 Theorem 2.10 — The gamma law: moments, scaling and sums

Let $X\sim\operatorname{Gamma}(\alpha,\lambda)$.

  1. For real $r\gt-\alpha$, $E[X^r]=\dfrac{\Gamma(\alpha+r)}{\Gamma(\alpha)\lambda^r}$, and $E[X^r]=\infty$ for $r\le-\alpha$. In particular $E[X]=\frac\alpha\lambda$, $\operatorname{Var}(X)=\frac{\alpha}{\lambda^2}$, and $E[1/X]=\frac{\lambda}{\alpha-1}$ for $\alpha\gt1$.
  2. $M_X(t)=\left(\dfrac{\lambda}{\lambda-t}\right)^\alpha$ for $t\lt\lambda$.
  3. For $c\gt0$, $cX\sim\operatorname{Gamma}(\alpha,\lambda/c)$. In particular $2\lambda X\sim\chi^2_{2\alpha}$.
  4. (Reproductive property) If $X_i\sim\operatorname{Gamma}(\alpha_i,\lambda)$ are independent with a common rate, then $\sum_iX_i\sim\operatorname{Gamma}(\sum_i\alpha_i,\lambda)$.
  5. Consequently, a sum of $n$ iid $\operatorname{Exp}(\lambda)$ variables is $\operatorname{Gamma}(n,\lambda)$, and if $Z_1,\dots,Z_n$ are iid $N(0,1)$, then $Z_1^2+\dots+Z_n^2\sim\chi^2_n$.
  6. For $\alpha\ge1$ the mode is $\frac{\alpha-1}{\lambda}$. For $\alpha\lt1$ the density is unbounded at $0$ and decreasing.

Proof. (1) $E[X^r]=\frac{\lambda^\alpha}{\Gamma(\alpha)}\int_0^\infty x^{\alpha+r-1}e^{-\lambda x}dx$. Near $0$ this converges iff $\alpha+r\gt0$. Then the substitution $y=\lambda x$ gives $\frac{\lambda^\alpha}{\Gamma(\alpha)}\cdot\frac{\Gamma(\alpha+r)}{\lambda^{\alpha+r}}$. With $r=1,2$ and Theorem 2.8(1), $E[X]=\frac\alpha\lambda$ and $E[X^2]=\frac{\alpha(\alpha+1)}{\lambda^2}$, so $\operatorname{Var}(X)=\frac{\alpha}{\lambda^2}$. With $r=-1$, $\frac{\Gamma(\alpha-1)}{\Gamma(\alpha)}\lambda=\frac{\lambda}{\alpha-1}$.

(2) For $t\lt\lambda$, $M_X(t)=\frac{\lambda^\alpha}{\Gamma(\alpha)}\int_0^\infty x^{\alpha-1}e^{-(\lambda-t)x}dx=\frac{\lambda^\alpha}{(\lambda-t)^\alpha}$, by the same substitution with rate $\lambda-t$. For $t\ge\lambda$ the integral diverges.

(3) By Theorem 1.26 of §1.3, $cX$ has density $\frac1cf(y/c)=\frac{(\lambda/c)^\alpha}{\Gamma(\alpha)}y^{\alpha-1}e^{-(\lambda/c)y}$. With $c=2\lambda$ the rate becomes $\tfrac12$ and the shape is $\alpha=\tfrac{2\alpha}{2}$.

(4) The mgfs multiply to $\left(\frac{\lambda}{\lambda-t}\right)^{\sum\alpha_i}$ on $t\lt\lambda$, and an mgf determines the law (Theorem 1.32(5) of §1.3).

(5) $\operatorname{Exp}(\lambda)=\operatorname{Gamma}(1,\lambda)$. By Example 3(b) of §1.3, $Z^2$ has density $\frac{1}{\sqrt{2\pi y}}e^{-y/2}=\frac{(1/2)^{1/2}}{\Gamma(1/2)}y^{-1/2}e^{-y/2}$ (using $\Gamma(\frac12)=\sqrt\pi$), which is $\operatorname{Gamma}(\tfrac12,\tfrac12)=\chi^2_1$. Now apply (4).

(6) $\frac{d}{dx}\log f(x)=\frac{\alpha-1}{x}-\lambda$, which is positive for $x\lt\frac{\alpha-1}{\lambda}$ and negative after when $\alpha\gt1$, and negative everywhere when $\alpha\le1$. ∎

📐 Theorem 2.11 — Gamma waiting times and Poisson counts

Let $E_1,E_2,\dots$ be iid $\operatorname{Exp}(\lambda)$, let $S_n=E_1+\dots+E_n$ (so $S_n\sim\operatorname{Gamma}(n,\lambda)$), and for $t\gt0$ let $N(t)=\#\{n\ge1:S_n\le t\}$ be the number of "arrivals" by time $t$. Then for every $n\in\mathbb{N}$, $$P(S_n\le t)=P(N(t)\ge n)=1-\sum_{k=0}^{n-1}e^{-\lambda t}\frac{(\lambda t)^k}{k!},$$ and $N(t)\sim\operatorname{Poi}(\lambda t)$.

Proof. The $E_i$ are positive, so $S_1\lt S_2\lt\cdots$, and at least $n$ arrivals have occurred by time $t$ iff the $n$-th has: $\{N(t)\ge n\}=\{S_n\le t\}$. Let $H(t)$ be the right-hand side. Then $H(0)=0$ and, differentiating term by term, $$H'(t)=\sum_{k=0}^{n-1}\left[\lambda e^{-\lambda t}\frac{(\lambda t)^k}{k!}-\lambda e^{-\lambda t}\frac{(\lambda t)^{k-1}}{(k-1)!}\right]=\lambda e^{-\lambda t}\frac{(\lambda t)^{n-1}}{(n-1)!},$$ where the $k=0$ term has no second part and the sum telescopes. This is the $\operatorname{Gamma}(n,\lambda)$ density, so $H(t)=\int_0^tf_{S_n}=P(S_n\le t)$. Finally, for $n\ge1$, $P(N(t)=n)=P(N(t)\ge n)-P(N(t)\ge n+1)$ $=e^{-\lambda t}\frac{(\lambda t)^n}{n!}$, and $P(N(t)=0)=P(E_1\gt t)=e^{-\lambda t}$. ∎

This is the Poisson process of rate $\lambda$ (§3.3): exponential gaps between events, gamma arrival times and Poisson counts are three descriptions of one object. The identity is also a computational tool. A gamma probability with integer shape is a finite Poisson sum, and a Poisson tail is a gamma integral.

📐 Theorem 2.12 — The beta law and gamma ratios
  1. If $X\sim\operatorname{Beta}(a,b)$, then $E[X^r]=\dfrac{B(a+r,b)}{B(a,b)}$ for $r\gt-a$; in particular $$E[X]=\frac{a}{a+b},\qquad\operatorname{Var}(X)=\frac{ab}{(a+b)^2(a+b+1)}.$$ For $a,b\gt1$ the mode is $\frac{a-1}{a+b-2}$. Also $1-X\sim\operatorname{Beta}(b,a)$.
  2. If $X\sim\operatorname{Gamma}(a,\lambda)$ and $Y\sim\operatorname{Gamma}(b,\lambda)$ are independent, then $U=X+Y$ and $V=\dfrac{X}{X+Y}$ are independent, with $U\sim\operatorname{Gamma}(a+b,\lambda)$ and $V\sim\operatorname{Beta}(a,b)$.
  3. If $U\sim U(0,1)$ and $a\gt0$, then $U^{1/a}\sim\operatorname{Beta}(a,1)$.

Proof. (1) $E[X^r]=\frac{1}{B(a,b)}\int_0^1x^{a+r-1}(1-x)^{b-1}dx=\frac{B(a+r,b)}{B(a,b)}$. By Theorem 2.8, $\frac{B(a+1,b)}{B(a,b)}=\frac{\Gamma(a+1)\Gamma(a+b)}{\Gamma(a)\Gamma(a+b+1)}=\frac{a}{a+b}$ and similarly $E[X^2]=\frac{a(a+1)}{(a+b)(a+b+1)}$. Then $$\operatorname{Var}(X)=\frac{a(a+1)}{(a+b)(a+b+1)}-\frac{a^2}{(a+b)^2}=\frac{a\left[(a+1)(a+b)-a(a+b+1)\right]}{(a+b)^2(a+b+1)}=\frac{ab}{(a+b)^2(a+b+1)}.$$ The mode solves $\frac{a-1}{x}=\frac{b-1}{1-x}$. The law of $1-X$ follows from the substitution $x\mapsto1-x$.

(2) The joint density of $(X,Y)$ is $\frac{\lambda^{a+b}}{\Gamma(a)\Gamma(b)}x^{a-1}y^{b-1}e^{-\lambda(x+y)}$ on $(0,\infty)^2$. The map $(x,y)\mapsto(u,v)=(x+y,\frac{x}{x+y})$ is a bijection onto $(0,\infty)\times(0,1)$ with inverse $x=uv$, $y=u(1-v)$ and $|J|=u$. By the Jacobian formula (Theorem 1.39 of §1.4) and Theorem 2.8(2), $$f_{U,V}(u,v)=\frac{\lambda^{a+b}}{\Gamma(a)\Gamma(b)}u^{a+b-1}e^{-\lambda u}v^{a-1}(1-v)^{b-1}=\underbrace{\frac{\lambda^{a+b}u^{a+b-1}e^{-\lambda u}}{\Gamma(a+b)}}_{\operatorname{Gamma}(a+b,\lambda)}\cdot\underbrace{\frac{v^{a-1}(1-v)^{b-1}}{B(a,b)}}_{\operatorname{Beta}(a,b)}.$$ A joint density that factorises on a product set gives independence (Theorem 1.34 of §1.4).

(3) $P(U^{1/a}\le x)=P(U\le x^a)=x^a$ on $(0,1)$, with derivative $ax^{a-1}=\frac{x^{a-1}}{B(a,1)}$, because $B(a,1)=\frac1a$. ∎

📐 Theorem 2.13 — Two links between the beta and binomial laws

(a) (Binomial tails are beta integrals) For $1\le k\le n$ and $0\le p\le1$, $$\sum_{j=k}^n\binom njp^j(1-p)^{n-j}=\frac{n!}{(k-1)!\,(n-k)!}\int_0^pt^{k-1}(1-t)^{n-k}\,dt.$$ That is, $P(\operatorname{Bin}(n,p)\ge k)=P(\operatorname{Beta}(k,n-k+1)\le p)$.

(b) (Beta–binomial law) Let $P\sim\operatorname{Beta}(a,b)$ and, given $P=p$, let $X\sim\operatorname{Bin}(n,p)$. Then $$P(X=k)=\binom nk\frac{B(a+k,\,b+n-k)}{B(a,b)},\qquad k=0,\dots,n,$$ $E[X]=\dfrac{na}{a+b}$, $\operatorname{Var}(X)=\dfrac{nab(a+b+n)}{(a+b)^2(a+b+1)}$, and given $X=k$ the conditional law of $P$ is $\operatorname{Beta}(a+k,\,b+n-k)$.

Proof. (a) Both sides vanish at $p=0$, so it suffices to compare derivatives in $p$. Using $j\binom nj=n\binom{n-1}{j-1}$ and $(n-j)\binom nj=n\binom{n-1}{j}$, $$\frac{d}{dp}\sum_{j=k}^n\binom njp^j(1-p)^{n-j}=n\sum_{j=k}^n\left[\binom{n-1}{j-1}p^{j-1}(1-p)^{n-j}-\binom{n-1}{j}p^j(1-p)^{n-1-j}\right].$$ The sum telescopes, because the second term for $j$ is the first term for $j+1$, and $\binom{n-1}{n}=0$. What remains is $n\binom{n-1}{k-1}p^{k-1}(1-p)^{n-k}$ $=\frac{n!}{(k-1)!(n-k)!}p^{k-1}(1-p)^{n-k}$, the derivative of the right side. Finally, $B(k,n-k+1)=\frac{(k-1)!(n-k)!}{n!}$ by Theorem 2.8, so the right side is the $\operatorname{Beta}(k,n-k+1)$ cdf at $p$.

(b) Integrate the conditional pmf against the beta density: $$P(X=k)=\int_0^1\binom nkp^k(1-p)^{n-k}\frac{p^{a-1}(1-p)^{b-1}}{B(a,b)}dp=\binom nk\frac{B(a+k,b+n-k)}{B(a,b)}.$$ By the tower property, $E[X]=E[nP]=\frac{na}{a+b}$. By the law of total variance (Theorem 1.38 of §1.4), $\operatorname{Var}(X)=E[nP(1-P)]+\operatorname{Var}(nP)$. Theorem 2.12(1) gives $E[P(1-P)]=\frac{a}{a+b}-\frac{a(a+1)}{(a+b)(a+b+1)}=\frac{ab}{(a+b)(a+b+1)}$, so $$\operatorname{Var}(X)=\frac{nab}{(a+b)(a+b+1)}+\frac{n^2ab}{(a+b)^2(a+b+1)}=\frac{nab(a+b+n)}{(a+b)^2(a+b+1)}.$$ Bayes' theorem for densities gives a posterior proportional to $p^k(1-p)^{n-k}\cdot p^{a-1}(1-p)^{b-1}$, the $\operatorname{Beta}(a+k,b+n-k)$ kernel. ∎

Part (a) has a probabilistic reading. If $U_1,\dots,U_n$ are iid uniform, then the $k$-th smallest of them is at most $p$ iff at least $k$ of the $U_i$ are at most $p$, a binomial event. So the $k$-th order statistic of a uniform sample is $\operatorname{Beta}(k,n-k+1)$ (§4.1). Part (b) with $a=b=1$ makes $X$ uniform on $\{0,\dots,n\}$, the Bayes–Laplace urn of Example 4 in §1.4. The posterior statement is the first example of a conjugate prior (§4.3).

✏️ Example 4 — Emergency calls: exponential gaps, gamma arrival times, Poisson counts
Calls reach an ambulance control room as a Poisson process of rate $3$ per hour: the gaps between calls are iid $\operatorname{Exp}(3)$, measured in hours. (a) Find the mean and standard deviation of the time $S_4$ of the fourth call. (b) Find $P(S_4\le1)$. (c) No call has arrived in the first $20$ minutes. What is the probability that none arrives in the next $20$ minutes? (d) Find $P(S_1\lt\tfrac12S_4)$.
(a) $S_4\sim\operatorname{Gamma}(4,3)$, so $E[S_4]=\tfrac43$ h $=80$ minutes, and $\operatorname{SD}=\frac{\sqrt4}{3}=\tfrac23$ h $=40$ minutes.
(b) By Theorem 2.11 with $\lambda t=3$, $$P(S_4\le1)=P(N(1)\ge4)=1-e^{-3}\left(1+3+\tfrac92+\tfrac{27}{6}\right)=1-13e^{-3}\approx0.3528.$$ No gamma integral is needed.
(c) By memorylessness (Theorem 2.9), the remaining wait is again $\operatorname{Exp}(3)$, so the probability is $P(E_1\gt\tfrac13)=e^{-1}\approx0.3679$, the same as for the first $20$ minutes.
(d) $S_4=E_1+(E_2+E_3+E_4)$ with $E_1\sim\operatorname{Gamma}(1,3)$ independent of $E_2+E_3+E_4\sim\operatorname{Gamma}(3,3)$. By Theorem 2.12(2), $V=S_1/S_4\sim\operatorname{Beta}(1,3)$, with density $3(1-v)^2$. So $P(V\lt\tfrac12)=1-(\tfrac12)^3=\tfrac78$, whatever the rate.
✏️ Example 5 — Beta integrals and an overdispersed count
(a) $X\sim\operatorname{Beta}(2,3)$. Find its density, mean, variance and mode, and find $P(X\gt\tfrac12)$ using Theorem 2.13(a). (b) A new treatment has unknown success probability $P\sim\operatorname{Beta}(2,2)$. It is given to $4$ patients, who respond independently given $P$. Find the distribution, mean and variance of the number $X$ of successes, and compare with $\operatorname{Bin}(4,\tfrac12)$. (c) If $3$ patients respond, what is the updated (posterior) mean of $P$?
(a) $B(2,3)=\frac{1!\,2!}{4!}=\frac1{12}$, so $f(x)=12x(1-x)^2$ on $(0,1)$. Mean $\frac25$, variance $\frac{6}{25\cdot6}=\frac1{25}$, mode $\frac{1}{3}$. With $k=2$, $n=4$, Theorem 2.13(a) says $P(X\le p)=P(\operatorname{Bin}(4,p)\ge2)$. At $p=\tfrac12$ this is $\frac{6+4+1}{16}=\frac{11}{16}$, so $P(X\gt\tfrac12)=\frac{5}{16}$. Check: $\int_{1/2}^112x(1-x)^2dx=12\left[\tfrac{x^2}{2}-\tfrac{2x^3}{3}+\tfrac{x^4}{4}\right]_{1/2}^1=12\left(\tfrac1{12}-\tfrac{11}{192}\right)=\tfrac{5}{16}$.
(b) $B(2,2)=\frac16$. By Theorem 2.13(b), $P(X=k)=6\binom4kB(2+k,6-k)$. Using $B(2,6)=\frac1{42}$, $B(3,5)=\frac1{105}$, $B(4,4)=\frac1{140}$: $$P(X=0)=P(X=4)=\tfrac17,\quad P(X=1)=P(X=3)=\tfrac{8}{35},\quad P(X=2)=\tfrac{9}{35}.$$ (They add up to $\frac{5+8+9+8+5}{35}=1$.) $E[X]=2$ and $\operatorname{Var}(X)=\frac{4\cdot4\cdot8}{16\cdot5}=1.6$, against $1$ for $\operatorname{Bin}(4,\tfrac12)$. The uncertainty about $P$ adds the term $\operatorname{Var}(4P)=0.6$: the beta–binomial is overdispersed, with more weight on the extremes $0$ and $4$ ($\tfrac17\approx0.143$ against $\tfrac1{16}\approx0.063$).
(c) The posterior is $\operatorname{Beta}(2+3,2+1)=\operatorname{Beta}(5,3)$, with mean $\frac58=0.625$. The data pull the prior mean $0.5$ towards the observed proportion $0.75$.

The Normal Family and Its Heavy-Tailed Relatives

The normal law is the law of sums of many small independent effects; the central limit theorem of §2.3 makes this precise. Its relatives arise by simple transformations. Exponentiating a normal variable gives the lognormal law, dividing two independent normals gives the Cauchy law, the difference of two exponentials gives the Laplace law, and a power of an exponential gives the Weibull law. They differ most in their tails, and the tails decide which moments exist.

📘 Definition 2.6 — Normal law
For $\mu\in\mathbb{R}$ and $\sigma\gt0$, $X\sim N(\mu,\sigma^2)$ if it has density $$f(x)=\frac{1}{\sigma\sqrt{2\pi}}\exp\left(-\frac{(x-\mu)^2}{2\sigma^2}\right),\qquad x\in\mathbb{R}.$$ $N(0,1)$ is the standard normal law, with density $\varphi$ and cdf $\Phi$. We also allow $\sigma=0$, meaning $X\equiv\mu$, when stating closure properties.
📐 Theorem 2.14 — Properties of the normal law
  1. $f$ is a density, and $X\sim N(\mu,\sigma^2)$ iff $Z=\frac{X-\mu}{\sigma}\sim N(0,1)$. Hence $P(X\le x)=\Phi\left(\frac{x-\mu}{\sigma}\right)$, and $\Phi(-z)=1-\Phi(z)$.
  2. $M_X(t)=\exp\left(\mu t+\tfrac12\sigma^2t^2\right)$ for all $t$.
  3. $E[Z^{2k+1}]=0$ and $E[Z^{2k}]=\dfrac{(2k)!}{2^kk!}=1\cdot3\cdot5\cdots(2k-1)$. So $E[X]=\mu$, $\operatorname{Var}(X)=\sigma^2$, the skewness is $0$ and the kurtosis is $\beta_2=3$.
  4. (Closure under linear combinations) If $X_i\sim N(\mu_i,\sigma_i^2)$, $i=1,\dots,n$, are independent and $a_i,b$ are constants, then $$b+\sum_ia_iX_i\sim N\left(b+\sum_ia_i\mu_i,\ \sum_ia_i^2\sigma_i^2\right).$$
  5. (Mills' upper bound) For $z\gt0$, $1-\Phi(z)\le\dfrac{\varphi(z)}{z}$.

Proof. (1) By Theorem 2.8(3) and the substitution $z=(x-\mu)/\sigma$, $\int f=\frac{1}{\sqrt{2\pi}}\int e^{-z^2/2}dz=1$. The equivalence is Theorem 1.26 of §1.3 for the affine map $x\mapsto(x-\mu)/\sigma$. Symmetry of $\varphi$ gives $\Phi(-z)=1-\Phi(z)$.

(2) Complete the square: $tz-\frac{z^2}{2}=\frac{t^2}{2}-\frac{(z-t)^2}{2}$, so $M_Z(t)=e^{t^2/2}\int\varphi(z-t)dz=e^{t^2/2}$. Then $M_X(t)=e^{\mu t}M_Z(\sigma t)$ by Theorem 1.32(3) of §1.3.

(3) Expand $e^{t^2/2}=\sum_k\frac{t^{2k}}{2^kk!}$ and compare with $M_Z(t)=\sum_r\frac{t^r}{r!}E[Z^r]$ (Theorem 1.32(2)). Odd coefficients vanish, and $\frac{E[Z^{2k}]}{(2k)!}=\frac{1}{2^kk!}$. Since $(2k)!=\left[2\cdot4\cdots(2k)\right]\left[1\cdot3\cdots(2k-1)\right]$ and $2\cdot4\cdots(2k)=2^kk!$, the second form follows. With $k=2$, $E[Z^4]=3$.

(4) The mgf of the combination is $$e^{bt}\prod_iM_{X_i}(a_it)=\exp\left(\left(b+\sum_ia_i\mu_i\right)t+\tfrac12\left(\sum_ia_i^2\sigma_i^2\right)t^2\right),$$ the mgf of the stated normal law, and mgfs determine laws.

(5) For $t\ge z\gt0$, $1\le t/z$, so $1-\Phi(z)=\int_z^\infty\varphi(t)dt\le\frac1z\int_z^\infty t\varphi(t)dt=\frac{\varphi(z)}{z}$, because $\varphi'(t)=-t\varphi(t)$. ∎

Values worth remembering (all checked numerically): $\Phi(1)=0.8413$, $\Phi(1.645)=0.9500$, $\Phi(1.96)=0.9750$, $\Phi(2)=0.9772$, $\Phi(2.576)=0.9950$; $P(|Z|\le1)=0.6827$, $P(|Z|\le2)=0.9545$, $P(|Z|\le3)=0.9973$. Exercise IV.4 gives the matching lower bound $\left(\frac1z-\frac1{z^3}\right)\varphi(z)\le1-\Phi(z)$, so $1-\Phi(z)\sim\varphi(z)/z$ as $z\to\infty$.

📘 Definition 2.7 — Lognormal, Cauchy, Laplace and Weibull laws
  1. Lognormal $\operatorname{LN}(\mu,\sigma^2)$: the law of $X=e^Y$ with $Y\sim N(\mu,\sigma^2)$. Its density is $\dfrac{1}{x\sigma\sqrt{2\pi}}\exp\left(-\dfrac{(\log x-\mu)^2}{2\sigma^2}\right)$, $x\gt0$.
  2. Cauchy $C(\theta,\gamma)$, location $\theta\in\mathbb{R}$ and scale $\gamma\gt0$: density $\dfrac{\gamma}{\pi\left(\gamma^2+(x-\theta)^2\right)}$, cdf $\tfrac12+\tfrac1\pi\arctan\frac{x-\theta}{\gamma}$.
  3. Laplace (double exponential) $\operatorname{Laplace}(\theta,b)$, $b\gt0$: density $\dfrac{1}{2b}e^{-|x-\theta|/b}$, $x\in\mathbb{R}$.
  4. Weibull $\operatorname{Weib}(k,\lambda)$, shape $k\gt0$ and scale $\lambda\gt0$: $P(X\gt x)=\exp\left(-(x/\lambda)^k\right)$ for $x\ge0$, density $\dfrac k\lambda\left(\dfrac x\lambda\right)^{k-1}e^{-(x/\lambda)^k}$. An equivalent common form is $f(x)=\alpha\beta x^{\beta-1}e^{-\alpha x^\beta}$, with $\beta=k$ and $\alpha=\lambda^{-k}$. $\operatorname{Weib}(1,\lambda)=\operatorname{Exp}(1/\lambda)$, and $k=2$ is the Rayleigh law.
📐 Theorem 2.15 — Properties of the lognormal, Cauchy, Laplace and Weibull laws
  1. (Lognormal) If $X\sim\operatorname{LN}(\mu,\sigma^2)$, then $E[X^r]=e^{r\mu+r^2\sigma^2/2}$ for every real $r$. So $E[X]=e^{\mu+\sigma^2/2}$, $\operatorname{Var}(X)=e^{2\mu+\sigma^2}\left(e^{\sigma^2}-1\right)$, the median is $e^\mu$ and the mode is $e^{\mu-\sigma^2}$. Products of independent lognormal variables are lognormal: $\operatorname{LN}(\mu_1,\sigma_1^2)\cdot\operatorname{LN}(\mu_2,\sigma_2^2)=\operatorname{LN}(\mu_1+\mu_2,\sigma_1^2+\sigma_2^2)$. Yet $E[e^{tX}]=\infty$ for every $t\gt0$.
  2. (Cauchy) If $X\sim C(0,1)$, then $\theta+\gamma X\sim C(\theta,\gamma)$ and $1/X\sim C(0,1)$; $E[X]$ does not exist; the median is $0$ and the quartiles are $\pm1$. If $Z_1,Z_2$ are iid $N(0,1)$, then $Z_1/Z_2\sim C(0,1)$. If $X_1,\dots,X_n$ are iid $C(\theta,\gamma)$, then $\bar X_n=\frac1n\sum X_i\sim C(\theta,\gamma)$ (statement here; the proof via characteristic functions is in §2.2).
  3. (Laplace) If $E_1,E_2$ are iid $\operatorname{Exp}(1)$, then $\theta+b(E_1-E_2)\sim\operatorname{Laplace}(\theta,b)$. Its mgf is $\dfrac{e^{\theta t}}{1-b^2t^2}$ for $|t|\lt\tfrac1b$, its mean is $\theta$ and its variance $2b^2$, and $|X-\theta|\sim\operatorname{Exp}(1/b)$.
  4. (Weibull) If $E\sim\operatorname{Exp}(1)$, then $\lambda E^{1/k}\sim\operatorname{Weib}(k,\lambda)$, and $E[X^r]=\lambda^r\Gamma\left(1+\tfrac rk\right)$ for $r\gt-k$. Its hazard rate $h(x)=\frac k\lambda\left(\frac x\lambda\right)^{k-1}$ increases for $k\gt1$, is constant for $k=1$ and decreases for $k\lt1$. The minimum of $n$ iid $\operatorname{Weib}(k,\lambda)$ variables is $\operatorname{Weib}(k,\lambda n^{-1/k})$.

Proof. (1) $E[X^r]=E[e^{rY}]=M_Y(r)$, which is finite for every real $r$ by Theorem 2.14(2). With $r=1,2$: $E[X]=e^{\mu+\sigma^2/2}$ and $E[X^2]=e^{2\mu+2\sigma^2}$, so $\operatorname{Var}(X)=e^{2\mu+\sigma^2}(e^{\sigma^2}-1)$. Since $x\mapsto e^x$ is increasing, $P(X\le e^\mu)=P(Y\le\mu)=\tfrac12$. For the mode, $\frac{d}{dx}\log f(x)=-\frac1x-\frac{\log x-\mu}{\sigma^2x}$ vanishes iff $\log x=\mu-\sigma^2$, where it changes sign from $+$ to $-$. The log of a product of independent lognormals is a sum of independent normals, which is normal by Theorem 2.14(4). The infinite mgf was shown in the ⚠️ box of §1.3: for $t\gt0$, $e^{te^y}$ grows faster than $e^{(y-\mu)^2/(2\sigma^2)}$ as $y\to\infty$.

(2) The affine map is Theorem 1.26 of §1.3. For $1/X$, the map $x\mapsto1/x$ is decreasing on each of $(-\infty,0)$ and $(0,\infty)$, each $y\ne0$ has one preimage $1/y$, and $|h'(y)|=y^{-2}$: $$f_{1/X}(y)=\frac{1}{\pi(1+y^{-2})}\cdot\frac1{y^2}=\frac{1}{\pi(1+y^2)}.$$ Nonexistence of the mean is Example 6(b) of §1.3, and the quartiles come from $\arctan(\pm1)=\pm\frac\pi4$. For the ratio, note that $W=Z_1/Z_2$ equals $(Z_1\operatorname{sgn}Z_2)/|Z_2|$, and $Z_1\operatorname{sgn}Z_2$ is $N(0,1)$ and independent of $|Z_2|$ (given $Z_2$, it is $\pm Z_1$, which is $N(0,1)$ whatever the sign). For independent $Z\sim N(0,1)$ and $V\gt0$ with density $g$, the map $(z,v)\mapsto(z/v,v)$ has inverse $z=wv$ with Jacobian $v$, so $Z/V$ has density $\int_0^\infty v\,\varphi(wv)\,g(v)\,dv$. With $g=2\varphi$ on $(0,\infty)$, $$f_W(w)=\int_0^\infty v\cdot\frac{2}{2\pi}e^{-(1+w^2)v^2/2}dv=\frac1\pi\cdot\frac{1}{1+w^2}.$$

(3) $M_{E_1-E_2}(t)=\frac{1}{1-t}\cdot\frac{1}{1+t}=\frac{1}{1-t^2}$ for $|t|\lt1$, which is the mgf of $\operatorname{Laplace}(0,1)$ computed in Exercise IV.8 of §1.3. By uniqueness and Theorem 1.32(3), $\theta+b(E_1-E_2)$ has mgf $\frac{e^{\theta t}}{1-b^2t^2}$ and is $\operatorname{Laplace}(\theta,b)$. Its variance is $b^2\cdot2\operatorname{Var}(E_1)=2b^2$. Finally, $|X-\theta|$ has density $2\cdot\frac{1}{2b}e^{-y/b}$ on $y\gt0$.

(4) $P(\lambda E^{1/k}\gt x)=P(E\gt(x/\lambda)^k)=e^{-(x/\lambda)^k}$. Then $E[X^r]=\lambda^rE[E^{r/k}]=\lambda^r\Gamma(1+\frac rk)$ by Theorem 2.10(1) with $\alpha=1$. The hazard is the density divided by the survival function, a power $x^{k-1}$ with the stated monotonicity. For the minimum, $P(\min\gt x)=e^{-n(x/\lambda)^k}=e^{-(x/(\lambda n^{-1/k}))^k}$. ∎

✏️ Example 6 — Normal, lognormal and Weibull computations
(a) A machine fills bags whose weights are $N(500,4^2)$ grams. Find the proportion of bags under $494$ g and the weight exceeded by $95\%$ of bags. (b) Two bags are packed together. Find the probability that the pack weighs less than $990$ g. (c) Grain diameters $D$ (in mm) satisfy $\log D\sim N(0,0.5^2)$. Find the median, the mean and $P(D\gt E[D])$. (d) Bearings have Weibull lifetimes with shape $2$ and scale $1000$ hours. Find the mean life, $P(T\gt500)$ and $P(T\gt1000\mid T\gt500)$, and compare with the memoryless case.
(a) $P(W\lt494)=\Phi\left(\frac{494-500}{4}\right)=\Phi(-1.5)\approx0.0668$, so about $6.7\%$ of bags. The weight exceeded by $95\%$ of bags is the $5\%$ quantile, $500-1.6449\times4\approx493.42$ g.
(b) By Theorem 2.14(4), $W_1+W_2\sim N(1000,32)$, so $P(W_1+W_2\lt990)=\Phi\left(\frac{-10}{\sqrt{32}}\right)=\Phi(-1.768)\approx0.0385$. A common error is to use variance $2^2\cdot16=64$, which is the variance of $2W_1$, not of $W_1+W_2$.
(c) $D\sim\operatorname{LN}(0,0.25)$. Median $e^0=1$ mm, mean $e^{0.125}\approx1.1331$ mm. Then $P(D\gt e^{0.125})=P(\log D\gt0.125)=1-\Phi(0.25)\approx0.4013$. The mean exceeds the median, so less than half the grains are larger than average: the lognormal law is right-skewed.
(d) $E[T]=1000\,\Gamma(1.5)=1000\cdot\tfrac{\sqrt\pi}{2}\approx886.2$ h. $P(T\gt500)=e^{-0.25}\approx0.7788$ and $$P(T\gt1000\mid T\gt500)=\frac{e^{-1}}{e^{-0.25}}=e^{-0.75}\approx0.4724.$$ A used bearing is worse than a new one: an exponential lifetime would give the same probability $0.7788$ for both. The hazard $h(t)=2t/10^6$ per hour increases, so the bearing ages.
Interactive 2 — Continuous Families: Shape, Moments and Hazard

Pick a family and move the parameters (or press Play to sweep the shape). The blue curve is the density $f$ and the dashed red curve is the hazard rate $h=f/(1-F)$. Watch the gamma and Weibull hazards fall, stay flat or rise as the shape crosses $1$ (only the exponential is memoryless), and watch the lognormal hazard rise and then fall. In the last mode the normal, Laplace and Cauchy laws have the same median and quartiles, yet their tails differ enormously: move the threshold $x$ and compare $P(|X|\gt x)$.

——
mean—
variance—
median—
mode—

Relationships and Reproductive Properties

Most CSIR questions on this section test a relationship rather than a single formula. We collect the closure properties under independent sums, then a map of the transformations proved above, and finally a reference table of moments and generating functions.

📐 Theorem 2.16 — Reproductive properties

Let $X_1,\dots,X_m$ be independent. Then $S=X_1+\dots+X_m$ has the following laws:

  1. $X_i\sim\operatorname{Bin}(n_i,p)$ ⟹ $S\sim\operatorname{Bin}(\sum n_i,p)$ — the same $p$ is essential;
  2. $X_i\sim\operatorname{Poi}(\lambda_i)$ ⟹ $S\sim\operatorname{Poi}(\sum\lambda_i)$;
  3. $X_i\sim\operatorname{NB}(r_i,p)$ ⟹ $S\sim\operatorname{NB}(\sum r_i,p)$; in particular a sum of $r$ iid geometric failure counts is $\operatorname{NB}(r,p)$;
  4. $X_i\sim\operatorname{Gamma}(\alpha_i,\lambda)$ ⟹ $S\sim\operatorname{Gamma}(\sum\alpha_i,\lambda)$ — the same rate is essential; $\chi^2_{n_1}+\chi^2_{n_2}=\chi^2_{n_1+n_2}$;
  5. $X_i\sim N(\mu_i,\sigma_i^2)$ ⟹ $S\sim N(\sum\mu_i,\sum\sigma_i^2)$;
  6. $X_i\sim C(\theta_i,\gamma_i)$ ⟹ $S\sim C(\sum\theta_i,\sum\gamma_i)$ (proof in §2.2).

The uniform, beta, lognormal, Laplace and Weibull families are not closed under independent sums, nor are exponentials with different rates or binomials with different $p$.

Proof. (1)–(5) are Theorems 2.1(2), 2.5(3), 2.3(3), 2.10(4) and 2.14(4); each is a product of pgfs or mgfs followed by the uniqueness theorem. For the negative statements: $U_1+U_2$ is triangular, not uniform (Example 6 of §1.4). A sum of two beta variables can exceed $1$, so it is not beta. Exponentials with different rates and binomials with different $p$ are Example 7(a) and (c) below. Two iid $\operatorname{Laplace}(0,1)$ variables have a sum with mgf $(1-t^2)^{-2}$; a Laplace law symmetric about $0$ has mgf $(1-b^2t^2)^{-1}$, and the two functions differ (one has a double pole at $t=1$, the other a simple pole at $t=\frac1b$). That sums of independent lognormal or Weibull variables leave their families is also true, but we do not prove it here. ∎

fromoperationresultwhere
Bernoulli$(p)$sum of $n$ iid$\operatorname{Bin}(n,p)$Thm 2.1
geometric failuressum of $r$ iid$\operatorname{NB}(r,p)$Thm 2.3
$\operatorname{Hyp}(N,K,n)$$N\to\infty$, $K/N\to p$$\operatorname{Bin}(n,p)$Thm 2.4
$\operatorname{Bin}(n,p_n)$$np_n\to\lambda$$\operatorname{Poi}(\lambda)$Thm 2.6
$\operatorname{NB}(r,p_r)$$r(1-p_r)\to\lambda$$\operatorname{Poi}(\lambda)$Ex. II.8
$\operatorname{Poi}(\lambda)$thinning with prob. $p$$\operatorname{Poi}(\lambda p)$, independent partsThm 2.7
$\operatorname{Poi}(\lambda),\operatorname{Poi}(\mu)$condition on the sum $n$$\operatorname{Bin}(n,\frac{\lambda}{\lambda+\mu})$Thm 2.7
$\operatorname{Gamma}(r,\lambda)$ ratemix a Poisson mean$\operatorname{NB}(r,\frac{\lambda}{1+\lambda})$Ex. II.9
$U(0,1)$$-\frac1\lambda\log U$; $U^{1/a}$$\operatorname{Exp}(\lambda)$; $\operatorname{Beta}(a,1)$§1.3, Thm 2.12
$\operatorname{Exp}(\lambda)$sum of $n$ iid; integer part$\operatorname{Gamma}(n,\lambda)$; geometricThm 2.10, Ex. III.2
Exp gapscount arrivals by $t$$\operatorname{Poi}(\lambda t)$Thm 2.11
$\operatorname{Gamma}(a,\lambda),\operatorname{Gamma}(b,\lambda)$$\frac{X}{X+Y}$$\operatorname{Beta}(a,b)$, independent of $X+Y$Thm 2.12
$\operatorname{Beta}(k,n-k+1)$cdf at $p$$P(\operatorname{Bin}(n,p)\ge k)$Thm 2.13
$\operatorname{Beta}(a,b)$ priorbinomial given $P$beta–binomialThm 2.13
$N(0,1)$$Z^2$; $\sum_1^nZ_i^2$$\chi^2_1=\operatorname{Gamma}(\frac12,\frac12)$; $\chi^2_n$Thm 2.10
$N(\mu,\sigma^2)$$e^X$$\operatorname{LN}(\mu,\sigma^2)$Thm 2.15
$N(0,1),N(0,1)$ratio $Z_1/Z_2$$C(0,1)$Thm 2.15
$\operatorname{Exp}(1),\operatorname{Exp}(1)$$\theta+b(E_1-E_2)$$\operatorname{Laplace}(\theta,b)$Thm 2.15
$\operatorname{Exp}(1)$$\lambda E^{1/k}$$\operatorname{Weib}(k,\lambda)$Thm 2.15
lawpmf / pdfmeanvariancepgf / mgf
$\operatorname{Bin}(n,p)$$\binom nkp^kq^{n-k}$$np$$npq$$(q+ps)^n$
$\operatorname{Poi}(\lambda)$$e^{-\lambda}\lambda^k/k!$$\lambda$$\lambda$$e^{\lambda(s-1)}$
$\operatorname{Geo}(p)$ on $\mathbb{N}$$q^{k-1}p$$1/p$$q/p^2$$\frac{ps}{1-qs}$
$\operatorname{NB}(r,p)$ failures$\binom{k+r-1}{k}p^rq^k$$rq/p$$rq/p^2$$\left(\frac{p}{1-qs}\right)^r$
$\operatorname{Hyp}(N,K,n)$$\binom Kk\binom{N-K}{n-k}/\binom Nn$$np$, $p=\frac KN$$npq\frac{N-n}{N-1}$—
uniform $\{1..n\}$$1/n$$\frac{n+1}{2}$$\frac{n^2-1}{12}$$\frac{s(1-s^n)}{n(1-s)}$
$U(a,b)$$\frac1{b-a}$$\frac{a+b}2$$\frac{(b-a)^2}{12}$$\frac{e^{bt}-e^{at}}{t(b-a)}$
$\operatorname{Exp}(\lambda)$$\lambda e^{-\lambda x}$$\frac1\lambda$$\frac1{\lambda^2}$$\frac{\lambda}{\lambda-t}$
$\operatorname{Gamma}(\alpha,\lambda)$$\frac{\lambda^\alpha x^{\alpha-1}e^{-\lambda x}}{\Gamma(\alpha)}$$\frac\alpha\lambda$$\frac\alpha{\lambda^2}$$\left(\frac{\lambda}{\lambda-t}\right)^\alpha$
$\operatorname{Beta}(a,b)$$\frac{x^{a-1}(1-x)^{b-1}}{B(a,b)}$$\frac{a}{a+b}$$\frac{ab}{(a+b)^2(a+b+1)}$no closed form
$N(\mu,\sigma^2)$$\frac{1}{\sigma\sqrt{2\pi}}e^{-(x-\mu)^2/2\sigma^2}$$\mu$$\sigma^2$$e^{\mu t+\sigma^2t^2/2}$
$\operatorname{LN}(\mu,\sigma^2)$see Def. 2.7$e^{\mu+\sigma^2/2}$$e^{2\mu+\sigma^2}(e^{\sigma^2}-1)$$\infty$ for $t\gt0$
$C(\theta,\gamma)$$\frac{\gamma}{\pi(\gamma^2+(x-\theta)^2)}$nonenonenone (cf $e^{i\theta t-\gamma|t|}$)
$\operatorname{Laplace}(\theta,b)$$\frac1{2b}e^{-|x-\theta|/b}$$\theta$$2b^2$$\frac{e^{\theta t}}{1-b^2t^2}$
$\operatorname{Weib}(k,\lambda)$$\frac k\lambda(\frac x\lambda)^{k-1}e^{-(x/\lambda)^k}$$\lambda\Gamma(1+\frac1k)$$\lambda^2\left[\Gamma(1+\frac2k)-\Gamma(1+\frac1k)^2\right]$no closed form
🧭 Problem-solving strategy — recognising a distribution
  1. Read the mechanism. Fixed number of independent trials → binomial; without replacement → hypergeometric; trials until a success → geometric or negative binomial; rare events in time or space → Poisson; waiting for the $r$-th event → gamma; a random proportion → beta.
  2. Read the generating function. $(q+ps)^n$, $e^{\lambda(s-1)}$, $\left(\frac{p}{1-qs}\right)^r$, $(1-t/\lambda)^{-\alpha}$, $e^{\mu t+\sigma^2t^2/2}$ identify the law at once (uniqueness). Match the constants carefully: $(1-3t)^{-2}$ is $\operatorname{Gamma}(2,\tfrac13)$, mean $6$.
  3. Read the kernel. A density proportional to $x^{\alpha-1}e^{-\lambda x}$ on $(0,\infty)$ is gamma, and one proportional to $x^{a-1}(1-x)^{b-1}$ on $(0,1)$ is beta. The normalising constant then follows from $\Gamma$ or $B$, with no integration.
  4. Convert between Poisson and gamma. $P(\operatorname{Gamma}(n,\lambda)\le t)=P(\operatorname{Poi}(\lambda t)\ge n)$, and binomial tails are beta cdfs.
  5. Check the convention (rate or mean; trials or failures) and the hypotheses of a closure property (same $p$, same rate, independence).
✏️ Example 7 — Prove or disprove
(a) If $X\sim\operatorname{Exp}(1)$ and $Y\sim\operatorname{Exp}(2)$ are independent, then $X+Y$ has a gamma law. (b) If $X,Y$ are iid $\operatorname{Geo}(p)$ on $\mathbb{N}$, then $\min(X,Y)$ is geometric. (c) If $X\sim\operatorname{Bin}(n,p)$ and $Y\sim\operatorname{Bin}(m,r)$ are independent with $p\ne r$ and $n,m\ge1$, then $X+Y$ is not binomial. (d) If $X\sim\operatorname{Gamma}(\alpha,\lambda)$, then $E[1/X]\lt\infty$ for every $\alpha\gt0$. (e) If $X\sim C(0,1)$, then $\frac{2X}{1-X^2}\sim C(0,1)$. (f) A Weibull lifetime always has an increasing hazard rate.
(a) False $M_{X+Y}(t)=\frac{1}{1-t}\cdot\frac{2}{2-t}$ for $t\lt1$. A gamma mgf $\left(\frac{\lambda}{\lambda-t}\right)^\alpha$ is finite exactly for $t\lt\lambda$ and blows up only at $t=\lambda$. Here $M$ is finite for $t\lt1$, which forces $\lambda=1$, but then $\left(\frac{1}{1-t}\right)^\alpha$ would have to equal $\frac{2}{(1-t)(2-t)}$ for all $t\lt1$. At $t=-1$ that requires $2^{-\alpha}=\tfrac13$, and at $t=-2$ it requires $3^{-\alpha}=\tfrac16$; these give different values of $\alpha$. The true density is the convolution $2(e^{-x}-e^{-2x})$, $x\gt0$ (a hypoexponential law).
(b) True $P(\min(X,Y)\gt n)=P(X\gt n)P(Y\gt n)=(q^2)^n$, so $\min(X,Y)\sim\operatorname{Geo}(1-q^2)$ by Theorem 2.2. (It is the waiting time until the first success in either of two parallel sequences of trials.)
(c) True Suppose $X+Y\sim\operatorname{Bin}(N,s)$. Its largest possible value is $n+m$, so $N=n+m$, and matching means gives $s=\frac{np+mr}{n+m}$. Now $X+Y$ is a sum of $n+m$ independent Bernoulli variables with probabilities $p_i$ (equal to $p$ or $r$), and $$\operatorname{Var}(X+Y)=\sum p_i-\sum p_i^2\lt(n+m)s-(n+m)s^2,$$ because $\sum p_i^2\gt\frac{(\sum p_i)^2}{n+m}$ (Cauchy–Schwarz, strict since the $p_i$ are not all equal). So the variance is too small for $\operatorname{Bin}(n+m,s)$.
(d) False By Theorem 2.10(1), $E[X^{-1}]$ is finite iff $-1\gt-\alpha$, i.e. $\alpha\gt1$. For $\operatorname{Exp}(\lambda)$, $E[1/X]=\int_0^\infty\frac{\lambda e^{-\lambda x}}{x}dx=\infty$, because the integrand behaves like $\lambda/x$ near $0$.
(e) True Write $X=\tan\Theta$ with $\Theta\sim U(-\frac\pi2,\frac\pi2)$ (this is the quantile transform of the Cauchy cdf). Then $\frac{2X}{1-X^2}=\tan2\Theta$ (when $X\ne\pm1$, an event of probability $1$). As $\Theta$ ranges uniformly over an interval of length $\pi$, $2\Theta$ ranges uniformly over an interval of length $2\pi$, and $\tan$ has period $\pi$, so $\tan2\Theta$ has the law of $\tan\Theta'$ with $\Theta'\sim U(-\frac\pi2,\frac\pi2)$, which is $C(0,1)$.
(f) False $h(x)=\frac k\lambda(x/\lambda)^{k-1}$ decreases for $k\lt1$ (infant mortality) and is constant for $k=1$. Only $k\gt1$ gives ageing.
⚠️ Counterexamples and pitfalls
  1. Parametrisations differ. "Exponential with parameter $2$" may mean rate $2$ (mean $\tfrac12$) or mean $2$. The gamma law appears with rate $\lambda$ or scale $\theta=1/\lambda$. The geometric law lives on $\{1,2,\dots\}$ (mean $\tfrac1p$) or on $\{0,1,\dots\}$ (mean $\tfrac qp$). Always read off the support and the density first.
  2. Sums of exponentials with different rates are not gamma (Example 7(a)). The gamma reproductive property needs a common rate.
  3. Sums of binomials with different $p$ are not binomial (Example 7(c)): the variance is too small.
  4. Mean equal to variance does not make a law Poisson. $P(X=0)=P(X=2)=\tfrac12$ has mean $1$ and variance $1$.
  5. Only two laws are memoryless: the geometric among laws on $\mathbb{N}$ and the exponential among positive laws (Theorems 2.2, 2.9). The uniform law on $(0,1)$ has $P(X\gt s+t\mid X\gt s)=\frac{1-s-t}{1-s}\lt1-t$ for $s,t\gt0$, $s+t\lt1$; gamma laws with $\alpha\ne1$ and Weibull laws with $k\ne1$ are not memoryless either.
  6. The Poisson approximation needs small $p$, not just large $n$. $\operatorname{Bin}(100,\tfrac12)$ has variance $25$, while $\operatorname{Poi}(50)$ has variance $50$. Le Cam's bound $np^2$ quantifies this.
  7. Without replacement is not with replacement. Ignoring the factor $\frac{N-n}{N-1}$ overstates the hypergeometric variance; it matters when $n/N$ is not small (Example 2).
  8. The Cauchy law has no mean, and averaging does not help. $\bar X_n\sim C(0,1)$ for every $n$, so the sample mean of Cauchy data never settles. The law of large numbers (§2.3) needs $E|X|\lt\infty$.
  9. All moments, no mgf. The lognormal law has $E[X^r]$ finite for all $r$ but $E[e^{tX}]=\infty$ for all $t\gt0$.
  10. Normal marginals are not enough. If $Z\sim N(0,1)$ and $S=\pm1$ with probability $\tfrac12$ each, independent of $Z$, then $SZ\sim N(0,1)$, but $Z+SZ=(1+S)Z$ equals $0$ with probability $\tfrac12$, so it is not normal. Theorem 2.14(4) needs independence (or joint normality, §7.1).
  11. Gamma laws with shape below $1$ have densities unbounded at $0$, mode $0$ and $E[1/X]=\infty$ for $\alpha\le1$ (Example 7(d)).
  12. A beta–binomial count is not binomial. Mixing over an uncertain $p$ inflates the variance from $npq$ to $npq\frac{a+b+n}{a+b+1}$ (Example 5).

Exercises

I
Laws built from Bernoulli trials.
  • 1 $X\sim\operatorname{Bin}(8,\tfrac14)$. Find $P(X=2)$, $P(X\ge1)$, the mode and $E[X^2]$.
  • 2 For $X\sim\operatorname{Bin}(n,p)$, show that $E\left[\dfrac{1}{X+1}\right]=\dfrac{1-q^{n+1}}{(n+1)p}$.
  • 3 $X\sim\operatorname{Bin}(n,p)$ and $Y\sim\operatorname{Bin}(m,p)$ are independent. Show that, given $X+Y=k$, $X$ is hypergeometric $\operatorname{Hyp}(n+m,n,k)$, whatever the value of $p$.
  • 4 A fair die is rolled until a six appears, and $N$ is the number of rolls. Find $P(N\gt6)$, $P(N\text{ is even})$ and $E[N\mid N\gt3]$.
  • 5 $X,Y$ are iid $\operatorname{Geo}(p)$ on $\mathbb{N}$. Find $P(X=Y)$, and show that given $X+Y=n$ ($n\ge2$), $X$ is uniform on $\{1,\dots,n-1\}$.
  • 6 (Sport) A team wins each game independently with probability $0.6$. A series ends when one team has won $4$ games. Find the probability that the team wins the series, and the probability that the series lasts all $7$ games.
  • 7 A committee of $5$ is chosen at random from $8$ women and $7$ men. Let $X$ be the number of women. Find $P(X=3)$, $E[X]$ and $\operatorname{Var}(X)$.
  • 8 (Ecology: capture–recapture) $100$ fish in a lake are tagged and released. Later $40$ fish are caught and $x$ of them are tagged. Treating the number $N$ of fish as unknown, show that $P_N(X=x)$, as a function of $N$, is maximised at $N=\lfloor4000/x\rfloor$, and evaluate it for $x=12$.
1 $P(X=2)=\binom82\left(\tfrac14\right)^2\left(\tfrac34\right)^6=\frac{28\cdot729}{65536}\approx0.3115$. $P(X\ge1)=1-\left(\tfrac34\right)^8\approx0.8999$. $(n+1)p=2.25$, so the mode is $2$. $E[X^2]=npq+(np)^2=1.5+4=5.5$.
2 Use $\frac{1}{k+1}\binom nk=\frac{1}{n+1}\binom{n+1}{k+1}$: $$E\left[\tfrac{1}{X+1}\right]=\frac{1}{(n+1)p}\sum_{k=0}^n\binom{n+1}{k+1}p^{k+1}q^{n-k}=\frac{(p+q)^{n+1}-q^{n+1}}{(n+1)p}.$$ (Check: $n=5$, $p=0.3$ gives $0.49019$ both ways.)
3 $X+Y\sim\operatorname{Bin}(n+m,p)$, so $$P(X=j\mid X+Y=k)=\frac{\binom njp^jq^{n-j}\binom{m}{k-j}p^{k-j}q^{m-k+j}}{\binom{n+m}{k}p^kq^{n+m-k}}=\frac{\binom nj\binom m{k-j}}{\binom{n+m}k}.$$ This is $\operatorname{Hyp}(n+m,n,k)$: given $k$ successes in $n+m$ trials, every set of $k$ positions is equally likely.
4 $P(N\gt6)=\left(\tfrac56\right)^6\approx0.3349$. $P(N\text{ even})=\sum_{j\ge1}\left(\tfrac56\right)^{2j-1}\tfrac16=\frac{5/36}{1-25/36}=\frac5{11}$. By memorylessness, given $N\gt3$, $N-3\sim\operatorname{Geo}(\tfrac16)$, so $E[N\mid N\gt3]=3+6=9$.
5 $P(X=Y)=\sum_kp^2q^{2(k-1)}=\frac{p^2}{1-q^2}=\frac{p}{2-p}$. For $1\le j\le n-1$, $P(X=j,Y=n-j)=pq^{j-1}pq^{n-j-1}=p^2q^{n-2}$, which does not depend on $j$, so the conditional law is uniform on $\{1,\dots,n-1\}$.
6 The team wins the series iff its $4$th win comes before its $4$th loss, i.e. the number of losses before the $4$th win is at most $3$ (play out all games: this does not change the winner). By Theorem 2.3, $P=\sum_{k=0}^3\binom{3+k}{k}0.6^4\,0.4^k=0.6^4(1+1.6+1.6+1.28)=0.1296\times5.48\approx0.7102$. The series lasts $7$ games iff the first $6$ games are split $3$–$3$: $\binom63\,0.6^3\,0.4^3=0.27648$.
7 $X\sim\operatorname{Hyp}(15,8,5)$. $P(X=3)=\frac{\binom83\binom72}{\binom{15}5}=\frac{56\cdot21}{3003}=\frac{56}{143}\approx0.3916$. $E[X]=5\cdot\frac8{15}=\frac83$. $\operatorname{Var}(X)=5\cdot\frac8{15}\cdot\frac7{15}\cdot\frac{10}{14}=\frac89$.
8 $X\sim\operatorname{Hyp}(N,100,40)$. The ratio of consecutive likelihoods is $$\frac{P_N(X=x)}{P_{N-1}(X=x)}=\frac{(N-100)(N-40)}{N(N-140+x)},$$ which is at least $1$ iff $N^2-140N+4000\ge N^2-140N+xN$, i.e. iff $N\le4000/x$. So the likelihood increases up to $\lfloor4000/x\rfloor$ and decreases after it. For $x=12$, $\hat N=\lfloor333.3\rfloor=333$ fish (a numerical search confirms it). This is the Lincoln–Petersen estimate, the maximum likelihood estimate of §4.2.
II
The Poisson law and discrete limits.
  • 1 $X\sim\operatorname{Poi}(2)$. Find $P(X=0)$, $P(X\le2)$, $P(X\ge3)$ and all modes.
  • 2 $X$ is Poisson with $P(X=1)=P(X=2)$. Find $E[X]$ and $P(X=4)$.
  • 3 For $X\sim\operatorname{Poi}(\lambda)$, find $E[X^3]$ and the third central moment, and deduce the skewness $\lambda^{-1/2}$.
  • 4 For $X\sim\operatorname{Poi}(\lambda)$, show that $P(X\text{ is even})=\tfrac12\left(1+e^{-2\lambda}\right)$.
  • 5 For $X\sim\operatorname{Poi}(\lambda)$, show that $E\left[\frac{1}{X+1}\right]=\frac{1-e^{-\lambda}}{\lambda}$.
  • 6 (Insurance) Claims arrive at an insurer at a Poisson rate of $4$ per month, and each claim is "large" with probability $0.1$, independently. Find the probability of no large claim in a month, of at least two large claims in a quarter, and, given $6$ claims in a month, of exactly one large claim.
  • 7 $X\sim\operatorname{Bin}(500,0.004)$. Compare $P(X=2)$ with its Poisson approximation and state Le Cam's bound for the error.
  • 8 Prove that if $r\to\infty$ and $p_r\to1$ with $r(1-p_r)\to\lambda\gt0$, then the $\operatorname{NB}(r,p_r)$ probabilities converge to those of $\operatorname{Poi}(\lambda)$.
  • 9 (Gamma–Poisson mixture) Given $\Lambda=\theta$, let $X\sim\operatorname{Poi}(\theta)$, where $\Lambda\sim\operatorname{Gamma}(r,\lambda)$. Show that $X\sim\operatorname{NB}\left(r,\frac{\lambda}{1+\lambda}\right)$, and verify $\operatorname{Var}(X)\gt E[X]$.
1 $P(X=0)=e^{-2}\approx0.1353$; $P(X\le2)=e^{-2}(1+2+2)=5e^{-2}\approx0.6767$; $P(X\ge3)\approx0.3233$. Since $\lambda=2$ is an integer, the modes are $1$ and $2$ (both have probability $2e^{-2}$).
2 $\lambda e^{-\lambda}=\frac{\lambda^2}{2}e^{-\lambda}$ gives $\lambda=2$. So $E[X]=2$ and $P(X=4)=e^{-2}\frac{16}{24}=\tfrac23e^{-2}\approx0.0902$.
3 $X^3=X(X-1)(X-2)+3X(X-1)+X$, so with factorial moments $\lambda^r$, $E[X^3]=\lambda^3+3\lambda^2+\lambda$. Then $E[(X-\lambda)^3]=E[X^3]-3\lambda E[X^2]+3\lambda^2E[X]-\lambda^3=\lambda^3+3\lambda^2+\lambda-3\lambda(\lambda^2+\lambda)+3\lambda^3-\lambda^3=\lambda$. Skewness $\lambda/\lambda^{3/2}=\lambda^{-1/2}$: the Poisson law becomes symmetric as $\lambda$ grows.
4 $P(\text{even})-P(\text{odd})=E[(-1)^X]=G_X(-1)=e^{-2\lambda}$, and $P(\text{even})+P(\text{odd})=1$. Add and halve.
5 $E\left[\frac1{X+1}\right]=\sum_k\frac{e^{-\lambda}\lambda^k}{(k+1)!}=\frac{e^{-\lambda}}{\lambda}\sum_{j\ge1}\frac{\lambda^j}{j!}=\frac{e^{-\lambda}(e^\lambda-1)}{\lambda}=\frac{1-e^{-\lambda}}{\lambda}$.
6 By thinning, large claims per month are $\operatorname{Poi}(0.4)$: $P(0)=e^{-0.4}\approx0.6703$. Per quarter they are $\operatorname{Poi}(1.2)$ by the reproductive property: $P(\ge2)=1-e^{-1.2}(1+1.2)\approx0.3374$. Given $6$ claims, the number of large ones is $\operatorname{Bin}(6,0.1)$: $6\times0.1\times0.9^5\approx0.3543$.
7 Exactly, $\binom{500}{2}0.004^2\,0.996^{498}\approx0.27121$. Poisson with $\lambda=2$: $2e^{-2}\approx0.27067$. Le Cam: every probability differs by at most $np^2=500\times0.004^2=0.008$ (the actual total variation distance is about $0.0009$).
8 Let $q_r=1-p_r$, so $rq_r\to\lambda$. Then $$P(Y=k)=\frac{r(r+1)\cdots(r+k-1)}{k!}p_r^rq_r^k=\frac{(rq_r)^k}{k!}\prod_{i\lt k}\left(1+\frac ir\right)\cdot p_r^r.$$ The first factor tends to $\frac{\lambda^k}{k!}$ and the product to $1$, and $r\log p_r=r\log(1-q_r)=-rq_r+O(rq_r^2)\to-\lambda$, so $p_r^r\to e^{-\lambda}$. Interactive 1 (third mode) shows the convergence.
9 $$P(X=k)=\int_0^\infty e^{-\theta}\frac{\theta^k}{k!}\cdot\frac{\lambda^r\theta^{r-1}e^{-\lambda\theta}}{\Gamma(r)}d\theta=\frac{\lambda^r}{k!\,\Gamma(r)}\cdot\frac{\Gamma(k+r)}{(1+\lambda)^{k+r}}=\frac{\Gamma(k+r)}{k!\,\Gamma(r)}\left(\frac{\lambda}{1+\lambda}\right)^r\left(\frac{1}{1+\lambda}\right)^k,$$ the $\operatorname{NB}(r,\frac{\lambda}{1+\lambda})$ pmf. Then $E[X]=E[\Lambda]=\frac r\lambda$, and by the law of total variance $\operatorname{Var}(X)=E[\Lambda]+\operatorname{Var}(\Lambda)=\frac r\lambda+\frac{r}{\lambda^2}\gt E[X]$, in agreement with Theorem 2.3: $\frac{rq}{p^2}=\frac{r(1+\lambda)}{\lambda^2}$.
III
Exponential, gamma and beta laws.
  • 1 $X$ is exponential with median $2$. Find its rate, its mean and $P(X\gt4\mid X\gt1)$.
  • 2 $X\sim\operatorname{Exp}(\lambda)$. Show that $\lfloor X\rfloor$ is geometric on $\{0,1,\dots\}$ with $p=1-e^{-\lambda}$, and that $X-\lfloor X\rfloor$ is independent of $\lfloor X\rfloor$ with density $\frac{\lambda e^{-\lambda u}}{1-e^{-\lambda}}$ on $(0,1)$.
  • 3 $X\sim\operatorname{Gamma}(3,2)$. Find $E[X]$, $\operatorname{Var}(X)$, $E[1/X]$ and $P(X\le1)$.
  • 4 Evaluate $\int_0^\infty x^5e^{-2x}dx$ and $\int_0^1x^3(1-x)^4dx$ without integrating by parts.
  • 5 $X\sim\operatorname{Beta}(a,b)$ with $a\gt1$. Show that $E[1/X]=\frac{a+b-1}{a-1}$.
  • 6 $X,Y$ are iid $\operatorname{Exp}(\lambda)$. Show that $\frac{X}{X+Y}\sim U(0,1)$, and find the density of $X/Y$. Does $X/Y$ have a mean?
  • 7 (Reliability) A machine runs on one component and has two spares; when the working component fails, a spare is switched in instantly. Component lifetimes are iid exponential with mean $200$ hours. Find the mean and standard deviation of the machine's lifetime, and the probability that it runs for more than $800$ hours.
  • 8 Write $P(\operatorname{Bin}(5,0.3)\ge3)$ as a beta integral using Theorem 2.13(a), evaluate the integral, and check it against the binomial sum.
Check lab: beta integrals are binomial tails (Theorem 2.13(a))

Choose $n$, $k$ and $p$. The lab computes $P(\operatorname{Bin}(n,p)\ge k)$ as a finite sum and $\frac{n!}{(k-1)!(n-k)!}\int_0^pt^{k-1}(1-t)^{n-k}dt$ by a 2000-step Simpson rule. They agree to many decimals, as the telescoping proof promises.

530.30
binomial sum—
beta integral—
difference—
1 $e^{-2\lambda}=\tfrac12$ gives $\lambda=\frac{\log2}{2}\approx0.3466$. The mean is $\frac{2}{\log2}\approx2.885$. By memorylessness $P(X\gt4\mid X\gt1)=P(X\gt3)=e^{-3\lambda}=2^{-3/2}\approx0.3536$.
2 Put $q=e^{-\lambda}$. For $k\ge0$ and $0\le u\lt1$, $$P(\lfloor X\rfloor=k,\ X-\lfloor X\rfloor\le u)=P(k\le X\le k+u)=q^k(1-e^{-\lambda u}).$$ This factorises as $q^k(1-q)\cdot\frac{1-e^{-\lambda u}}{1-q}$, a geometric pmf with $p=1-q$ times a cdf on $[0,1)$. So the two parts are independent, and differentiating the second factor gives the stated density.
3 $E[X]=\tfrac32$, $\operatorname{Var}(X)=\tfrac34$, $E[1/X]=\frac{\lambda}{\alpha-1}=1$. By Theorem 2.11, $P(X\le1)=P(\operatorname{Poi}(2)\ge3)=1-5e^{-2}\approx0.3233$.
4 $\int_0^\infty x^5e^{-2x}dx=\frac{\Gamma(6)}{2^6}=\frac{120}{64}=1.875$. $\int_0^1x^3(1-x)^4dx=B(4,5)=\frac{3!\,4!}{8!}=\frac{1}{280}$.
5 $E[X^{-1}]=\frac{B(a-1,b)}{B(a,b)}=\frac{\Gamma(a-1)\Gamma(a+b)}{\Gamma(a)\Gamma(a+b-1)}=\frac{a+b-1}{a-1}$, using Theorem 2.12(1) with $r=-1\gt-a$.
6 Theorem 2.12(2) with $a=b=1$ gives $V=\frac{X}{X+Y}\sim\operatorname{Beta}(1,1)=U(0,1)$. Then $\frac XY=\frac{V}{1-V}$, and $P\left(\frac XY\le t\right)=P\left(V\le\frac{t}{1+t}\right)=\frac{t}{1+t}$ for $t\gt0$, with density $\frac{1}{(1+t)^2}$. Since $\int_0^\infty\frac{t\,dt}{(1+t)^2}=\infty$, $X/Y$ has no mean, although $X$ and $Y$ have all moments.
7 The lifetime is $T=E_1+E_2+E_3\sim\operatorname{Gamma}(3,\tfrac1{200})$: mean $600$ h, SD $200\sqrt3\approx346.4$ h. By Theorem 2.11 with $\lambda t=4$, $P(T\gt800)=P(\operatorname{Poi}(4)\le2)=e^{-4}(1+4+8)=13e^{-4}\approx0.2381$.
8 With $n=5$, $k=3$: $\frac{5!}{2!\,2!}=30$, so $P=30\int_0^{0.3}t^2(1-t)^2dt=30\left[\tfrac{t^3}{3}-\tfrac{t^4}{2}+\tfrac{t^5}{5}\right]_0^{0.3}=30(0.009-0.00405+0.000486)=0.16308$. Binomial: $10(0.027)(0.49)+5(0.0081)(0.7)+0.00243=0.1323+0.02835+0.00243=0.16308$.
IV
The normal family and its relatives.
  • 1 $X\sim N(10,4)$. Find $P(8\lt X\lt13)$, the $90$th percentile and $P(|X-10|\gt3)$.
  • 2 Exam marks are $N(\mu,\sigma^2)$. $10\%$ of candidates score at most $60$ and $5\%$ score above $90$. Find $\mu$ and $\sigma$.
  • 3 For $X\sim N(\mu,\sigma^2)$, show that $E|X-\mu|=\sigma\sqrt{2/\pi}$, and compute $E[Z^6]$ for $Z\sim N(0,1)$.
  • 4 Prove that $\left(\frac1z-\frac1{z^3}\right)\varphi(z)\le1-\Phi(z)$ for $z\gt0$. Give the two Mills bounds at $z=3$ and compare with $1-\Phi(3)$.
  • 5 (Income) A lognormal income distribution has median $\text{₹}\,100$ thousand and mean $\text{₹}\,120$ thousand. Find $\sigma^2$, the variance, and the proportion of incomes above $\text{₹}\,200$ thousand.
  • 6 $X\sim C(0,1)$. Find $P(|X|\gt1)$ and $P(X\gt\sqrt3)$. By computing $P(|Y|\lt2)$, show that $Y=X+1/X$ does not have a Cauchy law.
  • 7 $X\sim\operatorname{Laplace}(0,b)$ has variance $8$. Find $b$, $P(|X|\gt4)$ and the kurtosis $\beta_2$.
  • 8 (Reliability) Units have Weibull lifetimes with shape $1.5$ and scale $1000$ h. Find the mean and median life, the hazard rate at $500$ h and at $2000$ h, and the probability that a unit which has survived $1000$ h survives another $500$ h.
1 $\sigma=2$. $P(8\lt X\lt13)=\Phi(1.5)-\Phi(-1)=0.9332-0.1587=0.7745$. The $90$th percentile is $10+1.2816\times2\approx12.563$. $P(|X-10|\gt3)=2(1-\Phi(1.5))\approx0.1336$.
2 $\frac{60-\mu}{\sigma}=\Phi^{-1}(0.10)=-1.2816$ and $\frac{90-\mu}{\sigma}=\Phi^{-1}(0.95)=1.6449$. Subtracting, $\frac{30}{\sigma}=2.9264$, so $\sigma\approx10.25$ and $\mu=60+1.2816\sigma\approx73.14$.
3 $E|Z|=2\int_0^\infty z\varphi(z)dz=2\varphi(0)=\sqrt{2/\pi}$, and $X-\mu=\sigma Z$. By Theorem 2.14(3), $E[Z^6]=1\cdot3\cdot5=15$.
4 Differentiate: $\frac{d}{dt}\left[-\left(\tfrac1t-\tfrac1{t^3}\right)\varphi(t)\right]=\left(\tfrac1{t^2}-\tfrac3{t^4}\right)\varphi(t)+\left(\tfrac1t-\tfrac1{t^3}\right)t\varphi(t)=\left(1-\tfrac3{t^4}\right)\varphi(t)$. Integrating from $z$ to $\infty$, $$\left(\tfrac1z-\tfrac1{z^3}\right)\varphi(z)=\int_z^\infty\left(1-\tfrac{3}{t^4}\right)\varphi(t)\,dt\le\int_z^\infty\varphi(t)\,dt=1-\Phi(z).$$ At $z=3$, $\varphi(3)\approx0.004432$, so $0.001313\le1-\Phi(3)\le0.001477$; the true value is $0.001350$.
5 Median $e^\mu=100$, and $\frac{\text{mean}}{\text{median}}=e^{\sigma^2/2}=1.2$, so $\sigma^2=2\log1.2\approx0.3646$. Variance $=\text{mean}^2(e^{\sigma^2}-1)=14400\times0.44=6336$ (thousand rupees squared), an SD of about $79.6$ thousand. $P(X\gt200)=1-\Phi\left(\frac{\log2}{\sigma}\right)=1-\Phi(1.148)\approx0.1255$.
6 $P(|X|\gt1)=1-\frac2\pi\arctan1=\frac12$, and $P(X\gt\sqrt3)=\frac12-\frac1\pi\cdot\frac\pi3=\frac16$. For $Y=X+\frac1X$: $|x+\frac1x|\ge2$ for every real $x\ne0$ (AM–GM), so $P(|Y|\lt2)=0$, whereas every Cauchy law gives positive probability to every interval. (Compare Example 7(e), where a different rational function of $X$ is Cauchy.)
7 $2b^2=8$ gives $b=2$. $P(|X|\gt4)=e^{-4/2}=e^{-2}\approx0.1353$, since $|X|\sim\operatorname{Exp}(\tfrac12)$. From the mgf $(1-b^2t^2)^{-1}=\sum b^{2k}t^{2k}$, $E[X^4]=4!\,b^4$ and $E[X^2]=2b^2$, so $\beta_2=\frac{24b^4}{4b^4}=6$: heavier tails than the normal.
8 Mean $1000\,\Gamma(1+\tfrac{1}{1.5})=1000\,\Gamma(\tfrac53)\approx902.7$ h. Median $1000(\log2)^{1/1.5}\approx783.2$ h. $h(t)=\frac{1.5}{1000}\left(\frac{t}{1000}\right)^{0.5}$: $h(500)\approx0.00106$ and $h(2000)\approx0.00212$ per hour. $P(T\gt1500\mid T\gt1000)=\exp\left(-(1.5^{1.5}-1)\right)=e^{-0.8371}\approx0.4330$, much less than $P(T\gt500)=e^{-0.3536}\approx0.7022$ for a new unit.
🎯 Exam corner CSIR Part BCSIR Part CGATE ST

E1. CSIR Part B $X$ is a Poisson random variable with $P(X=2)=3P(X=4)$. Then $E[X^2]$ equals (A) $2$; (B) $4$; (C) $6$; (D) $8$.

Idea Cancel $e^{-\lambda}$ in the given equation, then use $E[X^2]=\operatorname{Var}(X)+E[X]^2=\lambda+\lambda^2$.
Compute $\frac{\lambda^2}{2}=3\cdot\frac{\lambda^4}{24}$ gives $\lambda^2=4$, so $\lambda=2$ and $E[X^2]=2+4=6$. Option (A) is the mean (and variance), and (B) is $\lambda^2$ alone.
Answer (C).

E2. CSIR Part B GATE ST $X_1,\dots,X_5$ are iid exponential with mean $2$. Then $P(X_1+\dots+X_5\gt10)$ equals (A) $e^{-5}\left(1+5+\tfrac{25}{2}+\tfrac{125}{6}+\tfrac{625}{24}\right)$; (B) $e^{-5}\left(1+5+\tfrac{25}{2}+\tfrac{125}{6}+\tfrac{625}{24}+\tfrac{3125}{120}\right)$; (C) $1-e^{-5}\left(1+5+\tfrac{25}{2}+\tfrac{125}{6}+\tfrac{625}{24}\right)$; (D) $e^{-2}$.

Idea The sum is $\operatorname{Gamma}(5,\tfrac12)$ (rate $\tfrac12$). By Theorem 2.11, $P(S_5\gt t)=P(N(t)\le4)$ with $N(t)\sim\operatorname{Poi}(t/2)$.
Compute With $t=10$, $N(10)\sim\operatorname{Poi}(5)$, so $P(S_5\gt10)=\sum_{k=0}^4e^{-5}\frac{5^k}{k!}\approx0.4405$. Option (B) sums to $k=5$ (an off-by-one error, $\approx0.6160$), and (C) is $P(S_5\le10)$.
Answer (A).

E3. CSIR Part B $U_1,\dots,U_4$ are iid $U(0,1)$ and $0\lt p\lt1$. The probability that at least two of the $U_i$ are at most $p$ equals $\int_0^pc\,t(1-t)^2\,dt$, where $c$ is (A) $4$; (B) $6$; (C) $12$; (D) $24$.

Idea The count of $U_i\le p$ is $\operatorname{Bin}(4,p)$, and Theorem 2.13(a) with $n=4$, $k=2$ writes its tail as a $\operatorname{Beta}(2,3)$ cdf.
Compute $c=\frac{4!}{1!\,2!}=12=\frac{1}{B(2,3)}$. Check at $p=1$: $12\int_0^1t(1-t)^2dt=12\cdot\frac1{12}=1$, as it must be. Option (B) is $\binom42$, the coefficient of a single binomial term.
Answer (C).

E4. CSIR Part C Which of the following statements are true for all $s,t\gt0$? (One or more options may be correct.) (A) If $X\sim\operatorname{Exp}(\lambda)$, then $P(X\gt2t\mid X\gt t)=P(X\gt t)$. (B) If $X\sim U(0,1)$, then $P(X\gt s+t\mid X\gt s)=P(X\gt t)$ whenever $s+t\lt1$. (C) If $X\sim\operatorname{Gamma}(2,1)$, then $P(X\gt s+t\mid X\gt s)\le P(X\gt t)$. (D) If $P(X\gt x)=e^{-\sqrt x}$ for $x\ge0$, then $P(X\gt s+t\mid X\gt s)\le P(X\gt t)$.

(A) True Memorylessness with $s=t$: $e^{-2\lambda t}/e^{-\lambda t}=e^{-\lambda t}$.
(B) False $\frac{1-s-t}{1-s}=1-\frac{t}{1-s}\lt1-t$ for $s,t\gt0$. A used uniform unit is worse than a new one.
(C) True $P(X\gt x)=e^{-x}(1+x)$ for $\operatorname{Gamma}(2,1)$ (Theorem 2.11 with $n=2$). The conditional probability is $e^{-t}\frac{1+s+t}{1+s}=e^{-t}\left(1+\frac{t}{1+s}\right)\le e^{-t}(1+t)$.
(D) False This is $\operatorname{Weib}(\tfrac12,1)$, with decreasing hazard. Since $\sqrt{s+t}-\sqrt s\lt\sqrt t$ for $s,t\gt0$, the conditional probability $e^{-(\sqrt{s+t}-\sqrt s)}$ is strictly greater than $e^{-\sqrt t}$: a used unit is better than a new one.
Answer (A) and (C).

E5. CSIR Part C GATE ST Let $X$ and $Y$ be independent. Which of the following are true? (A) If $X\sim\operatorname{Poi}(1)$ and $Y\sim\operatorname{Poi}(2)$, then $P(X=1\mid X+Y=3)=\tfrac49$. (B) If $X\sim\operatorname{Bin}(3,\tfrac12)$ and $Y\sim\operatorname{Bin}(3,\tfrac13)$, then $X+Y$ is binomial. (C) If $X,Y$ are iid $N(0,1)$, then $\tfrac12(X+Y)^2\sim\chi^2_1$. (D) If $X,Y$ are iid $\operatorname{Exp}(1)$, then $X-Y$ has density $\tfrac12e^{-|x|}$.

(A) True By Theorem 2.7(b), $X\mid X+Y=3\sim\operatorname{Bin}(3,\tfrac13)$, so the probability is $3\cdot\tfrac13\cdot\tfrac49=\tfrac49$.
(B) False Example 7(c): the mean is $2.5$, so it would have to be $\operatorname{Bin}(6,\tfrac5{12})$ with variance $\frac{35}{24}\approx1.458$, but $\operatorname{Var}(X+Y)=\tfrac34+\tfrac23=\tfrac{17}{12}\approx1.417$.
(C) True $X+Y\sim N(0,2)$ by Theorem 2.14(4), so $\frac{X+Y}{\sqrt2}\sim N(0,1)$ and its square is $\chi^2_1$.
(D) True Theorem 2.15(3) with $\theta=0$, $b=1$.
Answer (A), (C) and (D).

E6. CSIR Part C Which of the following are true? (A) If $X_1,\dots,X_n$ are iid standard Cauchy, then $\bar X_n$ is standard Cauchy. (B) If $Z\sim N(0,1)$, then $E\left[e^{Z^2/4}\right]\lt\infty$. (C) If $\log X\sim N(0,1)$, then $E[e^{tX}]\lt\infty$ for all $t$ in some neighbourhood of $0$. (D) If $X\sim\operatorname{Gamma}(\tfrac12,1)$, then $E[1/X]\lt\infty$.

(A) True Theorem 2.15(2) (proved with characteristic functions in §2.2). So averaging Cauchy observations gives no gain at all.
(B) True $Z^2\sim\chi^2_1=\operatorname{Gamma}(\tfrac12,\tfrac12)$, whose mgf $\left(\frac{1/2}{1/2-t}\right)^{1/2}=(1-2t)^{-1/2}$ is finite for $t\lt\tfrac12$. At $t=\tfrac14$ it equals $\sqrt2$.
(C) False $X$ is lognormal, and $E[e^{tX}]=\infty$ for every $t\gt0$ (Theorem 2.15(1)), although $E[X]=e^{1/2}$ and all moments are finite.
(D) False $E[X^{-1}]$ is finite iff $\alpha\gt1$ (Theorem 2.10(1)); here the integrand behaves like $x^{-3/2}$ near $0$.
Answer (A) and (B).
🧑‍🏫 For the teacher

Suggested plan: 3 lectures × 60 min.

  1. Bernoulli trials and Poisson. Definition 2.1 with the four questions about a sequence of trials. Theorem 2.1 (do the mode proof on the board), Theorem 2.2 with the characterisation, Theorem 2.3 via pgfs, Theorem 2.4 via indicators (stress the negative covariance). Example 1. Then the Poisson law: Theorem 2.6(a), and the coupling proof of Le Cam's inequality if the class is strong. Project Interactive 1 in all three modes. Example 3 and Theorem 2.7.
  2. Waiting times. Theorem 2.8 (prove $\Gamma(\frac12)=\sqrt\pi$ through the beta integral, which also normalises the normal density). Theorem 2.9 with the Cauchy functional equation. Theorems 2.10–2.11: emphasise the Poisson–gamma identity as a calculation tool. Example 4. Theorems 2.12–2.13 with Example 5 and the check lab.
  3. The normal family and the map. Theorem 2.14, values of $\Phi$, Mills' bound. Definition 2.7 and Theorem 2.15 (the ratio of normals is Cauchy is a good Jacobian exercise). Interactive 2 for the hazard shapes and for the tails comparison. Theorem 2.16, the two tables and Example 7 as a class poll. Finish with E4–E6.

Discussion questions. (1) The Poisson law has mean equal to variance. Real count data (goals per match, hospital admissions per day) often have variance larger than the mean. What mechanism could cause this, and which law from this section would you fit instead? (2) Why can a sum of independent binomials with different $p$ never be binomial, while a sum of independent Poissons with different means always is Poisson? (3) The normal, Laplace and Cauchy laws in Interactive 2 have the same median and quartiles. Which of them would you choose to model daily stock returns, and why?

Misconceptions to watch for. Mixing up rate and mean of the exponential and gamma laws; using the trials form of the geometric law with the failures formula for the mean; forgetting the finite population correction; applying the gamma reproductive property with different rates; thinking that a symmetric law (Cauchy) must have mean equal to its centre; believing that the Poisson approximation works whenever $n$ is large; adding standard deviations instead of variances for sums of independent normals.

Suggested homework. I(1, 3, 4, 6, 8), II(2, 4, 6, 8, 9), III(2, 3, 6, 7), IV(2, 4, 5, 8); Exam corner E2, E4, E6.

🔬 Research–Teaching NexusResearch-tutored · optional, about 60 minutes

Research question. Are real counts of rare events (goals in football matches, daily emergency admissions, earthquakes above a given magnitude per year) really Poisson, or are they overdispersed, and does it matter?

Why it matters. Hospital managers staff emergency departments, insurers price policies and seismologists issue hazard maps using Poisson models. If the true variance is larger than the mean, a Poisson model underestimates the probability of extreme days: too few beds, too little capital. Exercise II.9 shows one cause: if the Poisson rate itself varies (between days, teams or regions) according to a gamma law, the counts become negative binomial with variance $\mu+\mu^2/r$.

What to do.

  1. Find a public data set of counts: for example, goals per team per match in a national football league for one season, or daily counts of a notifiable disease. Compute the sample mean $\bar x$, the sample variance $s^2$ and the dispersion index $s^2/\bar x$.
  2. Under a Poisson model the dispersion index should be close to $1$. Simulate $1000$ data sets of the same size from $\operatorname{Poi}(\bar x)$ with a seeded generator, compute the dispersion index of each, and find where your real value lies in this simulated distribution.
  3. Fit a negative binomial law by the method of moments: $\hat p=\bar x/s^2$ and $\hat r=\bar x^2/(s^2-\bar x)$ (valid when $s^2\gt\bar x$). Compare the observed frequencies of $0,1,2,\dots$ with the Poisson and negative binomial fitted frequencies in a table and a bar chart.
  4. Estimate, under each model, the probability of a count at least three standard deviations above the mean, and discuss the practical consequences of the difference.

Tools: Python (numpy, scipy.stats) or R; Interactive 1 in its negative binomial mode as a picture of how $r$ controls the excess variance.

Hand in: A two-page report with the dispersion index and its simulated reference distribution, the fitted-frequency table and chart, and a recommendation on which model to use.

Going further. The chi-square goodness-of-fit test (§5.2) turns the comparison of observed and fitted frequencies into a formal test. Read about zero-inflated Poisson models, used when there are more zeros than either model predicts, and about Bortkiewicz's classical data on deaths by horse kicks in the Prussian army, often quoted as an example of a good Poisson fit.

Instructor note: goals per team per match in major leagues typically have a mean around $1.3$–$1.5$ and a dispersion index only slightly above $1$, so the Poisson model is reasonable there; daily disease or admission counts usually show strong overdispersion (indices of $2$ or more), driven by weekly and seasonal variation. For a simulated Poisson sample of size $n$, the dispersion index has standard deviation about $\sqrt{2/(n-1)}$, which gives students a quick yardstick.

🧪 Self-check quiz

0/5 correct

Q1. A binomial random variable has mean $6$ and variance $4.2$. Its parameters are

Q2. Which of these positive continuous distributions is memoryless?

Q3. $X$ has density proportional to $x^2e^{-2x}$ on $(0,\infty)$. Then $\operatorname{Var}(X)$ is

Q4. $X\sim\operatorname{Poi}(3)$ and $Y\sim\operatorname{Poi}(5)$ are independent. Then $E[X\mid X+Y=8]$ equals

Q5. $Z_1,Z_2$ are independent standard normal variables. The ratio $Z_1/Z_2$ has