When $A\mathbf{x}=\mathbf{b}$ has no solution, find the next best thing: project $\mathbf{b}$ onto $\operatorname{Col}A$ and solve the normal equations $A^{T}A\hat{\mathbf{x}}=A^{T}\mathbf{b}$.
Real measurements are never perfect. A physics student hangs weights on a spring and records its length; a climate scientist records a yearly mean temperature; an engineer logs the output of a sensor. Theory says the points should lie on a straight line (or a parabola), but the data always wobble a little. If we write "the line passes through every point" as a linear system $A\mathbf{x}=\mathbf{b}$, there are more equations than unknowns and the system is inconsistent — it has no solution at all.
Giving up is not an option: we still want the line that the data suggest. The idea of this section is geometric. Every vector $A\mathbf{x}$ lies in the column space $\operatorname{Col}A$; the vector $\mathbf{b}$ does not. So we look for the point of $\operatorname{Col}A$ that is closest to $\mathbf{b}$. That closest point is the orthogonal projection of $\mathbf{b}$ onto $\operatorname{Col}A$, and the $\hat{\mathbf{x}}$ that produces it is called a least-squares solution. Projections onto lines were met in Section 4.2 (and used again in Sections 8.1 and 8.2); here we project onto planes and higher-dimensional subspaces, and then turn the geometry into a practical method for fitting lines and curves to data.
The Orthogonal Decomposition Theorem
Recall from Section 4.2 that the projection of $\mathbf{y}$ onto a nonzero vector $\mathbf{u}$ is $\operatorname{proj}_{\mathbf{u}}\mathbf{y}=\dfrac{\mathbf{y}\cdot\mathbf{u}}{\mathbf{u}\cdot\mathbf{u}}\,\mathbf{u}$, and that $\mathbf{y}-\operatorname{proj}_{\mathbf{u}}\mathbf{y}$ is orthogonal to $\mathbf{u}$. Also recall from Section 8.1 that the orthogonal complement of a subspace $W$ of $\mathbb{R}^n$ is $W^{\perp}=\{\mathbf{z}\in\mathbb{R}^n:\ \mathbf{z}\cdot\mathbf{w}=0\ \text{for every}\ \mathbf{w}\in W\}$. To project onto a subspace with an orthogonal basis, we simply project onto each basis vector and add.
📘 Definition 8.6 — Orthogonal projection onto a subspace
Let $W$ be a nonzero subspace of $\mathbb{R}^n$ with an orthogonal basis $\{\mathbf{u}_1,\dots,\mathbf{u}_p\}$. The orthogonal projection of $\mathbf{y}\in\mathbb{R}^n$ onto $W$ is
$$\operatorname{proj}_W\mathbf{y}=\frac{\mathbf{y}\cdot\mathbf{u}_1}{\mathbf{u}_1\cdot\mathbf{u}_1}\,\mathbf{u}_1+\frac{\mathbf{y}\cdot\mathbf{u}_2}{\mathbf{u}_2\cdot\mathbf{u}_2}\,\mathbf{u}_2+\cdots+\frac{\mathbf{y}\cdot\mathbf{u}_p}{\mathbf{u}_p\cdot\mathbf{u}_p}\,\mathbf{u}_p .$$
We usually write $\hat{\mathbf{y}}=\operatorname{proj}_W\mathbf{y}$. If the basis is orthonormal, every denominator is $1$ and $\hat{\mathbf{y}}=(\mathbf{y}\cdot\mathbf{u}_1)\mathbf{u}_1+\cdots+(\mathbf{y}\cdot\mathbf{u}_p)\mathbf{u}_p$.
📐 Theorem 8.9 — The Orthogonal Decomposition Theorem
Let $W$ be a subspace of $\mathbb{R}^n$. Then every $\mathbf{y}\in\mathbb{R}^n$ can be written uniquely in the form
$$\mathbf{y}=\hat{\mathbf{y}}+\mathbf{z},\qquad\hat{\mathbf{y}}\in W,\quad\mathbf{z}\in W^{\perp}.$$
If $\{\mathbf{u}_1,\dots,\mathbf{u}_p\}$ is any orthogonal basis of $W$, then $\hat{\mathbf{y}}=\operatorname{proj}_W\mathbf{y}$ is given by Definition 8.6 and $\mathbf{z}=\mathbf{y}-\hat{\mathbf{y}}$. In particular $\hat{\mathbf{y}}$ does not depend on which orthogonal basis is used. (If $W=\{\mathbf{0}\}$, then $\hat{\mathbf{y}}=\mathbf{0}$ and $\mathbf{z}=\mathbf{y}$.)
Why it is true. $\hat{\mathbf{y}}$ is a combination of the $\mathbf{u}_i$, so it lies in $W$. For each $j$, all the cross terms vanish because the basis is orthogonal:
$$\mathbf{z}\cdot\mathbf{u}_j=\mathbf{y}\cdot\mathbf{u}_j-\frac{\mathbf{y}\cdot\mathbf{u}_j}{\mathbf{u}_j\cdot\mathbf{u}_j}\,(\mathbf{u}_j\cdot\mathbf{u}_j)=0,$$
so $\mathbf{z}$ is orthogonal to every basis vector, hence to all of $W$. For uniqueness, if $\hat{\mathbf{y}}+\mathbf{z}=\hat{\mathbf{y}}_1+\mathbf{z}_1$ are two such decompositions, then $\hat{\mathbf{y}}-\hat{\mathbf{y}}_1=\mathbf{z}_1-\mathbf{z}$ lies in both $W$ and $W^{\perp}$, so it is orthogonal to itself and must be $\mathbf{0}$.
✏️ Example 1 — Decomposing a vector in $\mathbb{R}^3$
Let $\mathbf{u}_1=\begin{bmatrix}1\\1\\0\end{bmatrix}$, $\mathbf{u}_2=\begin{bmatrix}1\\-1\\2\end{bmatrix}$, $W=\operatorname{span}\{\mathbf{u}_1,\mathbf{u}_2\}$ and $\mathbf{y}=\begin{bmatrix}3\\1\\5\end{bmatrix}$. Write $\mathbf{y}$ as the sum of a vector in $W$ and a vector in $W^{\perp}$, and find the distance from $\mathbf{y}$ to the plane $W$.
💡 Show solution▶
Orthogonal? $\mathbf{u}_1\cdot\mathbf{u}_2=1-1+0=0$, so $\{\mathbf{u}_1,\mathbf{u}_2\}$ is an orthogonal basis of $W$ and Definition 8.6 applies.
Remainder $\mathbf{z}=\mathbf{y}-\hat{\mathbf{y}}=\begin{bmatrix}-1\\1\\1\end{bmatrix}$, so $$\begin{bmatrix}3\\1\\5\end{bmatrix}=\underbrace{\begin{bmatrix}4\\0\\4\end{bmatrix}}_{\in W}+\underbrace{\begin{bmatrix}-1\\1\\1\end{bmatrix}}_{\in W^{\perp}}.$$
Check $\mathbf{z}\cdot\mathbf{u}_1=-1+1+0=0$ and $\mathbf{z}\cdot\mathbf{u}_2=-1-1+2=0$. ✓ By the Best Approximation Theorem below, the distance from $\mathbf{y}$ to $W$ is $\|\mathbf{z}\|=\sqrt{1+1+1}=\sqrt3\approx1.732$.
The closest point
The picture behind Theorem 8.9 is a right angle: $\mathbf{y}$ is the hypotenuse, $\hat{\mathbf{y}}$ lies in the "floor" $W$ and $\mathbf{z}$ stands straight up from it. Any other point $\mathbf{v}$ of the floor is further from $\mathbf{y}$, by Pythagoras.
📐 Theorem 8.10 — The Best Approximation Theorem
Let $W$ be a subspace of $\mathbb{R}^n$, let $\mathbf{y}\in\mathbb{R}^n$, and let $\hat{\mathbf{y}}=\operatorname{proj}_W\mathbf{y}$. Then $\hat{\mathbf{y}}$ is the point of $W$ closest to $\mathbf{y}$:
$$\|\mathbf{y}-\hat{\mathbf{y}}\|\lt\|\mathbf{y}-\mathbf{v}\|\qquad\text{for every }\mathbf{v}\in W\text{ with }\mathbf{v}\ne\hat{\mathbf{y}}.$$
The number $\|\mathbf{y}-\hat{\mathbf{y}}\|$ is called the distance from $\mathbf{y}$ to $W$.
Proof. Write $\mathbf{y}-\mathbf{v}=(\mathbf{y}-\hat{\mathbf{y}})+(\hat{\mathbf{y}}-\mathbf{v})$. The first piece is in $W^{\perp}$ and the second is in $W$ (a difference of two vectors of $W$), so they are orthogonal and the Pythagorean Theorem gives $\|\mathbf{y}-\mathbf{v}\|^2=\|\mathbf{y}-\hat{\mathbf{y}}\|^2+\|\hat{\mathbf{y}}-\mathbf{v}\|^2$. The last term is positive when $\mathbf{v}\ne\hat{\mathbf{y}}$. $\square$
The projection matrix
Definition 8.6 needs an orthogonal basis. When we only have an ordinary basis, put it into the columns of a matrix $A$, so that $W=\operatorname{Col}A$. The projection $\hat{\mathbf{y}}$ is then $A\mathbf{c}$ for some weights $\mathbf{c}$, and $\mathbf{y}-A\mathbf{c}$ must be orthogonal to every column of $A$, i.e. $A^{T}(\mathbf{y}-A\mathbf{c})=\mathbf{0}$. Solving for $\mathbf{c}$ gives a formula that needs no orthogonality at all.
📐 Theorem 8.11 — The projection matrix
Let $A$ be an $m\times n$ matrix with linearly independent columns and let $W=\operatorname{Col}A$. Then $A^{T}A$ is invertible and, for every $\mathbf{y}\in\mathbb{R}^m$,
$$\operatorname{proj}_W\mathbf{y}=P\mathbf{y},\qquad P=A\left(A^{T}A\right)^{-1}A^{T}.$$
The $m\times m$ matrix $P$ is the projection matrix onto $W$. It is symmetric ($P^{T}=P$) and idempotent ($P^2=P$: projecting twice changes nothing). If the columns of $A$ are orthonormal, say $A=U$, then $U^{T}U=I$ and the formula collapses to $P=UU^{T}$.
✏️ Example 2 — Projecting onto a plane through the origin
Let $W$ be the plane $x-2y+z=0$ in $\mathbb{R}^3$. Find the projection matrix onto $W$, the projection of $\mathbf{y}=\begin{bmatrix}1\\1\\7\end{bmatrix}$, and the distance from $\mathbf{y}$ to $W$.
💡 Show solution▶
Basis Solve $x=2y-z$ with $y,z$ free: $W=\operatorname{span}\left\{\begin{bmatrix}2\\1\\0\end{bmatrix},\begin{bmatrix}-1\\0\\1\end{bmatrix}\right\}$. These are not orthogonal (their dot product is $-2$), so we use Theorem 8.11 with $A=\begin{bmatrix}2&-1\\1&0\\0&1\end{bmatrix}$.
Check $\hat{\mathbf{y}}$ is in the plane: $0-6+6=0$. ✓ And $\mathbf{z}=(1,-2,1)$ is exactly the normal vector of the plane, so it is orthogonal to $W$. ✓ The distance is $\|\mathbf{z}\|=\sqrt6\approx2.449$. (Interactive 1 shows this picture in 3-D.)
Interactive 1 — Dropping a Perpendicular onto a Plane
The green plane is $W:\ x-2y+z=0$ from Example 2. Choose $\mathbf{y}$ with the sliders. Its projection $\hat{\mathbf{y}}=P\mathbf{y}$ (blue) sits in the plane and $\mathbf{z}=\mathbf{y}-\hat{\mathbf{y}}$ (red, dashed) meets the plane at a right angle. Press Play: an orange point $\mathbf{v}$ wanders around the plane, and $\|\mathbf{y}-\mathbf{v}\|$ is never smaller than $\|\mathbf{z}\|$ — the Best Approximation Theorem.
$\mathbf{y}$—
$\hat{\mathbf{y}}=P\mathbf{y}$—
$\mathbf{z}=\mathbf{y}-\hat{\mathbf{y}}$—
$\|\mathbf{z}\|$ (distance to $W$)—
$\|\mathbf{y}-\mathbf{v}\|$—
Least-Squares Problems
Now return to an inconsistent system $A\mathbf{x}=\mathbf{b}$. No $\mathbf{x}$ makes the residual $\mathbf{b}-A\mathbf{x}$ zero, so we make it as short as possible. Since $\|\mathbf{b}-A\mathbf{x}\|^2$ is a sum of squares, this is called the method of least squares.
📘 Definition 8.7 — Least-squares solution
Let $A$ be an $m\times n$ matrix and $\mathbf{b}\in\mathbb{R}^m$. A vector $\hat{\mathbf{x}}\in\mathbb{R}^n$ is a least-squares solution of $A\mathbf{x}=\mathbf{b}$ if
$$\|\mathbf{b}-A\hat{\mathbf{x}}\|\le\|\mathbf{b}-A\mathbf{x}\|\qquad\text{for every }\mathbf{x}\in\mathbb{R}^n.$$
The vector $\mathbf{b}-A\hat{\mathbf{x}}$ is the least-squares residual and its length $\|\mathbf{b}-A\hat{\mathbf{x}}\|$ is the least-squares error. (If the system is consistent, the least-squares solutions are just the ordinary solutions and the error is $0$.)
As $\mathbf{x}$ ranges over $\mathbb{R}^n$, $A\mathbf{x}$ ranges over $\operatorname{Col}A$. By the Best Approximation Theorem the closest such vector to $\mathbf{b}$ is $\hat{\mathbf{b}}=\operatorname{proj}_{\operatorname{Col}A}\mathbf{b}$, so $\hat{\mathbf{x}}$ is a least-squares solution exactly when $A\hat{\mathbf{x}}=\hat{\mathbf{b}}$. By Theorem 8.9 that happens exactly when $\mathbf{b}-A\hat{\mathbf{x}}$ is orthogonal to $\operatorname{Col}A$, i.e. orthogonal to every column of $A$: $A^{T}\left(\mathbf{b}-A\hat{\mathbf{x}}\right)=\mathbf{0}$.
📐 Theorem 8.12 — The normal equations
Let $A$ be any $m\times n$ matrix and $\mathbf{b}\in\mathbb{R}^m$. The set of least-squares solutions of $A\mathbf{x}=\mathbf{b}$ is exactly the solution set of the normal equations
$$A^{T}A\,\hat{\mathbf{x}}=A^{T}\mathbf{b}.$$
This system is always consistent, so least-squares solutions always exist.
📐 Theorem 8.13 — When the least-squares solution is unique
For an $m\times n$ matrix $A$ the following are equivalent:
$A\mathbf{x}=\mathbf{b}$ has a unique least-squares solution for every $\mathbf{b}\in\mathbb{R}^m$;
the columns of $A$ are linearly independent;
the $n\times n$ matrix $A^{T}A$ is invertible.
When they hold, $\hat{\mathbf{x}}=\left(A^{T}A\right)^{-1}A^{T}\mathbf{b}$, and $A\hat{\mathbf{x}}=P\mathbf{b}$ with $P$ the projection matrix of Theorem 8.11.
🧭 Problem-solving strategy — solving a least-squares problem
Compute the small square matrix $A^{T}A$ ($n\times n$) and the vector $A^{T}\mathbf{b}$.
Solve $A^{T}A\hat{\mathbf{x}}=A^{T}\mathbf{b}$ by row reduction (or with $\left(A^{T}A\right)^{-1}$ when $n=2$).
Compute the residual $\mathbf{r}=\mathbf{b}-A\hat{\mathbf{x}}$ and the error $\|\mathbf{r}\|$.
Check: $A^{T}\mathbf{r}$ must be $\mathbf{0}$ — the residual is orthogonal to every column of $A$.
✏️ Example 3 — An inconsistent system
Find the least-squares solution and the least-squares error for $$\begin{aligned}x_1+2x_2&=4\\x_1&=4\\x_2&=3.\end{aligned}$$
💡 Show solution▶
Inconsistent The last two equations force $x_1=4$, $x_2=3$, but then $x_1+2x_2=10\ne4$. So there is no exact solution. Here $A=\begin{bmatrix}1&2\\1&0\\0&1\end{bmatrix}$, $\mathbf{b}=\begin{bmatrix}4\\4\\3\end{bmatrix}$.
Normal equations $$A^{T}A=\begin{bmatrix}2&2\\2&5\end{bmatrix},\qquad A^{T}\mathbf{b}=\begin{bmatrix}8\\11\end{bmatrix}.$$ The augmented matrix $\left[\begin{array}{cc|c}2&2&8\\2&5&11\end{array}\right]$ with $R_2\to R_2-R_1$ becomes $\left[\begin{array}{cc|c}2&2&8\\0&3&3\end{array}\right]$, so $\hat x_2=1$ and $2\hat x_1+2=8$, $\hat x_1=3$: $$\hat{\mathbf{x}}=\begin{bmatrix}3\\1\end{bmatrix}.$$
Residual $A\hat{\mathbf{x}}=\begin{bmatrix}5\\3\\1\end{bmatrix}$, $\mathbf{r}=\mathbf{b}-A\hat{\mathbf{x}}=\begin{bmatrix}-1\\1\\2\end{bmatrix}$, and the least-squares error is $\|\mathbf{r}\|=\sqrt{1+1+4}=\sqrt6\approx2.449$.
Check $A^{T}\mathbf{r}=\begin{bmatrix}-1+1+0\\-2+0+2\end{bmatrix}=\begin{bmatrix}0\\0\end{bmatrix}$. ✓ No other $\mathbf{x}$ does better: for instance $\mathbf{x}=(4,3)$ satisfies two equations exactly but leaves residual $(-6,0,0)$ of length $6$.
⚠️ Common mistakes with projections and least squares
1. The projection formula needs an orthogonal basis. Adding the projections onto the vectors of a non-orthogonal basis gives the wrong answer (Exercise I.8). Either make the basis orthogonal first (Gram–Schmidt, Section 8.2) or use $P=A\left(A^{T}A\right)^{-1}A^{T}$.
2. You cannot "cancel" $A^{T}$. From $A^{T}A\hat{\mathbf{x}}=A^{T}\mathbf{b}$ it does not follow that $A\hat{\mathbf{x}}=\mathbf{b}$ — that is the inconsistent system we started with. $A^{T}$ is not square and has no inverse.
3. $(A^{T}A)^{-1}\ne A^{-1}(A^{T})^{-1}$ here, because $A$ is not square and has no inverse. If $A$ were square and invertible, $P$ would be $I$ and the "least-squares solution" would be the exact one.
4. Least squares minimises vertical distances — the residuals $y_i-(c_0+c_1x_i)$ — not the perpendicular distances from the points to the line.
Fitting Lines and Parabolas to Data
Suppose we have data points $(x_1,y_1),\dots,(x_m,y_m)$ and want the line $y=c_0+c_1x$ that fits them best. "The line passes through every point" means
$$\underbrace{\begin{bmatrix}1&x_1\\1&x_2\\\vdots&\vdots\\1&x_m\end{bmatrix}}_{A\ (\text{design matrix})}\underbrace{\begin{bmatrix}c_0\\c_1\end{bmatrix}}_{\mathbf{c}}=\underbrace{\begin{bmatrix}y_1\\y_2\\\vdots\\y_m\end{bmatrix}}_{\mathbf{y}} .$$
This is $A\mathbf{x}=\mathbf{b}$ with new letters: $\mathbf{c}$ is the unknown and $\mathbf{y}$ holds the observations. The $i$-th entry of the residual $\mathbf{y}-A\mathbf{c}$ is the vertical gap $y_i-(c_0+c_1x_i)$ between the data point and the line, so least squares picks the line that minimises the sum of squared residuals (often called $SSE$).
📘 Definition 8.8 — Least-squares line
The least-squares line (or regression line) of the data $(x_1,y_1),\dots,(x_m,y_m)$ is $y=\hat c_0+\hat c_1x$, where $\hat{\mathbf{c}}=(\hat c_0,\hat c_1)$ is the least-squares solution of $A\mathbf{c}=\mathbf{y}$ above. Its normal equations are
$$\begin{bmatrix}m&\sum x_i\\\sum x_i&\sum x_i^2\end{bmatrix}\begin{bmatrix}\hat c_0\\\hat c_1\end{bmatrix}=\begin{bmatrix}\sum y_i\\\sum x_iy_i\end{bmatrix}.$$
The solution is unique as long as the $x_i$ are not all equal (then the two columns of $A$ are independent). The same idea fits a least-squares parabola $y=c_0+c_1x+c_2x^2$: use the design matrix with rows $\begin{bmatrix}1&x_i&x_i^2\end{bmatrix}$.
✏️ Example 4 — A temperature trend
A weather station reports its mean annual temperature anomaly (difference from a long-term average) every ten years. The figures below are illustrative sample data, not an official record. With $t$ = decades since 1980:
year
1980
1990
2000
2010
2020
$t$
0
1
2
3
4
anomaly $y$ (°C)
0.2
0.3
0.6
0.7
1.0
Find the least-squares line, the warming rate it suggests, the residuals and $SSE$.
Normal equations $$\begin{bmatrix}5&10\\10&30\end{bmatrix}\begin{bmatrix}\hat c_0\\\hat c_1\end{bmatrix}=\begin{bmatrix}2.8\\7.6\end{bmatrix}.$$ The determinant is $150-100=50$, so $$\begin{bmatrix}\hat c_0\\\hat c_1\end{bmatrix}=\frac1{50}\begin{bmatrix}30&-10\\-10&5\end{bmatrix}\begin{bmatrix}2.8\\7.6\end{bmatrix}=\frac1{50}\begin{bmatrix}8\\10\end{bmatrix}=\begin{bmatrix}0.16\\0.2\end{bmatrix}.$$
Interpret The least-squares line is $y=0.16+0.2t$: the data suggest warming of about $0.2$ °C per decade.
Residuals The line predicts $0.16,\ 0.36,\ 0.56,\ 0.76,\ 0.96$, so the residuals are $0.04,\ -0.06,\ 0.04,\ -0.06,\ 0.04$ (°C) and $$SSE=3(0.04)^2+2(0.06)^2=0.0048+0.0072=0.012\ \text{°C}^2 .$$
Check The residuals add to $0$ and $\sum t_ir_i=0-0.06+0.08-0.18+0.16=0$: the residual vector is orthogonal to both columns of $A$. ✓ Try to beat $SSE=0.012$ in Interactive 2 — you can't.
Interactive 2 — Least-Squares Line Lab
Drag the data points. The blue least-squares line and its red residuals update instantly. The orange dashed line is your line — move it with the sliders (or press Play to let it wander). However you set it, its sum of squared residuals is never below the least-squares value.
least-squares line—
its $SSE$ (minimum)—
your line—
your $SSE$—
✏️ Example 5 — A least-squares parabola: measuring $g$
A ball is thrown upwards and a phone camera records its height $h$ (in metres) every half second:
$t$ (s)
0
0.5
1.0
1.5
2.0
$h$ (m)
1.97
8.27
12.12
13.07
12.07
Physics predicts $h=c_0+c_1t+c_2t^2$ with $c_2=-\tfrac12g$. Fit the least-squares parabola and estimate $g$.
💡 Show solution▶
Design matrix Each row is $\begin{bmatrix}1&t_i&t_i^2\end{bmatrix}$: $$A=\begin{bmatrix}1&0&0\\1&0.5&0.25\\1&1&1\\1&1.5&2.25\\1&2&4\end{bmatrix},\qquad\mathbf{y}=\begin{bmatrix}1.97\\8.27\\12.12\\13.07\\12.07\end{bmatrix}.$$
Normal equations Using $\sum t_i=5$, $\sum t_i^2=7.5$, $\sum t_i^3=12.5$, $\sum t_i^4=22.125$: $$\begin{bmatrix}5&5&7.5\\5&7.5&12.5\\7.5&12.5&22.125\end{bmatrix}\begin{bmatrix}\hat c_0\\\hat c_1\\\hat c_2\end{bmatrix}=\begin{bmatrix}47.5\\60\\91.875\end{bmatrix}.$$
Solve Row reduction (or a calculator) gives $\hat c_0=2$, $\hat c_1=15$, $\hat c_2=-5$, so the least-squares parabola is $$h=2+15t-5t^2 .$$ Then $-\tfrac12g\approx-5$ gives $g\approx10$ m/s² — within 2% of the accepted $9.81$ m/s², which is good for a phone video.
Error Predicted heights $2,\ 8.25,\ 12,\ 13.25,\ 12$; residuals $-0.03,\ 0.02,\ 0.12,\ -0.18,\ 0.07$ m; $SSE=0.053$ m². The check $A^{T}\mathbf{r}=\mathbf{0}$ holds (try it: $\sum r_i=0$, $\sum t_ir_i=0$, $\sum t_i^2r_i=0$). ✓
Least squares with a QR factorisation
Forming $A^{T}A$ squares the sizes of the numbers involved, and on a computer this can amplify rounding errors badly. The QR factorisation of Section 8.2 avoids $A^{T}A$ altogether.
📐 Theorem 8.14 — Least squares by QR
Let $A$ be an $m\times n$ matrix with linearly independent columns and let $A=QR$ be a QR factorisation: $Q$ is $m\times n$ with orthonormal columns and $R$ is $n\times n$, upper triangular and invertible. Then for each $\mathbf{b}\in\mathbb{R}^m$ the unique least-squares solution of $A\mathbf{x}=\mathbf{b}$ is
$$\hat{\mathbf{x}}=R^{-1}Q^{T}\mathbf{b},\qquad\text{found in practice by solving }R\hat{\mathbf{x}}=Q^{T}\mathbf{b}\text{ by back substitution.}$$
Why. The columns of $Q$ are an orthonormal basis of $\operatorname{Col}A$, so $QQ^{T}\mathbf{b}=\operatorname{proj}_{\operatorname{Col}A}\mathbf{b}$ (Theorem 8.11 with $U=Q$). With $\hat{\mathbf{x}}=R^{-1}Q^{T}\mathbf{b}$ we get $A\hat{\mathbf{x}}=QRR^{-1}Q^{T}\mathbf{b}=QQ^{T}\mathbf{b}$, the projection — exactly what a least-squares solution must give.
✏️ Example 6 — A least-squares line via QR
Fit a least-squares line $y=c_0+c_1x$ to the points $(0,1)$, $(1,2)$, $(2,2)$, $(3,4)$ using the QR factorisation of the design matrix.
💡 Show solution▶
Gram–Schmidt $A=\begin{bmatrix}1&0\\1&1\\1&2\\1&3\end{bmatrix}$. With $\mathbf{a}_1=(1,1,1,1)$: $\|\mathbf{a}_1\|=2$, $\mathbf{q}_1=\tfrac12(1,1,1,1)$. Next $\mathbf{a}_2-(\mathbf{a}_2\cdot\mathbf{q}_1)\mathbf{q}_1=(0,1,2,3)-3\cdot\tfrac12(1,1,1,1)=\left(-\tfrac32,-\tfrac12,\tfrac12,\tfrac32\right)$, of length $\sqrt5$, so $\mathbf{q}_2=\tfrac{1}{2\sqrt5}(-3,-1,1,3)$.
$Q$ and $R$ $$Q=\begin{bmatrix}\tfrac12&\tfrac{-3}{2\sqrt5}\\\tfrac12&\tfrac{-1}{2\sqrt5}\\\tfrac12&\tfrac{1}{2\sqrt5}\\\tfrac12&\tfrac{3}{2\sqrt5}\end{bmatrix},\qquad R=Q^{T}A=\begin{bmatrix}2&3\\0&\sqrt5\end{bmatrix}.$$
$Q^{T}\mathbf{y}$ With $\mathbf{y}=(1,2,2,4)$: $\mathbf{q}_1\cdot\mathbf{y}=\tfrac12(9)=\tfrac92$ and $\mathbf{q}_2\cdot\mathbf{y}=\tfrac{1}{2\sqrt5}(-3-2+2+12)=\tfrac{9}{2\sqrt5}$.
Back substitute $R\hat{\mathbf{c}}=Q^{T}\mathbf{y}$: $\sqrt5\,\hat c_1=\tfrac{9}{2\sqrt5}$ gives $\hat c_1=\tfrac{9}{10}$; then $2\hat c_0+3\left(\tfrac9{10}\right)=\tfrac92$ gives $\hat c_0=\tfrac{9}{10}$. The line is $y=0.9+0.9x$.
Check Normal equations: $A^{T}A=\begin{bmatrix}4&6\\6&14\end{bmatrix}$, $A^{T}\mathbf{y}=\begin{bmatrix}9\\18\end{bmatrix}$, and $4(0.9)+6(0.9)=9$, $6(0.9)+14(0.9)=18$. ✓ Residuals $0.1,\ 0.2,\ -0.7,\ 0.4$; $SSE=0.7$.
Exercises
I
Orthogonal projections and the orthogonal decomposition.
1 Write $\mathbf{y}=\begin{bmatrix}2\\4\end{bmatrix}$ as $\hat{\mathbf{y}}+\mathbf{z}$ with $\hat{\mathbf{y}}\in W=\operatorname{span}\left\{\begin{bmatrix}1\\1\end{bmatrix}\right\}$ and $\mathbf{z}\in W^{\perp}$.
2 Find the projection of $\mathbf{y}=\begin{bmatrix}5\\-1\end{bmatrix}$ onto the line through the origin in the direction $\mathbf{u}=\begin{bmatrix}3\\4\end{bmatrix}$, and the distance from $\mathbf{y}$ to that line.
3 Let $\mathbf{u}_1=\begin{bmatrix}1\\0\\1\end{bmatrix}$, $\mathbf{u}_2=\begin{bmatrix}1\\1\\-1\end{bmatrix}$ and $\mathbf{y}=\begin{bmatrix}3\\2\\1\end{bmatrix}$. Check that $\{\mathbf{u}_1,\mathbf{u}_2\}$ is orthogonal and find $\operatorname{proj}_W\mathbf{y}$ for $W=\operatorname{span}\{\mathbf{u}_1,\mathbf{u}_2\}$.
4 For part 3, find $\mathbf{z}=\mathbf{y}-\hat{\mathbf{y}}$, verify that it is orthogonal to $W$, and find the distance from $\mathbf{y}$ to $W$.
5 In $\mathbb{R}^4$ let $W=\operatorname{span}\{(1,1,1,1),(1,-1,1,-1)\}$ and $\mathbf{y}=(4,0,2,-2)$. Find the closest point of $W$ to $\mathbf{y}$ and the distance from $\mathbf{y}$ to $W$.
7 Find the point of the plane $W=\operatorname{span}\{(1,1,0),(-1,1,1)\}$ closest to $\mathbf{y}=(2,3,5)$.
8A trap. Let $\mathbf{v}_1=(1,1,0)$, $\mathbf{v}_2=(1,0,1)$, $W=\operatorname{span}\{\mathbf{v}_1,\mathbf{v}_2\}$ and $\mathbf{y}=(1,2,3)$. (a) Compute $\dfrac{\mathbf{y}\cdot\mathbf{v}_1}{\mathbf{v}_1\cdot\mathbf{v}_1}\mathbf{v}_1+\dfrac{\mathbf{y}\cdot\mathbf{v}_2}{\mathbf{v}_2\cdot\mathbf{v}_2}\mathbf{v}_2$. (b) Find the true $\operatorname{proj}_W\mathbf{y}$ and show the answer to (a) is wrong.
2 $\mathbf{y}\cdot\mathbf{u}=15-4=11$, $\mathbf{u}\cdot\mathbf{u}=25$: $\hat{\mathbf{y}}=\tfrac{11}{25}(3,4)=\left(\tfrac{33}{25},\tfrac{44}{25}\right)$. $\mathbf{z}=\left(\tfrac{92}{25},-\tfrac{69}{25}\right)=\tfrac{23}{25}(4,-3)$, so the distance is $\tfrac{23}{25}\cdot5=\tfrac{23}{5}=4.6$.
5 The two vectors are orthogonal, each with $\mathbf{u}\cdot\mathbf{u}=4$. $\mathbf{y}\cdot\mathbf{u}_1=4$, $\mathbf{y}\cdot\mathbf{u}_2=4-0+2+2=8$: $\hat{\mathbf{y}}=1\mathbf{u}_1+2\mathbf{u}_2=(3,-1,3,-1)$. $\mathbf{z}=(1,1,-1,-1)$, distance $\|\mathbf{z}\|=2$.
6 If $\mathbf{y}\in W$, then $\mathbf{y}=\mathbf{y}+\mathbf{0}$ is a decomposition with $\mathbf{y}\in W$, $\mathbf{0}\in W^{\perp}$; by uniqueness (Theorem 8.9) $\hat{\mathbf{y}}=\mathbf{y}$. If $\mathbf{y}\in W^{\perp}$, then $\mathbf{y}=\mathbf{0}+\mathbf{y}$ is the decomposition, so $\hat{\mathbf{y}}=\mathbf{0}$ (also: every $\mathbf{y}\cdot\mathbf{u}_i=0$).
7 $(1,1,0)\cdot(-1,1,1)=0$. ✓ $\hat{\mathbf{y}}=\tfrac52(1,1,0)+\tfrac63(-1,1,1)=\left(\tfrac12,\tfrac92,2\right)$. (Check: $\mathbf{z}=\left(\tfrac32,-\tfrac32,3\right)$ is orthogonal to both; distance $\tfrac{3\sqrt6}{2}\approx3.674$.)
8 (a) $\tfrac32(1,1,0)+\tfrac42(1,0,1)=\left(\tfrac72,\tfrac32,2\right)$. (b) With $A=[\mathbf{v}_1\ \mathbf{v}_2]$, $A^{T}A=\begin{bmatrix}2&1\\1&2\end{bmatrix}$, $A^{T}\mathbf{y}=\begin{bmatrix}3\\4\end{bmatrix}$, so $\mathbf{c}=\tfrac13\begin{bmatrix}2&-1\\-1&2\end{bmatrix}\begin{bmatrix}3\\4\end{bmatrix}=\left(\tfrac23,\tfrac53\right)$ and $\operatorname{proj}_W\mathbf{y}=\tfrac23\mathbf{v}_1+\tfrac53\mathbf{v}_2=\left(\tfrac73,\tfrac23,\tfrac53\right)$. Now $\mathbf{y}-\hat{\mathbf{y}}=\tfrac43(-1,1,1)$ is orthogonal to $\mathbf{v}_1$ and $\mathbf{v}_2$ ✓, whereas $\mathbf{y}-\left(\tfrac72,\tfrac32,2\right)=\left(-\tfrac52,\tfrac12,1\right)$ has dot product $-2\ne0$ with $\mathbf{v}_1$ — so (a) is not the projection. The shortcut needs an orthogonal basis.
II
Projection matrices and best approximation.
1 Find the projection matrix $P$ onto the line spanned by $\mathbf{u}=\begin{bmatrix}1\\2\end{bmatrix}$ and verify that $P^2=P$.
2 Find the projection matrix onto the plane $x+y+z=0$ in $\mathbb{R}^3$. (Hint: first project onto the normal line.)
3 Let $A=\begin{bmatrix}1&0\\1&1\\1&2\end{bmatrix}$. Compute $P=A\left(A^{T}A\right)^{-1}A^{T}$ and $P\mathbf{b}$ for $\mathbf{b}=(1,0,4)$. Compare $P$ with Example 2 and explain.
4 For $P$ in part 3, show that $P\mathbf{b}=\mathbf{b}$ for $\mathbf{b}=(1,3,5)$ without multiplying out — then confirm by multiplying.
5 Show that if $P$ is the projection matrix onto $W$, then $I-P$ is the projection matrix onto $W^{\perp}$. Write down $I-P$ for part 2.
6Looking ahead to 8.4. If $P$ projects $\mathbb{R}^3$ onto a plane $W$, what are the eigenvalues of $P$, and what are the eigenspaces?
2 The normal is $\mathbf{n}=(1,1,1)$, and projecting onto the normal line is $\tfrac13\mathbf{n}\mathbf{n}^{T}$ (every entry $\tfrac13$). Subtracting it from $\mathbf{y}$ leaves the projection onto the plane: $$P=I-\frac13\begin{bmatrix}1&1&1\\1&1&1\\1&1&1\end{bmatrix}=\frac13\begin{bmatrix}2&-1&-1\\-1&2&-1\\-1&-1&2\end{bmatrix}.$$ (Theorem 8.11 with $A=\begin{bmatrix}1&0\\-1&1\\0&-1\end{bmatrix}$ gives the same matrix.)
3 $A^{T}A=\begin{bmatrix}3&3\\3&5\end{bmatrix}$, $\left(A^{T}A\right)^{-1}=\tfrac16\begin{bmatrix}5&-3\\-3&3\end{bmatrix}$, and $$P=\frac16\begin{bmatrix}5&2&-1\\2&2&2\\-1&2&5\end{bmatrix},\qquad P\mathbf{b}=\frac16\begin{bmatrix}1\\10\\19\end{bmatrix}=\left(\tfrac16,\tfrac53,\tfrac{19}{6}\right).$$ This is the same $P$ as in Example 2: both columns $(1,1,1)$ and $(0,1,2)$ satisfy $x-2y+z=0$, so $\operatorname{Col}A$ is the same plane. The projection matrix depends only on the subspace, not on the basis.
4 $(1,3,5)=1\cdot(1,1,1)+2\cdot(0,1,2)$ lies in $\operatorname{Col}A=W$, so its projection is itself (Ex. I.6). Multiplying: $\tfrac16(5+6-5,\ 2+6+10,\ -1+6+25)=(1,3,5)$. ✓
5 By Theorem 8.9, $\mathbf{y}=P\mathbf{y}+(\mathbf{y}-P\mathbf{y})$ with $\mathbf{y}-P\mathbf{y}=(I-P)\mathbf{y}\in W^{\perp}$. Since $(W^{\perp})^{\perp}=W$, this is also the decomposition of $\mathbf{y}$ relative to $W^{\perp}$, so $(I-P)\mathbf{y}=\operatorname{proj}_{W^{\perp}}\mathbf{y}$. For part 2, $I-P=\tfrac13\begin{bmatrix}1&1&1\\1&1&1\\1&1&1\end{bmatrix}$, the projection onto the normal line.
6 If $\mathbf{w}\in W$ then $P\mathbf{w}=\mathbf{w}$ ($\lambda=1$, eigenspace $W$, dimension 2); if $\mathbf{z}\in W^{\perp}$ then $P\mathbf{z}=\mathbf{0}$ ($\lambda=0$, eigenspace the normal line). So the eigenvalues are $1,1,0$. Algebraically: $P\mathbf{v}=\lambda\mathbf{v}$ and $P^2=P$ give $\lambda^2\mathbf{v}=\lambda\mathbf{v}$, so $\lambda\in\{0,1\}$.
III
Least-squares solutions. Find all least-squares solutions and, where asked, the least-squares error.
2 $A=\begin{bmatrix}2&1\\1&0\\0&1\end{bmatrix}$, $\mathbf{b}=\begin{bmatrix}1\\5\\3\end{bmatrix}$; also find the least-squares error.
3 Find the least-squares error for part 1 and check that the residual is orthogonal to $\operatorname{Col}A$.
4 $A=\begin{bmatrix}1&1\\1&-1\\1&1\\1&-1\end{bmatrix}$, $\mathbf{b}=\begin{bmatrix}3\\1\\5\\-1\end{bmatrix}$. The columns of $A$ are orthogonal — use this to find $\hat{\mathbf{x}}$ in one line.
5 Show that if $\mathbf{b}\in\operatorname{Col}A$ and $A$ has independent columns, the least-squares solution is the exact solution of $A\mathbf{x}=\mathbf{b}$. Illustrate with Example 3's $A$ and $\mathbf{b}=(5,3,1)$.
6 $A=\begin{bmatrix}1&2\\1&2\\1&2\end{bmatrix}$, $\mathbf{b}=\begin{bmatrix}1\\2\\6\end{bmatrix}$. Show there are infinitely many least-squares solutions and describe them all. Which theorem explains this?
7QR. Given $A=QR$ with $Q=\begin{bmatrix}\tfrac23&-\tfrac23\\\tfrac23&\tfrac13\\\tfrac13&\tfrac23\end{bmatrix}$ and $R=\begin{bmatrix}3&6\\0&3\end{bmatrix}$, use Theorem 8.14 to solve the least-squares problem for $\mathbf{b}=(-1,2,7)$, and give the error.
8 True or false? Give a reason. (a) Every system $A\mathbf{x}=\mathbf{b}$ has at least one least-squares solution. (b) $A^{T}A$ is always invertible. (c) The least-squares residual is orthogonal to $\operatorname{Col}A$.
3 $A\hat{\mathbf{x}}=\left(\tfrac{20}{7},-\tfrac27,\tfrac{31}{7}\right)$, so $\mathbf{r}=\left(-\tfrac67,\tfrac27,\tfrac47\right)$ and the error is $\tfrac17\sqrt{36+4+16}=\tfrac{\sqrt{56}}{7}=\tfrac{2\sqrt{14}}{7}\approx1.069$. Check: $\mathbf{r}\cdot(1,1,1)=0$ and $\mathbf{r}\cdot(1,-1,2)=\tfrac17(-6-2+8)=0$. ✓
4 With orthogonal columns $A^{T}A=\begin{bmatrix}4&0\\0&4\end{bmatrix}$ is diagonal, so $\hat x_j=\dfrac{\mathbf{a}_j\cdot\mathbf{b}}{\mathbf{a}_j\cdot\mathbf{a}_j}$: $\hat x_1=\tfrac84=2$, $\hat x_2=\tfrac{3-1+5+1}{4}=2$. $\hat{\mathbf{x}}=(2,2)$ (error $\|(-1,1,1,-1)\|=2$).
5 If $A\mathbf{x}_0=\mathbf{b}$ then $\|\mathbf{b}-A\mathbf{x}_0\|=0$, the smallest possible value, so $\mathbf{x}_0$ is a least-squares solution; by Theorem 8.13 it is the only one. Example: $\mathbf{b}=(5,3,1)$, $A^{T}\mathbf{b}=(8,11)$ (the same as in Example 3, because $(5,3,1)$ is exactly the projection found there), and $\begin{bmatrix}2&2\\2&5\end{bmatrix}\hat{\mathbf{x}}=\begin{bmatrix}8\\11\end{bmatrix}$ gives $\hat{\mathbf{x}}=(3,1)$, which solves the system exactly (error $0$).
6 $A^{T}A=\begin{bmatrix}3&6\\6&12\end{bmatrix}$, $A^{T}\mathbf{b}=\begin{bmatrix}9\\18\end{bmatrix}$; row reduction gives the single equation $\hat x_1+2\hat x_2=3$. So $\hat{\mathbf{x}}=\begin{bmatrix}3-2s\\s\end{bmatrix}$, $s\in\mathbb{R}$. Every one gives $A\hat{\mathbf{x}}=(3,3,3)$ (the mean of the entries of $\mathbf{b}$ in every slot). The columns of $A$ are dependent, so by Theorem 8.13 the solution cannot be unique.
7 $Q^{T}\mathbf{b}=\begin{bmatrix}-\tfrac23+\tfrac43+\tfrac73\\\tfrac23+\tfrac23+\tfrac{14}{3}\end{bmatrix}=\begin{bmatrix}3\\6\end{bmatrix}$. Back substitution in $R\hat{\mathbf{x}}=Q^{T}\mathbf{b}$: $3\hat x_2=6$, $\hat x_2=2$; $3\hat x_1+12=3$, $\hat x_1=-3$. With $A=QR=\begin{bmatrix}2&2\\2&5\\1&4\end{bmatrix}$, $A\hat{\mathbf{x}}=(-2,4,5)$, residual $(1,-2,2)$, error $3$.
8 (a) True — the normal equations are always consistent, since $\operatorname{proj}_{\operatorname{Col}A}\mathbf{b}$ always lies in $\operatorname{Col}A$. (b) False — only when the columns of $A$ are independent (part 6 is a counterexample). (c) True — that is what $A^{T}(\mathbf{b}-A\hat{\mathbf{x}})=\mathbf{0}$ says.
IV
Fitting models to data.
1 Find the least-squares line through $(0,1)$, $(1,3)$, $(2,4)$, $(3,4)$, and its $SSE$.
2 Find the least-squares line through $(1,2)$, $(2,3)$, $(3,5)$, $(4,6)$, and its $SSE$.
3Hooke's law. A spring of natural length $L_0$ stretches by $F/k$ under a load $F$, so $L=L_0+\tfrac1kF$. Sample measurements (illustrative):$F=0,2,4,6,8$ N, $L=12.0,\ 13.1,\ 13.9,\ 15.2,\ 15.8$ cm. Find the least-squares line and estimate $L_0$ and the spring constant $k$ (in N/m).
4A line through the origin. A second spring is tested by measuring the force needed for a given extension (illustrative data): $x=0.1,\ 0.2,\ 0.3,\ 0.4$ m, $F=2.1,\ 3.9,\ 6.2,\ 7.9$ N. Fit $F=kx$ by least squares (the design matrix has a single column).
5 Find the least-squares parabola $y=c_0+c_1x+c_2x^2$ through $(-1,3)$, $(0,1)$, $(1,0)$, $(2,1)$.
6Biology. A bacterial culture has $N=100,\ 270,\ 740,\ 2000$ cells at $t=0,1,2,3$ hours. Assuming $N=N_0e^{rt}$, take logarithms ($\ln N=\ln N_0+rt$) and fit a least-squares line to estimate $N_0$ and the growth rate $r$.
7 Use the line from Example 4 to predict the anomaly in 2030. Why should such a prediction be treated with care?
8Proof. Show that the least-squares line always passes through the centre of mass $(\bar x,\bar y)$ of the data, where $\bar x=\tfrac1m\sum x_i$ and $\bar y=\tfrac1m\sum y_i$. (Use the first normal equation.)
Check: plot the data and the fitted model, and compare its $SSE$ with yours
3 $\sum F=20$, $\sum F^2=120$, $\sum L=70$, $\sum FL=299.4$. $\begin{bmatrix}5&20\\20&120\end{bmatrix}\begin{bmatrix}\hat L_0\\\hat s\end{bmatrix}=\begin{bmatrix}70\\299.4\end{bmatrix}$, determinant $200$: $\hat s=\tfrac{5(299.4)-20(70)}{200}=0.485$ cm/N and $\hat L_0=\tfrac{120(70)-20(299.4)}{200}=12.06$ cm. So $L=12.06+0.485F$, natural length $\approx12.06$ cm, and $k=\tfrac1{0.485}\approx2.062$ N/cm $\approx206$ N/m. ($SSE=0.091$ cm².)
4 $A=\mathbf{x}=(0.1,0.2,0.3,0.4)$, so the normal equation is the single equation $(\mathbf{x}\cdot\mathbf{x})\hat k=\mathbf{x}\cdot\mathbf{F}$: $0.3\,\hat k=0.21+0.78+1.86+3.16=6.01$, $\hat k=\tfrac{6.01}{0.3}\approx20.03$ N/m. ($SSE\approx0.0697$ N².)
6 $\ln N\approx4.6052,\ 5.5984,\ 6.6067,\ 7.6009$. Fit $\ln N=a+rt$: $\begin{bmatrix}4&6\\6&14\end{bmatrix}\begin{bmatrix}a\\r\end{bmatrix}=\begin{bmatrix}24.4111\\41.6144\end{bmatrix}$ gives $r\approx0.9995$ per hour and $a\approx4.6035$, so $N_0=e^{a}\approx99.8$ cells. Model: $N\approx99.8\,e^{1.00t}$ — the culture grows by a factor of about $e\approx2.72$ every hour.
7 $t=5$: $y=0.16+0.2(5)=1.16$ °C. Extrapolating beyond the data assumes the trend stays linear; five points cannot tell us whether warming is speeding up (compare the parabola fit in the Research–Teaching Nexus below).
8 The first normal equation is $m\hat c_0+\hat c_1\sum x_i=\sum y_i$. Dividing by $m$: $\hat c_0+\hat c_1\bar x=\bar y$, which says the point $(\bar x,\bar y)$ lies on $y=\hat c_0+\hat c_1x$. (Equivalently, the residuals add to zero.)
🔬 Research–Teaching NexusResearch-tutored · optional, about 30 minutes
Research question. Is the sea level at a coastal city rising at a steady rate, or is the rise speeding up — and how much difference does that make to a forecast for 2050?
Why it matters. Ports, desalination plants and coastal roads from Muscat to Rotterdam to Jakarta are designed decades ahead. Engineers fit models to tide-gauge records by least squares; whether they use a line or a parabola changes the design height of a sea wall.
What to do.
Sample case: a tide gauge gives the mean sea level (mm, relative to its 2000 value) $0,\ 13,\ 27,\ 38,\ 55,\ 69$ in the years $2000,2004,\dots,2020$. Use $t$ = years since 2000.
Fit the least-squares line $y=c_0+c_1t$. What rate of rise (mm per year) does it give?
Fit the least-squares parabola $y=c_0+c_1t+c_2t^2$. The acceleration of sea level is $2c_2$ (mm/year²).
Compare the two $SSE$ values and both forecasts for 2050 ($t=50$).
Tools: Calculator or spreadsheet (use a matrix inverse or the built-in linear-regression tools, and check one result by hand with the normal equations).
Hand in: A half-page note with both fitted models, their $SSE$s, the two 2050 forecasts, and one sentence on which model you would trust and why.
Going further. Download a real tide-gauge record for a city of your choice (national hydrographic offices publish monthly means) and repeat the fit. Real records also contain a yearly cycle — how could you add $\sin(2\pi t)$ and $\cos(2\pi t)$ columns to the design matrix to model it? It is still a linear least-squares problem.
Instructor note: line $y\approx-0.762+3.443t$ (about $3.4$ mm/yr), $SSE\approx8.42$; parabola $y\approx0.250+3.063t+0.01897t^2$, acceleration $\approx0.038$ mm/yr², $SSE\approx4.98$. Forecasts for 2050: $\approx171$ mm (line) and $\approx201$ mm (parabola). Adding a parameter can never raise $SSE$, so students should argue from physics and from longer records, not from $SSE$ alone.
🧪 Self-check quiz
0/5 correct
Q1. If $\hat{\mathbf{y}}=\operatorname{proj}_W\mathbf{y}$, then the vector $\mathbf{y}-\hat{\mathbf{y}}$ is
Q2. The normal equations for the least-squares problem $A\mathbf{x}=\mathbf{b}$ are
Q3. The least-squares solution of $A\mathbf{x}=\mathbf{b}$ is unique for every $\mathbf{b}$ exactly when
Q4. Which pair of properties does every projection matrix $P=A\left(A^{T}A\right)^{-1}A^{T}$ have?
Q5. The least-squares line through $(0,0)$, $(1,1)$, $(2,1)$ is