Chapter 8 — Orthogonality and Least Squares
Right angles everywhere: perpendicular vectors turn messy, noisy and overdetermined problems into ones we can solve one dot product at a time.
Real data never fits perfectly. A physics student times a falling ball six times and gets six slightly different answers; an economist plots fuel prices against demand and sees a cloud of points that is almost a line; a GPS receiver hears from eight satellites when four would be enough to fix its position, and every signal carries a little noise. In each case the system of equations $A\mathbf{x}=\mathbf{b}$ has no exact solution — too many equations, all slightly inconsistent. Chapter 1 would simply say "inconsistent" and stop. This chapter finds the best possible answer instead.
The key idea is geometric and very old: the shortest path from a point to a line or a plane is the perpendicular one. Turn that into algebra and you get orthogonal vectors, orthogonal bases, projections and the method of least squares — the engine behind best-fit lines, satellite navigation, image and sound compression, and much of modern statistics and machine learning. Along the way we meet the Gram–Schmidt process, which manufactures perpendicular directions from any basis, and we finish with the Spectral Theorem: every symmetric matrix can be diagonalised using perpendicular eigenvectors.
- compute lengths, distances and orthogonality with the dot product and with other inner products (weighted, on polynomials, and — with calculus — on functions);
- find orthogonal complements, and use $(\operatorname{Row}A)^{\perp}=\operatorname{Nul}A$;
- find coordinates in an orthogonal basis with one dot product per vector, and recognise and use orthogonal matrices;
- turn any basis into an orthonormal one with the Gram–Schmidt process, and factorise $A=QR$;
- project a vector onto a subspace, and solve inconsistent systems in the least-squares sense — including fitting lines and curves to data;
- diagonalise a symmetric matrix with an orthogonal matrix, $A=PDP^{T}$.
Sections in this chapter
Inner Products and Orthogonal Sets
Length and angle beyond the dot product, orthogonal complements, orthogonal bases and orthogonal matrices.
Orthogonal Bases and the Gram–Schmidt Process
Subtract projections one vector at a time to build an orthonormal basis; the factorisation $A=QR$.
Orthogonal Projections and Least Squares
The closest point in a subspace, and the best approximate solution of an inconsistent system.
Symmetric Matrices and the Spectral Theorem
Real eigenvalues, perpendicular eigenvectors and orthogonal diagonalisation of $A=A^{T}$.