Chapter 10 · Inner Product Spaces and Quadratic Forms

10.1 Inner Product Spaces and Orthonormal Bases

An inner product adds lengths and angles to a vector space. With it come the Cauchy–Schwarz inequality, orthonormal bases built by Gram–Schmidt, orthogonal projections as best approximations, Bessel's inequality and Parseval's identity, least-squares solutions and the adjoint of an operator.

Chapters 8 and 9 studied vector spaces with nothing but addition and scaling. In such a space there is no way to say that two vectors are perpendicular, that one vector is longer than another, or that a point of a subspace is the one closest to a given vector. Yet these are exactly the questions that applications ask: which straight line fits a cloud of measurements best, which polynomial of degree $2$ approximates $|x|$ best on $[-1,1]$, how much of a signal is captured by its first ten Fourier coefficients. All of them are answered by one extra piece of structure, the inner product, which generalises the dot product of $\mathbb{R}^n$ to polynomials, matrices, functions and complex vectors.

This section builds the geometry that an inner product brings: the Cauchy–Schwarz and triangle inequalities, the parallelogram law, orthonormal bases and the Gram–Schmidt process (with the $QR$ factorisation it hides), orthogonal complements and projections, the best-approximation theorem, Bessel's inequality and Parseval's identity, the adjoint $T^*$ of an operator, and least squares. In CSIR NET this material appears directly — "which of these formulas is an inner product", "the projection of $v$ onto $W$", "the distance from $b$ to the column space", "an orthogonal polynomial for $\int_0^1pq$" — and it is the foundation for the spectral theorem (§10.2) and for quadratic forms (§10.3).

Throughout, $\mathbb{F}$ is $\mathbb{R}$ or $\mathbb{C}$, and $\bar a$ is the complex conjugate of $a$ (so $\bar a=a$ when $\mathbb{F}=\mathbb{R}$). For a matrix $A$, $A^*=\bar A^{T}$ is its conjugate transpose, as in §8.3; over $\mathbb{R}$, $A^*=A^T$. Vectors of $\mathbb{F}^n$ are columns.

Inner Products, Norms and the Cauchy–Schwarz Inequality

📘 Definition 10.1 — Inner product and inner product space

Let $V$ be a vector space over $\mathbb{F}$. An inner product on $V$ is a function $V\times V\to\mathbb{F}$, $(x,y)\mapsto\langle x,y\rangle$, such that for all $x,y,z\in V$ and $a,b\in\mathbb{F}$:

(IP1) Linear in the first slot$\langle ax+by,z\rangle=a\langle x,z\rangle+b\langle y,z\rangle$.
(IP2) Conjugate symmetry$\langle y,x\rangle=\overline{\langle x,y\rangle}$; over $\mathbb{R}$ this is symmetry $\langle y,x\rangle=\langle x,y\rangle$.
(IP3) Positive definiteness$\langle x,x\rangle\gt0$ for every $x\ne0$.

A vector space with an inner product is an inner product space (a Euclidean space if $\mathbb{F}=\mathbb{R}$, a unitary space if $\mathbb{F}=\mathbb{C}$). Its norm is $\|x\|=\sqrt{\langle x,x\rangle}$ and the distance from $x$ to $y$ is $\|x-y\|$. A vector with $\|x\|=1$ is a unit vector. (Many physics texts make the inner product linear in the second slot instead; every result below holds with the conjugates moved accordingly.)

The standard examples, which you should be able to verify on sight:

  1. $\mathbb{F}^n$: $\langle x,y\rangle=\sum_{k=1}^nx_k\overline{y_k}=y^*x$. Over $\mathbb{R}$ this is the dot product $x\cdot y=y^Tx$. The conjugate is essential: without it, $(1,i)$ would have "length squared" $1+i^2=0$.
  2. Weighted inner products on $\mathbb{F}^n$: $\langle x,y\rangle=y^*Gx$, where $G$ is a Hermitian matrix with $x^*Gx\gt0$ for all $x\ne0$ (a positive definite matrix, studied in §10.2 and §10.3). For instance $\langle x,y\rangle=\sum w_kx_k\overline{y_k}$ with all weights $w_k\gt0$.
  3. Matrices: on $M_{m\times n}(\mathbb{F})$, the Frobenius inner product $\langle A,B\rangle=\operatorname{tr}(B^*A)=\sum_{i,j}a_{ij}\overline{b_{ij}}$; then $\|A\|^2=\sum|a_{ij}|^2$.
  4. Continuous functions: on $C[a,b]$ (real or complex valued), $\langle f,g\rangle=\int_a^bf(t)\overline{g(t)}\,dt$. Positive definiteness uses continuity: if $f\ne0$ then $|f|^2\gt0$ on some subinterval, so $\int_a^b|f|^2\gt0$. Restricting to polynomials gives inner products on $P_n(\mathbb{F})$, the polynomials of degree at most $n$.
  5. Square-summable sequences: $\ell^2=\{(x_k):\sum|x_k|^2\lt\infty\}$ with $\langle x,y\rangle=\sum_kx_k\overline{y_k}$; the series converges absolutely because $2|x_k\overline{y_k}|\le|x_k|^2+|y_k|^2$. Its norm is the $\ell^2$ norm of §7.1.
📐 Theorem 10.1 — First consequences of the axioms
Let $V$ be an inner product space. (a) $\langle x,ay+bz\rangle=\bar a\langle x,y\rangle+\bar b\langle x,z\rangle$, and $\langle x,0\rangle=\langle0,x\rangle=0$. (b) If $\langle x,y\rangle=0$ for every $y\in V$, then $x=0$; consequently, if $\langle x,z\rangle=\langle y,z\rangle$ for every $z$, then $x=y$. (c) $\|ax\|=|a|\,\|x\|$. (d) Every inner product on $\mathbb{F}^n$ has the form $\langle x,y\rangle=y^*Gx$, where $G=[g_{ij}]$ with $g_{ij}=\langle e_j,e_i\rangle$ is Hermitian ($G^*=G$) and satisfies $x^*Gx\gt0$ for $x\ne0$; conversely every such $G$ defines an inner product.

Proof. (a) $\langle x,ay+bz\rangle=\overline{\langle ay+bz,x\rangle}$ $=\overline{a\langle y,x\rangle+b\langle z,x\rangle}$ $=\bar a\langle x,y\rangle+\bar b\langle x,z\rangle$ by (IP2) and (IP1). With $a=b=0$ we get $\langle x,0\rangle=0$, and $\langle0,x\rangle=\overline{\langle x,0\rangle}=0$. (b) Take $y=x$: $\langle x,x\rangle=0$ forces $x=0$ by (IP3). For the second claim apply this to $x-y$. (c) $\|ax\|^2=\langle ax,ax\rangle=a\bar a\langle x,x\rangle=|a|^2\|x\|^2$. (d) Write $x=\sum_jx_je_j$, $y=\sum_iy_ie_i$. By (IP1) and (a), $\langle x,y\rangle=\sum_{i,j}x_j\overline{y_i}\langle e_j,e_i\rangle=\sum_i\overline{y_i}\sum_jg_{ij}x_j=y^*Gx$. Conjugate symmetry gives $\overline{g_{ji}}=\overline{\langle e_i,e_j\rangle}=\langle e_j,e_i\rangle=g_{ij}$, so $G^*=G$, and $x^*Gx=\langle x,x\rangle\gt0$ for $x\ne0$. Conversely, if $G$ is Hermitian with $x^*Gx\gt0$ for $x\ne0$, then $(x,y)\mapsto y^*Gx$ is linear in $x$, satisfies $\overline{y^*Gx}=(y^*Gx)^*=x^*G^*y=x^*Gy$, and is positive definite. ∎

Part (d) turns the question "is this formula an inner product on $\mathbb{R}^n$?" into "is this symmetric matrix positive definite?". For $2\times2$ real matrices there is a quick test, proved in Example 1: $\begin{pmatrix}a & b\\ b & c\end{pmatrix}$ is positive definite if and only if $a\gt0$ and $ac-b^2\gt0$. The general test (leading principal minors) is proved in §10.3.

📐 Theorem 10.2 — The Cauchy–Schwarz inequality
For all vectors $x,y$ in an inner product space, $$|\langle x,y\rangle|\le\|x\|\,\|y\|,$$ with equality if and only if $x$ and $y$ are linearly dependent.

Proof. If $y=0$ both sides are $0$ and $x,y$ are dependent. Let $y\ne0$, put $t=\dfrac{\langle x,y\rangle}{\|y\|^2}$ and $z=x-ty$. Then $\langle z,y\rangle=\langle x,y\rangle-t\langle y,y\rangle=0$: we have split $x=ty+z$ into a multiple of $y$ and a vector orthogonal to $y$. Expanding with (IP1) and Theorem 10.1(a),

$$\|x\|^2=\langle ty+z,ty+z\rangle=|t|^2\|y\|^2+t\langle y,z\rangle+\bar t\langle z,y\rangle+\|z\|^2=|t|^2\|y\|^2+\|z\|^2,$$

because $\langle z,y\rangle=0$ and $\langle y,z\rangle=\overline{\langle z,y\rangle}=0$. Hence $\|x\|^2\ge|t|^2\|y\|^2=\dfrac{|\langle x,y\rangle|^2}{\|y\|^2}$, which is the inequality. Equality holds iff $\|z\|=0$, i.e. iff $x=ty$. If $x=ty$ the vectors are dependent; conversely, if $x,y$ are dependent and $y\ne0$, then $x=sy$ for some scalar $s$, and then $t=\langle sy,y\rangle/\|y\|^2=s$, so $z=0$. ∎

In concrete spaces Cauchy–Schwarz becomes three famous inequalities: $\big|\sum x_k\overline{y_k}\big|^2\le\sum|x_k|^2\sum|y_k|^2$ in $\mathbb{F}^n$; $\big|\int_a^bf\bar g\big|^2\le\int_a^b|f|^2\int_a^b|g|^2$ in $C[a,b]$; and $|\operatorname{tr}(B^*A)|^2\le\operatorname{tr}(A^*A)\operatorname{tr}(B^*B)$ for matrices. Over $\mathbb{R}$ it allows us to define the angle $\theta\in[0,\pi]$ between nonzero vectors by $\cos\theta=\dfrac{\langle x,y\rangle}{\|x\|\,\|y\|}$, a number in $[-1,1]$.

📐 Theorem 10.3 — The norm of an inner product: triangle inequality, parallelogram law, polarisation
In an inner product space, for all $x,y$:
  1. $\|x+y\|^2=\|x\|^2+2\operatorname{Re}\langle x,y\rangle+\|y\|^2$;
  2. triangle inequality $\|x+y\|\le\|x\|+\|y\|$, with equality iff one of $x,y$ is a non-negative real multiple of the other; hence $\|\cdot\|$ is a norm in the sense of §7.4;
  3. parallelogram law $\|x+y\|^2+\|x-y\|^2=2\|x\|^2+2\|y\|^2$;
  4. polarisation: over $\mathbb{R}$, $\langle x,y\rangle=\tfrac14\left(\|x+y\|^2-\|x-y\|^2\right)$; over $\mathbb{C}$, $\langle x,y\rangle=\tfrac14\sum_{k=0}^{3}i^k\|x+i^ky\|^2$.

Proof. (1) $\langle x+y,x+y\rangle=\|x\|^2+\langle x,y\rangle+\langle y,x\rangle+\|y\|^2$ and $\langle x,y\rangle+\overline{\langle x,y\rangle}=2\operatorname{Re}\langle x,y\rangle$. (2) Since $\operatorname{Re}\langle x,y\rangle\le|\langle x,y\rangle|\le\|x\|\|y\|$ by Cauchy–Schwarz, (1) gives $\|x+y\|^2\le(\|x\|+\|y\|)^2$. Equality forces $\operatorname{Re}\langle x,y\rangle=|\langle x,y\rangle|=\|x\|\|y\|$: by Theorem 10.2, $x,y$ are dependent, say (if $y\ne0$) $x=sy$, and then $\operatorname{Re}(s)\|y\|^2=|s|\|y\|^2$ forces $s$ real and $\ge0$. The other norm axioms are (IP3) and Theorem 10.1(c). (3) Add (1) to (1) with $y$ replaced by $-y$. (4) Subtracting the same two identities gives $\|x+y\|^2-\|x-y\|^2=4\operatorname{Re}\langle x,y\rangle$, which is the real formula. The complex formula is Exercise I.8. ∎

📐 Theorem 10.4 — Jordan–von Neumann: which norms come from inner products
A norm $\|\cdot\|$ on a real or complex vector space is induced by an inner product (that is, $\|x\|^2=\langle x,x\rangle$ for some inner product) if and only if it satisfies the parallelogram law. The inner product is then unique and is given by the polarisation formula.

Proof (sketch, real case, all key steps). Necessity is Theorem 10.3(3), and uniqueness follows from polarisation. Conversely, assume the parallelogram law and define $\langle x,y\rangle=\tfrac14(\|x+y\|^2-\|x-y\|^2)$. Then $\langle x,y\rangle=\langle y,x\rangle$ and $\langle x,x\rangle=\tfrac14\|2x\|^2=\|x\|^2$, so only linearity in $x$ needs work. Fix $y$ and write $f(x)=\langle x,y\rangle$. The parallelogram law for the pairs $(x+y,x')$ and $(x-y,x')$ reads

$$\|x+x'+y\|^2+\|x-x'+y\|^2=2\|x+y\|^2+2\|x'\|^2,$$ $$\|x+x'-y\|^2+\|x-x'-y\|^2=2\|x-y\|^2+2\|x'\|^2.$$

Subtracting and dividing by $4$ gives $f(x+x')+f(x-x')=2f(x)$ for all $x,x'$. With $x'=x$ (and $f(0)=0$) this says $f(2x)=2f(x)$; substituting $x=\tfrac{a+b}{2}$, $x'=\tfrac{a-b}{2}$ gives $f(a)+f(b)=2f\!\left(\tfrac{a+b}2\right)=f(a+b)$. So $f$ is additive, hence $f(qx)=qf(x)$ for every rational $q$ (first for $q\in\mathbb{N}$ by induction, then for $-1$ and $1/n$). Finally $f$ is continuous, since $\big|\|u\|-\|v\|\big|\le\|u-v\|$, and $\mathbb{Q}$ is dense in $\mathbb{R}$, so $f(tx)=tf(x)$ for every real $t$. In the complex case one defines $\langle x,y\rangle$ by the complex polarisation formula and checks in the same way that $\langle ix,y\rangle=i\langle x,y\rangle$. ∎

Theorem 10.4 settles at once which of the norms of §7.4 are "Euclidean". In $\mathbb{R}^2$ with $x=e_1$, $y=e_2$: for $\|\cdot\|_1$, $\|x+y\|^2+\|x-y\|^2=4+4=8\ne4=2+2$; for $\|\cdot\|_\infty$, $1+1=2\ne4$. So neither comes from an inner product, while $\|\cdot\|_2$ does. The same test shows that the sup norm on $C[0,1]$ is not induced by an inner product (Exercise I.7).

✏️ Example 1 — Is it an inner product?
Decide, with proof, which of the following are inner products. (a) $\langle x,y\rangle=x_1y_1-x_1y_2-x_2y_1+3x_2y_2$ on $\mathbb{R}^2$. (b) $\langle x,y\rangle=x_1y_1-2x_1y_2-2x_2y_1+3x_2y_2$ on $\mathbb{R}^2$. (c) $\langle z,w\rangle=z_1w_1+z_2w_2$ on $\mathbb{C}^2$. (d) $\langle p,q\rangle=p(0)q(0)+p(1)q(1)$ on $P_1(\mathbb{R})$, and on $P_2(\mathbb{R})$. (e) $\langle A,B\rangle=\operatorname{tr}(B^*A)$ on $M_n(\mathbb{C})$. Then prove the $2\times2$ test quoted above.
(a) Yes The formula is $y^TGx$ with $G=\begin{pmatrix}1 & -1\\ -1 & 3\end{pmatrix}$, symmetric, so (IP1) and (IP2) hold. Completing the square, $\langle x,x\rangle=x_1^2-2x_1x_2+3x_2^2=(x_1-x_2)^2+2x_2^2$, which is $\gt0$ unless $x_2=0$ and $x_1=x_2$, i.e. unless $x=0$.
(b) No Now $\langle x,x\rangle=x_1^2-4x_1x_2+3x_2^2=(x_1-x_2)(x_1-3x_2)$; at $x=(2,1)$ it equals $(1)(-1)=-1\lt0$. Symmetry and linearity hold, but positive definiteness fails.
(c) No It is bilinear and symmetric, but $\langle(1,i),(1,i)\rangle=1+i^2=0$ although $(1,i)\ne0$. Worse, $\langle iz,iz\rangle=-\langle z,z\rangle$, so no vector can have positive "length squared" together with its multiple by $i$. On $\mathbb{C}^n$ the conjugate on the second factor cannot be dropped.
(d) The formula is symmetric and bilinear, and $\langle p,p\rangle=p(0)^2+p(1)^2\ge0$. On $P_1$: if $\langle p,p\rangle=0$ then $p(0)=p(1)=0$, and a polynomial of degree $\le1$ with two roots is $0$ — an inner product. On $P_2$: $p=x(x-1)\ne0$ has $\langle p,p\rangle=0$ — not an inner product. The number of evaluation points must exceed the degree.
(e) Yes $\operatorname{tr}(B^*A)=\sum_{i,j}\overline{b_{ij}}a_{ij}$ is the standard inner product of $\mathbb{C}^{n^2}$ applied to the entries, so all three axioms hold; $\langle A,A\rangle=\sum|a_{ij}|^2$.
The $2\times2$ test For $G=\begin{pmatrix}a & b\\ b & c\end{pmatrix}$, $q(x)=x^TGx=ax_1^2+2bx_1x_2+cx_2^2$. If $q\gt0$ off the origin then $a=q(e_1)\gt0$ and $q(-b,a)=a(ac-b^2)\gt0$, so $ac-b^2\gt0$. Conversely, if $a\gt0$ and $ac-b^2\gt0$, then $q(x)=a\left(x_1+\tfrac bax_2\right)^2+\tfrac{ac-b^2}{a}x_2^2$, a sum of two non-negative terms that vanish together only at $x=0$. In (a), $a=1$, $ac-b^2=2$; in (b), $ac-b^2=-1$. ✓
✏️ Example 2 — Cauchy–Schwarz as an optimisation tool
(a) Find the maximum and minimum of $2x-y+2z$ on the sphere $x^2+y^2+z^2=1$, and the points where they are attained. (b) Prove that $\left(\int_0^1f\right)^2\le\int_0^1f^2$ for every $f\in C[0,1]$, with equality iff $f$ is constant. (c) Find the minimum of $x^2+y^2+z^2$ subject to $x+2y+3z=7$.
(a) With $a=(2,-1,2)$ and $v=(x,y,z)$, $2x-y+2z=\langle v,a\rangle$ and $|\langle v,a\rangle|\le\|v\|\,\|a\|=1\cdot3$. Equality needs $v$ parallel to $a$: $v=\pm\tfrac13(2,-1,2)$, giving $\pm3$. Maximum $3$ at $\tfrac13(2,-1,2)$, minimum $-3$ at $-\tfrac13(2,-1,2)$.
(b) Apply Cauchy–Schwarz in $C[0,1]$ to $f$ and the constant $1$: $\left|\int_0^1f\cdot1\right|^2\le\int_0^1f^2\int_0^11^2=\int_0^1f^2$. Equality holds iff $f$ and $1$ are dependent, i.e. iff $f$ is constant.
(c) $7=\langle v,(1,2,3)\rangle\le\|v\|\sqrt{14}$, so $\|v\|^2\ge\tfrac{49}{14}=\tfrac72$, with equality iff $v=t(1,2,3)$; the constraint gives $14t=7$, $t=\tfrac12$. Minimum $\tfrac72$ at $\left(\tfrac12,1,\tfrac32\right)$.
Geometry In (c) we computed the squared distance from the origin to the plane $x+2y+3z=7$: it is $\left(7/\sqrt{14}\right)^2$, and the closest point is the foot of the perpendicular along the normal $(1,2,3)$. The best-approximation theorem (Theorem 10.11 below) explains this in general.

Orthonormal Bases and the Gram–Schmidt Process

📘 Definition 10.2 — Orthogonal and orthonormal sets
Vectors $x,y$ are orthogonal, $x\perp y$, if $\langle x,y\rangle=0$ (equivalently $\langle y,x\rangle=0$). A family $(v_i)_{i\in I}$ is orthogonal if $v_i\perp v_j$ for all $i\ne j$, and orthonormal if moreover $\|v_i\|=1$ for every $i$, i.e. $\langle v_i,v_j\rangle=\delta_{ij}$. An orthonormal basis of a finite-dimensional inner product space is a basis that is an orthonormal family.
📐 Theorem 10.5 — Pythagoras; coordinates in an orthonormal basis
  1. Pythagoras. If $v_1,\dots,v_k$ are pairwise orthogonal, then $\|v_1+\cdots+v_k\|^2=\|v_1\|^2+\cdots+\|v_k\|^2$.
  2. An orthogonal family of nonzero vectors is linearly independent.
  3. If $e_1,\dots,e_k$ are orthonormal and $x=\sum_ja_je_j$, then $a_j=\langle x,e_j\rangle$ and $\|x\|^2=\sum_j|a_j|^2$.
  4. If $e_1,\dots,e_n$ is an orthonormal basis of $V$, then for all $x,y\in V$: $$x=\sum_{j=1}^n\langle x,e_j\rangle e_j,\qquad \langle x,y\rangle=\sum_{j=1}^n\langle x,e_j\rangle\overline{\langle y,e_j\rangle}.$$

Proof. (1) Expanding $\langle\sum v_i,\sum v_j\rangle$ gives $\sum_{i,j}\langle v_i,v_j\rangle$, and only the terms $i=j$ survive. (2) If $\sum_ja_jv_j=0$, take the inner product with $v_i$: $0=\sum_ja_j\langle v_j,v_i\rangle=a_i\|v_i\|^2$, and $\|v_i\|\ne0$, so $a_i=0$. (3) $\langle x,e_i\rangle=\sum_ja_j\langle e_j,e_i\rangle=a_i$; the norm formula is (1) with $v_j=a_je_j$ and $\|a_je_j\|=|a_j|$. (4) The first formula is (3), applied to the expansion of $x$ in the basis. For the second, $\langle x,y\rangle=\big\langle\sum_ja_je_j,\sum_kb_ke_k\big\rangle$ $=\sum_{j,k}a_j\overline{b_k}\delta_{jk}=\sum_ja_j\overline{b_j}$ with $a_j=\langle x,e_j\rangle$, $b_j=\langle y,e_j\rangle$. ∎

Part (4) is the reason orthonormal bases are so convenient: coordinates are computed by inner products instead of by solving a linear system, and in those coordinates every inner product looks like the standard one on $\mathbb{F}^n$. Over $\mathbb{R}$ the converse of Pythagoras holds for two vectors ($\|x+y\|^2=\|x\|^2+\|y\|^2\iff\langle x,y\rangle=0$, by Theorem 10.3(1)); over $\mathbb{C}$ it only gives $\operatorname{Re}\langle x,y\rangle=0$.

📐 Theorem 10.6 — The Gram–Schmidt process

Let $v_1,v_2,\dots$ (finitely many or a sequence) be linearly independent vectors in an inner product space. Define recursively

$$u_1=v_1,\qquad u_k=v_k-\sum_{j=1}^{k-1}\langle v_k,e_j\rangle e_j,\qquad e_k=\frac{u_k}{\|u_k\|}.$$

Then every $u_k\ne0$, the family $(e_k)$ is orthonormal, and for every $k$: $\operatorname{span}\{e_1,\dots,e_k\}=\operatorname{span}\{v_1,\dots,v_k\}$ and $\langle v_k,e_k\rangle=\|u_k\|\gt0$. Moreover $(e_k)$ is the only orthonormal family with these two properties.

Proof. Induction on $k$. For $k=1$, $u_1=v_1\ne0$ (an independent family contains no zero vector) and $e_1=v_1/\|v_1\|$ has the required span; $\langle v_1,e_1\rangle=\|v_1\|\gt0$. Suppose $e_1,\dots,e_{k-1}$ are orthonormal with $\operatorname{span}\{e_1,\dots,e_{k-1}\}$ $=\operatorname{span}\{v_1,\dots,v_{k-1}\}=:W_{k-1}$.

$u_k\ne0$: the sum $\sum_{j\lt k}\langle v_k,e_j\rangle e_j$ lies in $W_{k-1}$; if $u_k=0$ then $v_k\in W_{k-1}=\operatorname{span}\{v_1,\dots,v_{k-1}\}$, contradicting independence. Orthogonality: for $i\lt k$, $\langle u_k,e_i\rangle=\langle v_k,e_i\rangle-\sum_{j\lt k}\langle v_k,e_j\rangle\langle e_j,e_i\rangle$ $=\langle v_k,e_i\rangle-\langle v_k,e_i\rangle=0$; so $e_k\perp e_1,\dots,e_{k-1}$, and $\|e_k\|=1$. Span: $e_k$ is a combination of $v_k$ and vectors of $W_{k-1}$, so $\operatorname{span}\{e_1,\dots,e_k\}\subseteq\operatorname{span}\{v_1,\dots,v_k\}$; conversely $v_k=u_k+\sum_{j\lt k}\langle v_k,e_j\rangle e_j$ $=\|u_k\|e_k+\sum_{j\lt k}\langle v_k,e_j\rangle e_j$. Both spans have dimension $k$ (by Theorem 10.5(2) the $e$'s are independent), so they are equal. Positivity: taking the inner product of the last display with $e_k$ gives $\langle v_k,e_k\rangle=\|u_k\|\gt0$.

Uniqueness: let $(f_k)$ be another orthonormal family with the two properties. Inductively assume $f_j=e_j$ for $j\lt k$. Then $f_k\in\operatorname{span}\{v_1,\dots,v_k\}=\operatorname{span}\{e_1,\dots,e_k\}$ and $f_k\perp e_1,\dots,e_{k-1}$, so by Theorem 10.5(3) $f_k=\langle f_k,e_k\rangle e_k=ce_k$ with $|c|=\|f_k\|=1$. Finally $0\lt\langle v_k,f_k\rangle=\bar c\langle v_k,e_k\rangle$ with $\langle v_k,e_k\rangle\gt0$ forces $\bar c\gt0$, hence $c=1$. ∎

If the input vectors are dependent, the same recursion produces $u_k=0$ exactly when $v_k\in\operatorname{span}\{v_1,\dots,v_{k-1}\}$; discarding those $v_k$ and continuing yields an orthonormal basis of $\operatorname{span}\{v_1,v_2,\dots\}$. Gram–Schmidt is therefore also a test for linear independence. In hand computations it is often easier to orthogonalise first and normalise at the end, using $u_k=v_k-\sum_{j\lt k}\dfrac{\langle v_k,u_j\rangle}{\langle u_j,u_j\rangle}u_j$ (the same vectors $u_k$, without square roots), and any $u_k$ may be replaced by a convenient nonzero multiple.

📐 Theorem 10.7 — Existence and extension of orthonormal bases
Every finite-dimensional inner product space $V\ne\{0\}$ has an orthonormal basis, and every orthonormal family in $V$ can be extended to an orthonormal basis. If $e_1,\dots,e_n$ is an orthonormal basis, the coordinate map $x\mapsto(\langle x,e_1\rangle,\dots,\langle x,e_n\rangle)$ is a linear bijection $V\to\mathbb{F}^n$ that carries $\langle\cdot,\cdot\rangle$ to the standard inner product of $\mathbb{F}^n$.

Proof. Apply Gram–Schmidt to any basis. For the extension, an orthonormal family $e_1,\dots,e_k$ is independent (Theorem 10.5(2)); extend it to a basis $e_1,\dots,e_k,v_{k+1},\dots,v_n$ (§8.1) and apply Gram–Schmidt. The first $k$ vectors are unchanged, because $u_j=e_j-\sum_{i\lt j}\langle e_j,e_i\rangle e_i=e_j$ has norm $1$. The last statement is Theorem 10.5(4). ∎

📐 Theorem 10.8 — The $QR$ factorisation
Let $A\in M_{m\times n}(\mathbb{F})$ have linearly independent columns ($\operatorname{rank}A=n$). Then $A=QR$, where $Q\in M_{m\times n}(\mathbb{F})$ has orthonormal columns ($Q^*Q=I_n$) and $R\in M_n(\mathbb{F})$ is upper triangular with positive diagonal entries. The factorisation is unique; the columns of $Q$ are the Gram–Schmidt vectors of the columns of $A$, and $r_{jk}=\langle a_k,q_j\rangle$ for $j\le k$, $r_{kk}=\|u_k\|$. If $A$ is square, $Q$ is unitary (orthogonal when $\mathbb{F}=\mathbb{R}$).

Proof. Let $a_1,\dots,a_n$ be the columns and $q_1,\dots,q_n$ their Gram–Schmidt vectors. From the proof of Theorem 10.6, $a_k=\|u_k\|q_k+\sum_{j\lt k}\langle a_k,q_j\rangle q_j$: column $k$ of $A$ is $Q$ times the column vector $(r_{1k},\dots,r_{kk},0,\dots,0)^T$. That is $A=QR$ with $R$ as stated. The $(i,j)$ entry of $Q^*Q$ is $q_i^*q_j=\langle q_j,q_i\rangle=\delta_{ij}$. For uniqueness, if $A=Q'R'$ is another such factorisation, then column $k$ of $A$ is $\sum_{j\le k}r'_{jk}q'_j$, so $\operatorname{span}\{a_1,\dots,a_k\}=\operatorname{span}\{q'_1,\dots,q'_k\}$ ($R'$ is invertible) and $\langle a_k,q'_k\rangle=r'_{kk}\gt0$; the uniqueness part of Theorem 10.6 gives $q'_k=q_k$, and then $R'=Q^*A=R$. A square matrix with $Q^*Q=I$ is unitary (§8.3). ∎

📘 Definition 10.3 — Gram matrix
The Gram matrix of vectors $v_1,\dots,v_k$ is the $k\times k$ matrix $G=G(v_1,\dots,v_k)$ with entries $g_{ij}=\langle v_j,v_i\rangle$. Its determinant is the Gram determinant. (By Theorem 10.1(d), an inner product on $\mathbb{F}^n$ is $y^*Gx$ with $G$ the Gram matrix of the standard basis.)
📐 Theorem 10.9 — Gram determinants measure independence and volume
Let $v_1,\dots,v_k$ be vectors in an inner product space and $G$ their Gram matrix. (a) $G$ is Hermitian and $c^*Gc=\big\|\sum_jc_jv_j\big\|^2\ge0$ for every $c\in\mathbb{F}^k$. (b) $\det G\ge0$, and $\det G=0$ if and only if $v_1,\dots,v_k$ are linearly dependent. (c) If they are independent and $u_1,\dots,u_k$ are their Gram–Schmidt vectors, $\det G=\|u_1\|^2\|u_2\|^2\cdots\|u_k\|^2$. In $\mathbb{R}^n$, $\sqrt{\det G}$ is the $k$-dimensional volume of the parallelotope spanned by the $v_j$; for $k=n$ and $A=[v_1\ \cdots\ v_n]$, $\det G=\det(A^TA)=(\det A)^2$.

Proof. (a) $\overline{g_{ji}}=\overline{\langle v_i,v_j\rangle}=\langle v_j,v_i\rangle=g_{ij}$. Also $c^*Gc=\sum_{i,j}\overline{c_i}c_j\langle v_j,v_i\rangle=\big\langle\sum_jc_jv_j,\sum_ic_iv_i\big\rangle$. (b), (c) If the $v_j$ are dependent, choose $c\ne0$ with $\sum_jc_jv_j=0$; then $(Gc)_i=\sum_j\langle v_j,v_i\rangle c_j=\big\langle\sum_jc_jv_j,v_i\big\rangle=0$, so $Gc=0$ and $\det G=0$. If they are independent, let $W=\operatorname{span}\{v_j\}$, take the orthonormal basis $q_1,\dots,q_k$ of $W$ produced by Gram–Schmidt, and let $A\in M_k(\mathbb{F})$ be the matrix whose column $j$ is the coordinate vector of $v_j$. By Theorem 10.5(4), $g_{ij}=\langle v_j,v_i\rangle=a_i^*a_j$, i.e. $G=A^*A$. By Theorem 10.8 in coordinates, $A=QR$ with $Q$ unitary and $R$ upper triangular with diagonal $\|u_1\|$, $\dots$, $\|u_k\|$. Then $G=R^*Q^*QR=R^*R$ and $\det G=|\det R|^2=\prod\|u_j\|^2\gt0$. The volume statement is the "base times height" description: $\|u_k\|$ is the height of $v_k$ above the span of the previous vectors. ∎

✏️ Example 3 — Gram–Schmidt and $QR$ in $\mathbb{R}^3$
Apply Gram–Schmidt to $v_1=(1,1,0)$, $v_2=(1,0,1)$, $v_3=(0,1,1)$ in $\mathbb{R}^3$ with the dot product, and write the $QR$ factorisation of $A=[v_1\ v_2\ v_3]$. Check the Gram determinant.
$e_1$ $\|v_1\|=\sqrt2$, so $e_1=\tfrac1{\sqrt2}(1,1,0)$.
$e_2$ $\langle v_2,e_1\rangle=\tfrac1{\sqrt2}$, so $u_2=(1,0,1)-\tfrac12(1,1,0)=\left(\tfrac12,-\tfrac12,1\right)$, $\|u_2\|=\sqrt{\tfrac32}=\tfrac{\sqrt6}{2}$, and $e_2=\tfrac1{\sqrt6}(1,-1,2)$. Check: $\langle e_1,e_2\rangle\propto1-1+0=0$ ✓.
$e_3$ $\langle v_3,e_1\rangle=\tfrac1{\sqrt2}$ and $\langle v_3,e_2\rangle=\tfrac{-1+2}{\sqrt6}=\tfrac1{\sqrt6}$, so $$u_3=(0,1,1)-\tfrac12(1,1,0)-\tfrac16(1,-1,2)=\left(-\tfrac23,\tfrac23,\tfrac23\right),$$ $\|u_3\|=\tfrac{2}{\sqrt3}$ and $e_3=\tfrac1{\sqrt3}(-1,1,1)$. Check: $e_3\perp e_1$ ($-1+1+0=0$), $e_3\perp e_2$ ($-1-1+2=0$) ✓.
$QR$ $Q=[e_1\ e_2\ e_3]$ and $$R=\begin{pmatrix}\sqrt2 & \tfrac1{\sqrt2} & \tfrac1{\sqrt2}\\ 0 & \tfrac{\sqrt6}{2} & \tfrac1{\sqrt6}\\ 0 & 0 & \tfrac{2}{\sqrt3}\end{pmatrix},$$ the entries above the diagonal being $\langle v_k,e_j\rangle$ and the diagonal $\|u_k\|$.
Check $G=A^TA=\begin{pmatrix}2 & 1 & 1\\ 1 & 2 & 1\\ 1 & 1 & 2\end{pmatrix}$ has determinant $4$, and $\|u_1\|^2\|u_2\|^2\|u_3\|^2=2\cdot\tfrac32\cdot\tfrac43=4$ ✓; also $\det A=-2$ and $(\det A)^2=4$ ✓.
Interactive 1 — Gram–Schmidt, One Projection at a Time

Three vectors $w_1,w_2,w_3$ in $\mathbb{R}^3$ start as $v_1,v_2,v_3$ (faint dashed copies stay behind). Each step either normalises a vector or subtracts its projection (amber) onto the span of the vectors already finished; the purple patch is $\operatorname{span}\{e_1,e_2\}$. Watch the inner-product readouts go to $0$ and the norms to $1$. The scene turns slowly so that you can see right angles in depth; the last data set is linearly dependent.

step—
⟨w₁,w₂⟩ · ⟨w₁,w₃⟩ · ⟨w₂,w₃⟩—
‖w₁‖ · ‖w₂‖ · ‖w₃‖—
diag R = ‖u₁‖, ‖u₂‖, ‖u₃‖—

Orthogonal Complements, Projections, Bessel and Parseval

📘 Definition 10.4 — Orthogonal complement
For a subset $S$ of an inner product space $V$, the orthogonal complement is $S^\perp=\{x\in V:\langle x,s\rangle=0\text{ for all }s\in S\}$. It is always a subspace (it is an intersection of kernels of the linear functionals $x\mapsto\langle x,s\rangle$), $S^\perp=(\operatorname{span}S)^\perp$, $\{0\}^\perp=V$, $V^\perp=\{0\}$, and $S\subseteq T\Rightarrow T^\perp\subseteq S^\perp$. For a subspace $W$, $W\cap W^\perp=\{0\}$, since $w\perp w$ forces $w=0$.
📐 Theorem 10.10 — The orthogonal decomposition $V=W\oplus W^\perp$
Let $W$ be a finite-dimensional subspace of an inner product space $V$ ($V$ itself may be infinite-dimensional), with orthonormal basis $e_1,\dots,e_k$.
  1. Every $x\in V$ can be written uniquely as $x=w+w'$ with $w\in W$ and $w'\in W^\perp$; namely $w=P_Wx:=\sum_{j=1}^k\langle x,e_j\rangle e_j$. Thus $V=W\oplus W^\perp$.
  2. $P_W$ is linear, $P_W^2=P_W$, $\operatorname{im}P_W=W$, $\ker P_W=W^\perp$, and $\|x\|^2=\|P_Wx\|^2+\|x-P_Wx\|^2$; in particular $\|P_Wx\|\le\|x\|$.
  3. If $\dim V=n$ is finite, then $\dim W^\perp=n-\dim W$, $(W^\perp)^\perp=W$, and $P_{W^\perp}=I-P_W$.

Proof. (1) Put $w=\sum_j\langle x,e_j\rangle e_j\in W$ and $w'=x-w$. For each $i$, $\langle w',e_i\rangle=\langle x,e_i\rangle-\sum_j\langle x,e_j\rangle\langle e_j,e_i\rangle=0$, so $w'$ is orthogonal to every $e_i$, hence to $W$. If also $x=w_1+w_1'$, then $w-w_1=w_1'-w'\in W\cap W^\perp=\{0\}$. (2) Linearity is visible from the formula. $P_Wx=x$ for $x\in W$ (by uniqueness, $x=x+0$), so $P_W^2=P_W$ and $\operatorname{im}P_W=W$; $P_Wx=0$ iff $x=w'\in W^\perp$. The norm identity is Pythagoras for $x=P_Wx+(x-P_Wx)$. (3) From $V=W\oplus W^\perp$, $\dim W^\perp=n-\dim W$. Every $w\in W$ is orthogonal to every vector of $W^\perp$, so $W\subseteq(W^\perp)^\perp$, and both have dimension $n-(n-\dim W)=\dim W$; hence they are equal. Finally $x=P_Wx+(I-P_W)x$ is the decomposition of $x$ relative to $W^\perp\oplus(W^\perp)^\perp$, so $(I-P_W)x=P_{W^\perp}x$. ∎

📘 Definition 10.5 — Orthogonal projection
The map $P_W$ of Theorem 10.10 is the orthogonal projection of $V$ onto $W$, and $P_Wx$ is the orthogonal projection of $x$ onto $W$. For a line $W=\operatorname{span}\{v\}$, $P_Wx=\dfrac{\langle x,v\rangle}{\langle v,v\rangle}\,v$, and in $\mathbb{F}^n$ its matrix is $\dfrac{vv^*}{v^*v}$.
📐 Theorem 10.11 — Best approximation
Let $W$ be a finite-dimensional subspace of an inner product space $V$ and $x\in V$. Then $P_Wx$ is the unique point of $W$ nearest to $x$: $$\|x-P_Wx\|\lt\|x-w\|\quad\text{for every }w\in W,\ w\ne P_Wx.$$ A vector $w_0\in W$ is the nearest point iff $x-w_0\perp W$. If $v_1,\dots,v_k$ is any basis of $W$, then $P_Wx=\sum_jc_jv_j$ where $c$ solves the normal equations $\sum_{j}\langle v_j,v_i\rangle c_j=\langle x,v_i\rangle$ ($i=1,\dots,k$), i.e. $Gc=\big(\langle x,v_i\rangle\big)_i$ with $G$ the Gram matrix. With an orthonormal basis $e_1,\dots,e_k$, $\operatorname{dist}(x,W)^2=\|x\|^2-\sum_j|\langle x,e_j\rangle|^2$.

Proof. For $w\in W$, $x-w=(x-P_Wx)+(P_Wx-w)$ with $x-P_Wx\in W^\perp$ and $P_Wx-w\in W$. By Pythagoras, $\|x-w\|^2=\|x-P_Wx\|^2+\|P_Wx-w\|^2$, which exceeds $\|x-P_Wx\|^2$ unless $w=P_Wx$. If $x-w_0\perp W$, then $x=w_0+(x-w_0)$ is the decomposition of Theorem 10.10(1), so $w_0=P_Wx$; conversely $x-P_Wx\in W^\perp$. The condition $x-\sum_jc_jv_j\perp v_i$ for each $i$ is exactly $\langle x,v_i\rangle=\sum_jc_j\langle v_j,v_i\rangle$, and $G$ is invertible by Theorem 10.9, so the normal equations have exactly one solution. The distance formula is Theorem 10.10(2) together with $\|P_Wx\|^2=\sum_j|\langle x,e_j\rangle|^2$. ∎

🧭 Problem-solving strategy — computing $P_Wx$ and $\operatorname{dist}(x,W)$
  1. Orthogonal basis available? Use $P_Wx=\sum_j\dfrac{\langle x,u_j\rangle}{\langle u_j,u_j\rangle}u_j$ — no normalising needed. Otherwise run Gram–Schmidt, or…
  2. …solve the normal equations $Gc=(\langle x,v_i\rangle)$ with the Gram matrix of any basis. For $W=C(A)\subseteq\mathbb{F}^m$ this is $A^*Ac=A^*x$ (Theorem 10.18), and the projection matrix is $A(A^*A)^{-1}A^*$.
  3. Is $W^\perp$ smaller? For a hyperplane $W=\{x:\langle x,n\rangle=0\}$ use $P_Wx=x-\dfrac{\langle x,n\rangle}{\langle n,n\rangle}n$ and $\operatorname{dist}(x,W)=\dfrac{|\langle x,n\rangle|}{\|n\|}$. In general $P_Wx=x-P_{W^\perp}x$.
  4. Check that $x-P_Wx$ is orthogonal to each basis vector of $W$, and that $\|x\|^2=\|P_Wx\|^2+\operatorname{dist}^2$.
✏️ Example 4 — Legendre polynomials and the best quadratic approximation to $|x|$
On $P_3(\mathbb{R})$ use $\langle p,q\rangle=\int_{-1}^{1}p(x)q(x)\,dx$. (a) Apply Gram–Schmidt (without normalising) to $1,x,x^2,x^3$. (b) Find the polynomial $p$ of degree at most $2$ that minimises $\int_{-1}^1\big(|x|-p(x)\big)^2dx$, and the minimum value.
Parity $\int_{-1}^1x^m\,dx=0$ for odd $m$ and $\tfrac{2}{m+1}$ for even $m$. So even and odd polynomials are automatically orthogonal, which removes most of the work.
(a) $u_0=1$, $\|u_0\|^2=2$. $u_1=x-\tfrac{\langle x,1\rangle}{2}\cdot1=x$, $\|u_1\|^2=\tfrac23$. $u_2=x^2-\tfrac{\langle x^2,1\rangle}{2}=x^2-\tfrac13$ (it is already orthogonal to $x$), $\|u_2\|^2=\tfrac25-\tfrac49+\tfrac29=\tfrac{8}{45}$. $u_3=x^3-\tfrac{\langle x^3,x\rangle}{2/3}x=x^3-\tfrac{2/5}{2/3}x=x^3-\tfrac35x$, $\|u_3\|^2=\tfrac{8}{175}$. These are multiples of the Legendre polynomials $1,\ x,\ \tfrac12(3x^2-1),\ \tfrac12(5x^3-3x)$.
(b) Project By Theorem 10.11 (with $V=C[-1,1]$, $W=P_2$), $p=\sum_{j\le2}\tfrac{\langle|x|,u_j\rangle}{\|u_j\|^2}u_j$. Here $\langle|x|,1\rangle=1$, $\langle|x|,x\rangle=0$ (odd integrand), $\langle|x|,x^2-\tfrac13\rangle=2\int_0^1\left(x^3-\tfrac x3\right)dx=2\left(\tfrac14-\tfrac16\right)=\tfrac16$. So $$p=\frac12+\frac{1/6}{8/45}\left(x^2-\frac13\right)=\frac12+\frac{15}{16}\left(x^2-\frac13\right)=\frac{15x^2+3}{16}.$$
Error $\operatorname{dist}^2=\|\,|x|\,\|^2-\sum_j\tfrac{\langle|x|,u_j\rangle^2}{\|u_j\|^2}=\tfrac23-\tfrac12-\tfrac{(1/6)^2}{8/45}=\tfrac16-\tfrac{5}{32}=\tfrac{1}{96}$. Check by direct integration: $2\int_0^1\left(x-\tfrac{15x^2+3}{16}\right)^2dx=\tfrac1{96}$ ✓. Note that $p$ is not a Taylor polynomial ($|x|$ has none at $0$): it is the best fit in the mean-square sense over the whole interval.
✏️ Example 5 — Complement, projection and distance in $\mathbb{R}^4$
Let $W$ be the subspace of all $x\in\mathbb{R}^4$ with $x_1+x_2+x_3+x_4=0$ and $x_1-x_2+x_3-x_4=0$, and let $b=(1,2,3,4)$. Find $W^\perp$, $P_Wb$, the matrix of $P_{W^\perp}$, and $\operatorname{dist}(b,W)$.
$W^\perp$ $W=\{n_1,n_2\}^\perp$ for $n_1=(1,1,1,1)$, $n_2=(1,-1,1,-1)$, so by Theorem 10.10(3) $W^\perp=(\operatorname{span}\{n_1,n_2\})^{\perp\perp}=\operatorname{span}\{n_1,n_2\}$, of dimension $2$; and $\dim W=2$.
Project onto the small space $n_1\perp n_2$ and $\|n_1\|^2=\|n_2\|^2=4$. $\langle b,n_1\rangle=10$, $\langle b,n_2\rangle=-2$, so $P_{W^\perp}b=\tfrac{10}{4}n_1-\tfrac24n_2=(2,3,2,3)$, and $P_Wb=b-P_{W^\perp}b=(-1,-1,1,1)$.
Matrix $P_{W^\perp}=\tfrac14\left(n_1n_1^T+n_2n_2^T\right)=\tfrac12\begin{pmatrix}1 & 0 & 1 & 0\\ 0 & 1 & 0 & 1\\ 1 & 0 & 1 & 0\\ 0 & 1 & 0 & 1\end{pmatrix}$ — it averages the odd-indexed and the even-indexed coordinates. Indeed $P_{W^\perp}b=(2,3,2,3)$ ✓.
Distance and check $\operatorname{dist}(b,W)=\|b-P_Wb\|=\|(2,3,2,3)\|=\sqrt{26}$. Check: $(-1,-1,1,1)$ satisfies both equations of $W$, and $\|b\|^2=30=\|P_Wb\|^2+26=4+26$ ✓.

Now let the orthonormal family be infinite, as it is for Fourier series. Theorem 10.10 still applies to each finite piece, and that is enough to prove one of the most useful inequalities of analysis.

📘 Definition 10.6 — Fourier coefficients
If $(e_k)$ is an orthonormal family in an inner product space $V$ and $x\in V$, the numbers $\langle x,e_k\rangle$ are the Fourier coefficients of $x$ with respect to $(e_k)$, and $\sum_k\langle x,e_k\rangle e_k$ is its Fourier series. For example, in $C[-\pi,\pi]$ with $\langle f,g\rangle=\int_{-\pi}^{\pi}f\bar g$, the functions $e_n(t)=\dfrac{e^{int}}{\sqrt{2\pi}}$ $(n\in\mathbb{Z})$ are orthonormal, since $\int_{-\pi}^{\pi}e^{i(m-n)t}dt=0$ for $m\ne n$; over $\mathbb{R}$ one uses $\tfrac1{\sqrt{2\pi}},\tfrac{\cos nt}{\sqrt\pi},\tfrac{\sin nt}{\sqrt\pi}$ $(n\ge1)$.
📐 Theorem 10.12 — Bessel's inequality
Let $(e_k)_{k\ge1}$ be an orthonormal sequence (or finite family) in an inner product space $V$. For every $x\in V$, $$\sum_{k}|\langle x,e_k\rangle|^2\le\|x\|^2 .$$ In particular the series converges and $\langle x,e_k\rangle\to0$ as $k\to\infty$. Moreover, for each $N$, $\big\|x-\sum_{k\le N}\langle x,e_k\rangle e_k\big\|^2$ $=\|x\|^2-\sum_{k\le N}|\langle x,e_k\rangle|^2$.

Proof. Fix $N$ and let $W_N=\operatorname{span}\{e_1,\dots,e_N\}$. By Theorem 10.10(2), $\|x\|^2=\|P_{W_N}x\|^2+\|x-P_{W_N}x\|^2$, where $P_{W_N}x=\sum_{k\le N}\langle x,e_k\rangle e_k$ has $\|P_{W_N}x\|^2=\sum_{k\le N}|\langle x,e_k\rangle|^2$ (Theorem 10.5(3)). This is the last identity, and it gives $\sum_{k\le N}|\langle x,e_k\rangle|^2\le\|x\|^2$. The partial sums of a series of non-negative terms are thus bounded, so the series converges (§2.3) with sum $\le\|x\|^2$, and its terms tend to $0$ (§2.3). ∎

📐 Theorem 10.13 — Parseval's identity and complete orthonormal families

(a) Finite dimension. Let $e_1,\dots,e_m$ be orthonormal in $V$, $\dim V=n$. The following are equivalent: (i) $e_1,\dots,e_m$ is a basis (so $m=n$); (ii) $x=\sum_k\langle x,e_k\rangle e_k$ for every $x$; (iii) $\langle x,y\rangle=\sum_k\langle x,e_k\rangle\overline{\langle y,e_k\rangle}$ for all $x,y$; (iv) Parseval's identity $\|x\|^2=\sum_k|\langle x,e_k\rangle|^2$ for every $x$; (v) the only vector orthogonal to every $e_k$ is $0$.

(b) Orthonormal sequences. For an orthonormal sequence $(e_k)$ in any inner product space and $x\in V$, Parseval's identity $\|x\|^2=\sum_{k=1}^{\infty}|\langle x,e_k\rangle|^2$ holds if and only if the Fourier series converges to $x$ in norm: $\big\|x-\sum_{k\le N}\langle x,e_k\rangle e_k\big\|\to0$. It holds for every $x$ whenever finite linear combinations of the $e_k$ are dense in $V$.

(c) Trigonometric system. For every $f\in C[-\pi,\pi]$ (complex valued), $\displaystyle\int_{-\pi}^{\pi}|f|^2=\sum_{n\in\mathbb{Z}}|\langle f,e_n\rangle|^2$ with $e_n=e^{int}/\sqrt{2\pi}$.

Proof. (a) (i)⇒(ii) and (i)⇒(iii) are Theorem 10.5(4); (iii)⇒(iv) by taking $y=x$; (ii)⇒(iv) by Theorem 10.5(3). (iv)⇒(v): if $x\perp e_k$ for all $k$ then $\|x\|^2=0$. (v)⇒(i): let $W=\operatorname{span}\{e_k\}$. For any $x$, $x-P_Wx$ is orthogonal to every $e_k$, so by (v) $x=P_Wx\in W$. Hence $W=V$, and the $e_k$ are independent (Theorem 10.5(2)), so they form a basis.

(b) The equivalence is the last identity of Theorem 10.12: $\|x-S_N\|^2=\|x\|^2-\sum_{k\le N}|\langle x,e_k\rangle|^2$, where $S_N$ is the $N$-th partial sum. If the span is dense, then given $\varepsilon\gt0$ there is a finite combination $y$ of $e_1,\dots,e_N$ with $\|x-y\|\lt\varepsilon$. By best approximation (Theorem 10.11), $\|x-S_M\|\le\|x-y\|\lt\varepsilon$ for every $M\ge N$, because $y\in W_M$. So $S_M\to x$.

(c) Sketch. By (b) it suffices that trigonometric polynomials are dense in $C[-\pi,\pi]$ for the norm $\|f\|=\left(\int|f|^2\right)^{1/2}$. Given $f$ and $\varepsilon\gt0$, first replace $f$ by a continuous $g$ with $g(-\pi)=g(\pi)$ that agrees with $f$ except on a short interval near $\pi$, where it is linear; this changes $\int|f-g|^2$ by as little as we like. The Stone–Weierstrass theorem (§7.4), applied on the circle, gives a trigonometric polynomial $T$ with $\sup|g-T|$ small, and $\|g-T\|\le\sqrt{2\pi}\sup|g-T|$. ∎

For an orthonormal family that is not "complete", Bessel's inequality is strict for some $x$: in $\mathbb{R}^3$ with $e_1,e_2$ and $x=(1,1,1)$, $\sum|\langle x,e_k\rangle|^2=2\lt3$. Parseval's identity is the infinite-dimensional Pythagoras theorem, and it evaluates series: Exercise III.6 obtains $\sum1/n^2=\pi^2/6$ from $f(t)=t$.

The Adjoint of a Linear Map

📐 Theorem 10.14 — The Riesz representation theorem (finite dimension)
Let $V$ be a finite-dimensional inner product space and $\varphi:V\to\mathbb{F}$ a linear functional. There is a unique $y\in V$ with $\varphi(x)=\langle x,y\rangle$ for all $x\in V$; if $e_1,\dots,e_n$ is an orthonormal basis, $y=\sum_k\overline{\varphi(e_k)}\,e_k$. The map $y\mapsto\langle\cdot,y\rangle$ is a bijection $V\to V^*$ which is additive and conjugate-linear.

Proof. With $y=\sum_k\overline{\varphi(e_k)}e_k$ and Theorem 10.1(a), $\langle x,y\rangle=\sum_k\varphi(e_k)\langle x,e_k\rangle$ $=\varphi\big(\sum_k\langle x,e_k\rangle e_k\big)=\varphi(x)$ by Theorem 10.5(4). If $y'$ also works, $\langle x,y-y'\rangle=0$ for all $x$, so $y=y'$ (Theorem 10.1(b)). The map $y\mapsto\langle\cdot,y\rangle$ is injective by the same argument and surjective by what we just proved; $\langle\cdot,ay\rangle=\bar a\langle\cdot,y\rangle$ makes it conjugate-linear. ∎

So an inner product identifies $V$ with its dual space $V^*$ (§8.2) without choosing a basis — but only in finite dimension: the ⚠️ box gives a functional with no representing vector.

📘 Definition 10.7 — Adjoint
Let $V,W$ be inner product spaces over $\mathbb{F}$ and $T:V\to W$ linear. A map $T^*:W\to V$ is an adjoint of $T$ if $$\langle Tx,y\rangle_W=\langle x,T^*y\rangle_V\qquad\text{for all }x\in V,\ y\in W.$$ An operator $T\in\mathcal{L}(V)$ is self-adjoint if $T^*=T$; these operators, and the normal and unitary ones, are the subject of §10.2.
📐 Theorem 10.15 — Existence, uniqueness and matrix of the adjoint
If $V$ and $W$ are finite-dimensional, every linear $T:V\to W$ has a unique adjoint $T^*$, and $T^*$ is linear. If $\mathcal{B}$ and $\mathcal{C}$ are orthonormal bases of $V$ and $W$, then $$[T^*]_{\mathcal{C}}^{\mathcal{B}}=\left([T]_{\mathcal{B}}^{\mathcal{C}}\right)^*\quad\text{(the conjugate transpose)}.$$ In particular, for $A\in M_{m\times n}(\mathbb{F})$ acting from $\mathbb{F}^n$ to $\mathbb{F}^m$ with the standard inner products, $\langle Ax,y\rangle=\langle x,A^*y\rangle$: the adjoint of $A$ is $A^*$ (and $A^T$ over $\mathbb{R}$).

Proof. Fix $y\in W$. The map $x\mapsto\langle Tx,y\rangle$ is a linear functional on $V$, so by Theorem 10.14 there is exactly one vector, which we call $T^*y$, with $\langle Tx,y\rangle=\langle x,T^*y\rangle$ for all $x$. This proves existence and uniqueness. For linearity, $\langle x,T^*(ay+bz)\rangle=\langle Tx,ay+bz\rangle$ $=\bar a\langle Tx,y\rangle+\bar b\langle Tx,z\rangle$ $=\langle x,aT^*y+bT^*z\rangle$ for every $x$, so $T^*(ay+bz)=aT^*y+bT^*z$ by Theorem 10.1(b). For the matrix, let $\mathcal{B}=(e_j)$, $\mathcal{C}=(f_i)$. By Theorem 10.5(4), $Te_j=\sum_i\langle Te_j,f_i\rangle f_i$, so the $(i,j)$ entry of $A=[T]_{\mathcal{B}}^{\mathcal{C}}$ is $a_{ij}=\langle Te_j,f_i\rangle$. Likewise the $(j,i)$ entry of $[T^*]_{\mathcal{C}}^{\mathcal{B}}$ is $\langle T^*f_i,e_j\rangle=\overline{\langle e_j,T^*f_i\rangle}=\overline{\langle Te_j,f_i\rangle}=\overline{a_{ij}}$. ∎

📐 Theorem 10.16 — Properties of the adjoint
Let $U,V,W$ be finite-dimensional inner product spaces, $S,T:V\to W$ and $R:W\to U$ linear, $a\in\mathbb{F}$.
  1. $(S+T)^*=S^*+T^*$, $(aT)^*=\bar aT^*$, $(T^*)^*=T$, $(RT)^*=T^*R^*$, $I^*=I$; if $T$ is invertible, $(T^{-1})^*=(T^*)^{-1}$.
  2. $\ker T^*=(\operatorname{im}T)^\perp$, $\operatorname{im}T^*=(\ker T)^\perp$, $\ker T=(\operatorname{im}T^*)^\perp$, $\operatorname{im}T=(\ker T^*)^\perp$. Hence $\operatorname{rank}T^*=\operatorname{rank}T$, and $V=\ker T\oplus\operatorname{im}T^*$ orthogonally.
  3. For an operator $T\in\mathcal{L}(V)$: $\det T^*=\overline{\det T}$, the characteristic polynomial of $T^*$ is obtained from that of $T$ by conjugating the coefficients, and the eigenvalues of $T^*$ are $\bar\lambda_1,\dots,\bar\lambda_n$ (with the same algebraic multiplicities).
  4. A subspace $M\subseteq V$ is invariant under $T\in\mathcal{L}(V)$ if and only if $M^\perp$ is invariant under $T^*$.

Proof. (1) Each identity follows from the defining property and uniqueness; for example $\langle RTx,u\rangle=\langle Tx,R^*u\rangle=\langle x,T^*R^*u\rangle$, so $(RT)^*=T^*R^*$; and $\langle T^*y,x\rangle=\overline{\langle x,T^*y\rangle}=\overline{\langle Tx,y\rangle}=\langle y,Tx\rangle$ shows $(T^*)^*=T$. Applying the product rule to $T^{-1}T=I$ and $TT^{-1}=I$ gives $T^*(T^{-1})^*=I=(T^{-1})^*T^*$. (2) $y\in\ker T^*$ $\iff\langle x,T^*y\rangle=0\ \forall x$ $\iff\langle Tx,y\rangle=0\ \forall x$ $\iff y\perp\operatorname{im}T$. Applying this to $T^*$ (whose adjoint is $T$) gives $\ker T=(\operatorname{im}T^*)^\perp$; taking complements and using $M^{\perp\perp}=M$ (Theorem 10.10(3)) gives the other two. Then $\operatorname{rank}T^*=\dim(\ker T)^\perp$ $=\dim V-\dim\ker T=\operatorname{rank}T$ by rank–nullity. (3) In an orthonormal basis $[T^*]=A^*$ with $A=[T]$, and $\det(A^*)=\overline{\det A^T}=\overline{\det A}$ (§8.3). Then $\det(tI-A^*)=\overline{\det(\bar tI-A)}$, so $\lambda$ is a root of $p_{A^*}$ of multiplicity $m$ iff $\bar\lambda$ is a root of $p_A$ of multiplicity $m$. (4) If $TM\subseteq M$, $y\in M^\perp$ and $m\in M$, then $\langle m,T^*y\rangle=\langle Tm,y\rangle=0$, so $T^*y\in M^\perp$. The converse is the same statement for $T^*$ and $M^\perp$, since $T^{**}=T$ and $M^{\perp\perp}=M$. ∎

Part (2) is the "fundamental theorem of linear algebra": for $A\in M_{m\times n}(\mathbb{F})$, $\mathbb{F}^n=N(A)\oplus C(A^*)$ and $\mathbb{F}^m=C(A)\oplus N(A^*)$, both orthogonal. It immediately re-proves $\operatorname{rank}A=\operatorname{rank}A^*$ (row rank equals column rank, §8.3). Part (4) is the engine behind Schur's triangularisation (§9.2) and the spectral theorem (§10.2).

📐 Theorem 10.17 — Which idempotents are orthogonal projections?
Let $V$ be a finite-dimensional inner product space and $P\in\mathcal{L}(V)$ with $P^2=P$. The following are equivalent: (i) $P=P_W$ is the orthogonal projection onto $W=\operatorname{im}P$; (ii) $\ker P=(\operatorname{im}P)^\perp$; (iii) $P$ is self-adjoint, $P^*=P$; (iv) $\|Px\|\le\|x\|$ for every $x\in V$.

Proof. An idempotent is the projection onto $\operatorname{im}P$ along $\ker P$, and $V=\operatorname{im}P\oplus\ker P$ (§8.2); $P_W$ is the one whose kernel is $W^\perp$. This gives (i)⇔(ii). (ii)⇒(iii): for $x,y\in V$, $y-Py\in\ker P\perp\operatorname{im}P$, so $\langle Px,y\rangle=\langle Px,Py\rangle+\langle Px,y-Py\rangle=\langle Px,Py\rangle$; in the same way $\langle x,Py\rangle=\langle Px,Py\rangle$. Hence $\langle Px,y\rangle=\langle x,Py\rangle$. (iii)⇒(ii): $\ker P=\ker P^*=(\operatorname{im}P)^\perp$ by Theorem 10.16(2). (ii)⇒(iv): Pythagoras for $x=Px+(x-Px)$. (iv)⇒(ii): let $u\in\operatorname{im}P$, $v\in\ker P$ and $t\in\mathbb{F}$. Then $P(u+tv)=u$, so (iv) gives $\|u\|^2\le\|u+tv\|^2$ $=\|u\|^2+2\operatorname{Re}\big(\bar t\langle u,v\rangle\big)+|t|^2\|v\|^2$. Choose $t=-s\langle u,v\rangle$ with $s\gt0$: $0\le-2s|\langle u,v\rangle|^2+s^2|\langle u,v\rangle|^2\|v\|^2$. Dividing by $s$ and letting $s\to0^+$ gives $|\langle u,v\rangle|^2\le0$. So $\ker P\perp\operatorname{im}P$; since $\dim\ker P=\dim V-\dim\operatorname{im}P=\dim(\operatorname{im}P)^\perp$, $\ker P=(\operatorname{im}P)^\perp$. ∎

✏️ Example 6 — The adjoint of differentiation is not "minus differentiation" here
On $P_1(\mathbb{R})$ with $\langle p,q\rangle=\int_0^1p(x)q(x)\,dx$, let $D(p)=p'$. (a) Find $D^*$. (b) Show that the matrix of $D^*$ in the basis $\{1,x\}$ is not the transpose of the matrix of $D$ in that basis, and explain why.
Orthonormal basis Gram–Schmidt on $1,x$: $\|1\|=1$, so $f_1=1$; $x-\langle x,1\rangle1=x-\tfrac12$, and $\int_0^1(x-\tfrac12)^2dx=\tfrac1{12}$, so $f_2=\sqrt{12}\,(x-\tfrac12)=\sqrt3\,(2x-1)$.
Matrix of $D$ $Df_1=0$ and $Df_2=2\sqrt3=2\sqrt3\,f_1$, so $[D]_{\{f_1,f_2\}}=\begin{pmatrix}0 & 2\sqrt3\\ 0 & 0\end{pmatrix}$. By Theorem 10.15, $[D^*]_{\{f_1,f_2\}}=\begin{pmatrix}0 & 0\\ 2\sqrt3 & 0\end{pmatrix}$: $D^*f_1=2\sqrt3f_2=6(2x-1)$ and $D^*f_2=0$.
Formula $a+bx=\left(a+\tfrac b2\right)f_1+\tfrac{b}{2\sqrt3}f_2$, hence $$D^*(a+bx)=\left(a+\tfrac b2\right)6(2x-1)=(6a+3b)(2x-1).$$ Check with $p=x$, $q=1$: $\langle Dp,q\rangle=\int_0^11\,dx=1$ and $\langle p,D^*q\rangle=\int_0^1x\cdot6(2x-1)\,dx=6\left(\tfrac23-\tfrac12\right)=1$ ✓. Integration by parts explains the shape: $\int_0^1p'q=p(1)q(1)-p(0)q(0)-\int_0^1pq'$, so $D^*$ is $-D$ plus a correction coming from the boundary terms.
(b) In the basis $\{1,x\}$, $[D]=\begin{pmatrix}0 & 1\\ 0 & 0\end{pmatrix}$, whose transpose would send $1\mapsto x$. But $D^*(1)=6(2x-1)=-6+12x$, so $[D^*]_{\{1,x\}}=\begin{pmatrix}-6 & -3\\ 12 & 6\end{pmatrix}$. The rule "adjoint = conjugate transpose" needs an orthonormal basis, and $\{1,x\}$ is not orthogonal ($\langle1,x\rangle=\tfrac12$).

Least Squares

A system $Ax=b$ with more equations than unknowns is usually inconsistent: the measurements contain noise, and $b$ is not in the column space $C(A)$. Gauss's idea (used in 1801 to recover the orbit of the asteroid Ceres from a few weeks of observations) is to accept the best compromise, the $x$ that makes the residual $b-Ax$ as short as possible. Geometrically, $Ax$ should be the point of $C(A)$ nearest to $b$ — an orthogonal projection.

📘 Definition 10.8 — Least-squares solution
Let $A\in M_{m\times n}(\mathbb{F})$ and $b\in\mathbb{F}^m$. A vector $\hat x\in\mathbb{F}^n$ is a least-squares solution of $Ax=b$ if $\|A\hat x-b\|\le\|Ax-b\|$ for every $x\in\mathbb{F}^n$ (standard norm of $\mathbb{F}^m$). The vector $r=b-A\hat x$ is the residual, and $\|r\|^2=\sum_i|r_i|^2$ is the minimal sum of squared errors.
📐 Theorem 10.18 — The normal equations
Let $A\in M_{m\times n}(\mathbb{F})$ and $b\in\mathbb{F}^m$.
  1. $\hat x$ is a least-squares solution of $Ax=b$ if and only if it solves the normal equations $A^*A\hat x=A^*b$, if and only if $A\hat x=P_{C(A)}b$.
  2. The normal equations are always consistent. Moreover $N(A^*A)=N(A)$ and $\operatorname{rank}(A^*A)=\operatorname{rank}A$.
  3. The least-squares solution is unique if and only if $\operatorname{rank}A=n$ (independent columns); then $\hat x=(A^*A)^{-1}A^*b$, and the orthogonal projection onto $C(A)$ has matrix $P=A(A^*A)^{-1}A^*$ (equal to $QQ^*$ if $A=QR$).
  4. Among all least-squares solutions there is exactly one of smallest norm, and it is the one lying in $C(A^*)$.

Proof. (1) $\{Ax:x\in\mathbb{F}^n\}=C(A)$ is a finite-dimensional subspace, so by Theorem 10.11, $\|Ax-b\|$ is minimal iff $Ax=P_{C(A)}b$ iff $b-Ax\perp C(A)$. By Theorem 10.16(2), $C(A)^\perp=N(A^*)$, so this says $A^*(b-Ax)=0$. (2) $P_{C(A)}b$ lies in $C(A)$, so it equals $A\hat x$ for some $\hat x$, which then solves the normal equations. If $A^*Ax=0$, then $\|Ax\|^2=x^*A^*Ax=0$, so $Ax=0$; the reverse inclusion is clear. Rank–nullity then gives $\operatorname{rank}(A^*A)=n-\dim N(A)=\operatorname{rank}A$. (3) The $n\times n$ matrix $A^*A$ is invertible iff its rank is $n$ iff $\operatorname{rank}A=n$; then $P_{C(A)}b=A\hat x=A(A^*A)^{-1}A^*b$. If $A=QR$, then $A^*A=R^*R$ and $A(A^*A)^{-1}A^*=QR(R^*R)^{-1}R^*Q^*=QQ^*$. (4) The least-squares solutions form the coset $\hat x+N(A)$, and $\mathbb{F}^n=C(A^*)\oplus N(A)$ orthogonally (Theorem 10.16(2)). Write $\hat x=x_0+k$ with $x_0\in C(A^*)$, $k\in N(A)$; then $x_0$ is also a solution, and every solution is $x_0+k'$ with $k'\in N(A)$, of norm $\sqrt{\|x_0\|^2+\|k'\|^2}\ge\|x_0\|$, with equality only for $k'=0$. ∎

The minimum-norm solution in (4) is $A^+b$, where $A^+$ is the Moore–Penrose pseudoinverse, most easily described through the singular value decomposition (§10.2). For straight-line fitting $y\approx c_0+c_1t$ through data $(t_i,y_i)$, $i=1,\dots,m$, the matrix $A$ has columns $\mathbf 1=(1,\dots,1)$ and $t=(t_1,\dots,t_m)$, and the normal equations read $\begin{pmatrix}m & \sum t_i\\ \sum t_i & \sum t_i^2\end{pmatrix}\begin{pmatrix}c_0\\ c_1\end{pmatrix}=\begin{pmatrix}\sum y_i\\ \sum t_iy_i\end{pmatrix}$. They say precisely that the residual vector is orthogonal to $\mathbf 1$ and to $t$: the residuals add up to $0$, and so do the residuals weighted by $t_i$.

✏️ Example 7 — Fitting a line, and seeing the orthogonality
Find the least-squares line $y=c_0+c_1t$ for the data $(0,1),(1,2),(2,2),(3,4)$, the residual vector and the minimal sum of squared errors. Verify that the residual is orthogonal to $C(A)$, and use this to show that no other line does better.
Set up $A=\begin{pmatrix}1 & 0\\ 1 & 1\\ 1 & 2\\ 1 & 3\end{pmatrix}$, $b=(1,2,2,4)^T$. Then $A^TA=\begin{pmatrix}4 & 6\\ 6 & 14\end{pmatrix}$ and $A^Tb=\begin{pmatrix}9\\ 18\end{pmatrix}$.
Solve $\det A^TA=56-36=20$, so $c_0=\dfrac{14\cdot9-6\cdot18}{20}=\dfrac{18}{20}=0.9$ and $c_1=\dfrac{4\cdot18-6\cdot9}{20}=\dfrac{18}{20}=0.9$. The line is $y=0.9+0.9t$.
Residual $A\hat x=(0.9,1.8,2.7,3.6)$, so $r=b-A\hat x=(0.1,0.2,-0.7,0.4)$ and $\|r\|^2=0.01+0.04+0.49+0.16=0.7$. Orthogonality: $\langle r,\mathbf 1\rangle=0.1+0.2-0.7+0.4=0$ and $\langle r,t\rangle=0+0.2-1.4+1.2=0$ ✓.
Optimality For any other coefficient vector $x$, $b-Ax=r+A(\hat x-x)$ with $r\perp A(\hat x-x)\in C(A)$, so by Pythagoras $\|b-Ax\|^2=0.7+\|A(\hat x-x)\|^2\gt0.7$ unless $A(\hat x-x)=0$, i.e. $x=\hat x$ (the columns of $A$ are independent).
Interactive 2 — Least Squares: Shrinking the Sum of the Squares

The blue line $y=a+bt$ is your candidate; the red segments are the residuals $r_i=y_i-(a+bt_i)$ and the amber squares have areas proportional to $r_i^2$. The dashed purple line is the least-squares line. Move the sliders, or press Play to slide the candidate to the optimum: the total area falls to its minimum exactly when $\langle r,\mathbf 1\rangle=\sum r_i$ and $\langle r,t\rangle=\sum t_ir_i$ both vanish — the normal equations.

candidate—
Σ rᵢ²—
⟨r, 1⟩—
⟨r, t⟩—
least squares—
⚠️ Counterexamples and pitfalls
  1. Symmetric and bilinear is not enough: $x_1y_1-x_2y_2$ on $\mathbb{R}^2$ (the Minkowski form) has $\langle e_2,e_2\rangle=-1$; $x_1y_2+x_2y_1$ has $\langle e_1,e_1\rangle=0$. Positive definiteness must be checked.
  2. No conjugate, no inner product over $\mathbb{C}$: $z_1w_1+z_2w_2$ gives $\langle(1,i),(1,i)\rangle=0$. And $x^TAy$ with $A$ not symmetric (e.g. $x_1y_1+x_1y_2+2x_2y_2$) fails symmetry: $\langle e_1,e_2\rangle=1\ne0=\langle e_2,e_1\rangle$.
  3. Evaluation "inner products" need enough points: $p(0)q(0)+p(1)q(1)$ is an inner product on $P_1$ but not on $P_2$ ($p=x(x-1)$).
  4. Not every norm is Euclidean: $\|\cdot\|_1$, $\|\cdot\|_\infty$ on $\mathbb{R}^n$ $(n\ge2)$ and the sup norm on $C[0,1]$ violate the parallelogram law.
  5. Orthogonal does not imply independent if $0$ is allowed, and orthogonality is not transitive: $e_1\perp e_2$ and $e_2\perp e_1$, but $e_1\not\perp e_1$.
  6. Pythagoras' converse fails over $\mathbb{C}$: in $\mathbb{C}$, $x=1$, $y=i$ satisfy $\|x+y\|^2=2=\|x\|^2+\|y\|^2$, yet $\langle x,y\rangle=\bar i=-i\ne0$.
  7. Gram–Schmidt on dependent vectors produces $u_k=0$; normalising it is impossible. Order matters too: the output depends on the order of the input vectors.
  8. $P^2=P$ does not make $P$ an orthogonal projection: $P=\begin{pmatrix}1 & 1\\ 0 & 0\end{pmatrix}$ is idempotent, $P\ne P^T$, and $x=\tfrac1{\sqrt2}(1,1)$ has $\|Px\|=\sqrt2\gt1=\|x\|$.
  9. "Adjoint = conjugate transpose" only in orthonormal bases: Example 6 gives $[D^*]_{\{1,x\}}\ne[D]_{\{1,x\}}^T$. Also, the adjoint (a map $W\to V$) has nothing to do with the adjugate $\operatorname{adj}A$ of §8.3.
  10. In infinite dimensions, $V\ne W\oplus W^\perp$ is possible: in $C[-1,1]$ with $\int f\bar g$, the polynomials $W$ are dense, so $W^\perp=\{0\}$ and $(W^\perp)^\perp=C[-1,1]\ne W$ (Exercise III.9).
  11. Riesz and adjoints can fail in infinite dimensions: on $c_{00}$ (finitely supported sequences, $\ell^2$ inner product), $\varphi(x)=\sum_nx_n/n$ is linear but equals $\langle x,y\rangle$ only for $y=(1/n)\notin c_{00}$; so $T(x)=\varphi(x)e_1$ has no adjoint.
  12. Bessel is not Parseval: for an incomplete orthonormal family the inequality can be strict ($e_1,e_2$ in $\mathbb{R}^3$ and $x=(1,1,1)$: $2\lt3$). And an orthonormal "basis" of $\ell^2$ (the $e_n$) is not a Hamel basis: $(1/n)\in\ell^2$ is not a finite combination of the $e_n$.
  13. Square-summable is not summable: Fourier coefficients satisfy $\sum|\langle x,e_n\rangle|^2\lt\infty$ but not necessarily $\sum|\langle x,e_n\rangle|\lt\infty$: $x=(1/n)\in\ell^2$.
  14. Over $\mathbb{C}$ use $A^*$, not $A^T$: the normal equations are $A^*A\hat x=A^*b$; $A^TA$ can even be $0$ for $A\ne0$, e.g. $A=(1\ \ i)^T$.
  15. Least-squares solutions always exist but need not be unique: for dependent columns there is a whole affine subspace of them; only the minimum-norm one is unique.

Exercises

I
Inner products, norms and Cauchy–Schwarz.
  • 1 Show that $\langle A,B\rangle=\operatorname{tr}(B^TA)$ is an inner product on $M_n(\mathbb{R})$ and that it equals $\sum_{i,j}a_{ij}b_{ij}$. Find $\|I_n\|$, and show that symmetric and skew-symmetric matrices are orthogonal to each other.
  • 2 For which real $k$ is $\langle x,y\rangle=x_1y_1+kx_1y_2+kx_2y_1+4x_2y_2$ an inner product on $\mathbb{R}^2$?
  • 3 Is $\langle p,q\rangle=p(0)q(0)+p(1)q(1)+p(2)q(2)$ an inner product on $P_2(\mathbb{R})$? On $P_3(\mathbb{R})$?
  • 4 In $\mathbb{C}^2$ with the standard inner product let $x=(1+i,2)$, $y=(i,1-i)$. Compute $\langle x,y\rangle$, $\langle y,x\rangle$, $\|x\|$, $\|y\|$ and verify the Cauchy–Schwarz inequality.
  • 5 Find the maximum of $3x+4y-12z$ on the unit sphere of $\mathbb{R}^3$ and the point where it is attained.
  • 6 Prove that $(a_1+\cdots+a_n)\left(\tfrac1{a_1}+\cdots+\tfrac1{a_n}\right)\ge n^2$ for positive reals $a_i$, with equality iff all $a_i$ are equal.
  • 7 Show that the sup norm on $C[0,1]$ is not induced by any inner product.
  • 8 Prove the complex polarisation identity $\langle x,y\rangle=\tfrac14\sum_{k=0}^3i^k\|x+i^ky\|^2$ (with our convention: linear in the first slot).
1 $(B^TA)_{jj}=\sum_ib_{ij}a_{ij}$, so $\operatorname{tr}(B^TA)=\sum_{i,j}a_{ij}b_{ij}$ — the dot product of $\mathbb{R}^{n^2}$, hence an inner product. $\|I_n\|=\sqrt n$. If $S^T=S$ and $K^T=-K$: $\langle S,K\rangle=\operatorname{tr}(K^TS)=-\operatorname{tr}(KS)$, while also $\langle S,K\rangle=\langle K,S\rangle=\operatorname{tr}(S^TK)=\operatorname{tr}(SK)=\operatorname{tr}(KS)$; so $\langle S,K\rangle=0$. Thus $M_n(\mathbb{R})=\mathrm{Sym}\oplus\mathrm{Skew}$ orthogonally.
2 $G=\begin{pmatrix}1 & k\\ k & 4\end{pmatrix}$; by the $2\times2$ test (Example 1) we need $1\gt0$ and $4-k^2\gt0$: exactly $|k|\lt2$. (At $k=2$, $\langle x,x\rangle=(x_1+2x_2)^2$ vanishes at $(2,-1)$.)
3 On $P_2$: yes — a nonzero polynomial of degree $\le2$ cannot vanish at the three points $0,1,2$, so $\langle p,p\rangle\gt0$; symmetry and bilinearity are clear. On $P_3$: no — $p=x(x-1)(x-2)\ne0$ has $\langle p,p\rangle=0$.
4 $\langle x,y\rangle=(1+i)\overline{i}+2\,\overline{(1-i)}=(1+i)(-i)+2(1+i)=(1-i)+(2+2i)=3+i$; $\langle y,x\rangle=3-i$. $\|x\|^2=2+4=6$, $\|y\|^2=1+2=3$. Then $|\langle x,y\rangle|=\sqrt{10}\approx3.162\le\sqrt{18}\approx4.243$ ✓.
5 $|\langle v,(3,4,-12)\rangle|\le\|v\|\cdot13$, so the maximum is $13$, attained at $v=\tfrac1{13}(3,4,-12)$.
6 Cauchy–Schwarz for $u=(\sqrt{a_i})$ and $w=(1/\sqrt{a_i})$: $n^2=\langle u,w\rangle^2\le\|u\|^2\|w\|^2=\sum a_i\sum\tfrac1{a_i}$. Equality iff $u=cw$, i.e. $a_i=c$ for all $i$.
7 With $f=1$, $g=x$: $\|f+g\|_\infty=2$, $\|f-g\|_\infty=1$, so $\|f+g\|^2+\|f-g\|^2=5$, while $2\|f\|^2+2\|g\|^2=4$. The parallelogram law fails, so by Theorem 10.4 no inner product induces $\|\cdot\|_\infty$.
8 For a scalar $c$ with $|c|=1$: $\|x+cy\|^2=\|x\|^2+\bar c\langle x,y\rangle+c\langle y,x\rangle+\|y\|^2$. Multiply by $c$ (here $c=i^k$, so $c\bar c=1$): $c\|x+cy\|^2=c(\|x\|^2+\|y\|^2)+\langle x,y\rangle+c^2\langle y,x\rangle$. Summing over $c=1,i,-1,-i$: $\sum c=0$ and $\sum c^2=1-1+1-1=0$, leaving $4\langle x,y\rangle$.
II
Orthonormal bases, Gram–Schmidt, $QR$ and Gram matrices.
  • 1 Apply Gram–Schmidt to $(1,2,2)$, $(4,1,1)$ in $\mathbb{R}^3$, and extend the result to an orthonormal basis of $\mathbb{R}^3$.
  • 2 Find an orthonormal basis of $W=\{x\in\mathbb{R}^4:x_1+x_2+x_3+x_4=0\}$.
  • 3 Find the $QR$ factorisation of the $4\times2$ matrix with columns $(1,1,1,1)$ and $(1,2,3,4)$.
  • 4 Use a Gram determinant to find the area of the parallelogram in $\mathbb{R}^3$ spanned by $(1,2,3)$ and $(1,0,1)$, and check it with the cross product.
  • 5 Show that $\tfrac1{\sqrt{2\pi}},\ \tfrac{\cos t}{\sqrt\pi},\ \tfrac{\sin t}{\sqrt\pi},\ \tfrac{\cos2t}{\sqrt\pi},\ \tfrac{\sin2t}{\sqrt\pi},\dots$ is an orthonormal family in $C[-\pi,\pi]$ with $\langle f,g\rangle=\int_{-\pi}^{\pi}fg$.
  • 6 Prove or disprove: if $x\perp y$ and $y\perp z$, then $x\perp z$.
  • 7 Let $e_1,\dots,e_n$ be an orthonormal basis of a real inner product space and $T$ an operator. Show that the $(i,j)$ entry of $[T]$ is $\langle Te_j,e_i\rangle$ and that $\operatorname{tr}T=\sum_i\langle Te_i,e_i\rangle$.
  • 8 Prove Hadamard's inequality: for $A\in M_n(\mathbb{C})$ with columns $a_1,\dots,a_n$, $|\det A|\le\|a_1\|\cdots\|a_n\|$, with equality iff some $a_j=0$ or the columns are pairwise orthogonal.
1 $e_1=\tfrac13(1,2,2)$. $\langle v_2,e_1\rangle=\tfrac83$, $u_2=(4,1,1)-\tfrac89(1,2,2)=\tfrac79(4,-1,-1)$, so $e_2=\tfrac{1}{3\sqrt2}(4,-1,-1)$. A third vector orthogonal to both: $(1,2,2)\times(4,-1,-1)=(0,9,-9)$, so $e_3=\tfrac1{\sqrt2}(0,1,-1)$. Check $\langle e_2,e_3\rangle\propto-1+1=0$ ✓.
2 A basis of $W$: $(1,-1,0,0)$, $(0,1,-1,0)$, $(0,0,1,-1)$. Gram–Schmidt: $u_1=(1,-1,0,0)$; $u_2=(0,1,-1,0)+\tfrac12u_1=\tfrac12(1,1,-2,0)$; $u_3=(0,0,1,-1)+\tfrac13(1,1,-2,0)=\tfrac13(1,1,1,-3)$. Normalised: $\tfrac1{\sqrt2}(1,-1,0,0)$, $\tfrac1{\sqrt6}(1,1,-2,0)$, $\tfrac1{\sqrt{12}}(1,1,1,-3)$.
3 $\|a_1\|=2$, $q_1=\tfrac12(1,1,1,1)$; $\langle a_2,q_1\rangle=5$; $u_2=a_2-5q_1=(-\tfrac32,-\tfrac12,\tfrac12,\tfrac32)$, $\|u_2\|=\sqrt5$, $q_2=\tfrac1{2\sqrt5}(-3,-1,1,3)$. So $Q=[q_1\ q_2]$ and $R=\begin{pmatrix}2 & 5\\ 0 & \sqrt5\end{pmatrix}$. Check: $Q^TQ=I_2$ and $QR$ has second column $5q_1+\sqrt5q_2=(1,2,3,4)$ ✓.
4 $G=\begin{pmatrix}14 & 4\\ 4 & 2\end{pmatrix}$, $\det G=28-16=12$, area $\sqrt{12}=2\sqrt3$. Cross product $(1,2,3)\times(1,0,1)=(2,2,-2)$, of length $\sqrt{12}$ ✓.
5 Product-to-sum formulas: for $m\ne n$ (both $\ge0$), $\int_{-\pi}^{\pi}\cos mt\cos nt\,dt=\tfrac12\int_{-\pi}^{\pi}\left(\cos(m-n)t+\cos(m+n)t\right)dt=0$, similarly for $\sin\cdot\sin$, and $\int\cos mt\sin nt=0$ for all $m,n$ (odd integrand). Norms: $\int_{-\pi}^{\pi}1=2\pi$, $\int\cos^2nt=\int\sin^2nt=\pi$ for $n\ge1$.
6 False: $x=z=e_1$, $y=e_2$ in $\mathbb{R}^2$. (It is true that $x\perp y$ and $x\perp z$ imply $x\perp(ay+bz)$.)
7 By Theorem 10.5(4), $Te_j=\sum_i\langle Te_j,e_i\rangle e_i$, so column $j$ of $[T]$ has entries $\langle Te_j,e_i\rangle$. The trace is the sum of the diagonal entries $\langle Te_i,e_i\rangle$.
8 If the columns are dependent, $\det A=0$ and the inequality is clear (equality iff some $a_j=0$, since otherwise the right side is positive). Otherwise $A=QR$ with $Q$ unitary, so $|\det A|=|\det R|=\prod r_{jj}=\prod\|u_j\|$. Since $a_j=u_j+(\text{a vector orthogonal to }u_j)$, Pythagoras gives $\|u_j\|\le\|a_j\|$, with equality iff $a_j=u_j$, i.e. iff $a_j\perp a_1,\dots,a_{j-1}$. Multiplying gives the result and the equality case.
III
Orthogonal complements, projections, Bessel and Parseval.
  • 1 Find a basis of $W^\perp$ for $W=\operatorname{span}\{(1,1,0,1),(0,1,1,1)\}\subseteq\mathbb{R}^4$.
  • 2 Find the matrix of the orthogonal projection of $\mathbb{R}^3$ onto the plane $x+y-z=0$, and the distance from $(2,1,0)$ to this plane.
  • 3 Find $a,b$ minimising $\int_0^1(x^2-a-bx)^2dx$ and the minimum value.
  • 4 Prove: for a finite-dimensional subspace $W$, $\|P_Wx\|=\|x\|$ if and only if $x\in W$.
  • 5 Give an idempotent $2\times2$ real matrix $P$ which is not an orthogonal projection, and a vector $x$ with $\|Px\|\gt\|x\|$. Identify $\operatorname{im}P$ and $\ker P$.
  • 6 In $C[-\pi,\pi]$ with $\langle f,g\rangle=\int_{-\pi}^{\pi}fg$, compute the Fourier coefficients of $f(t)=t$ with respect to the orthonormal family $\sin nt/\sqrt\pi$ $(n\ge1)$. What does Bessel's inequality give? Using Parseval's identity (Theorem 10.13(c); the cosine coefficients vanish because $f$ is odd), deduce $\sum_{n\ge1}1/n^2=\pi^2/6$.
  • 7 Prove that in a finite-dimensional inner product space $(U+W)^\perp=U^\perp\cap W^\perp$ and $(U\cap W)^\perp=U^\perp+W^\perp$.
  • 8 In real $C[-1,1]$ with $\langle f,g\rangle=\int_{-1}^1fg$, let $W$ be the subspace of even functions. Show that $W^\perp$ is the subspace of odd functions and find the distance from $e^x$ to $W$.
  • 9 Let $W$ be the subspace of polynomials in real $C[-1,1]$ (same inner product). Use the Weierstrass approximation theorem (§5.2) to show $W^\perp=\{0\}$. Conclude that $(W^\perp)^\perp\ne W$ and $C[-1,1]\ne W\oplus W^\perp$.
1 $W^\perp$ is the null space of $\begin{pmatrix}1 & 1 & 0 & 1\\ 0 & 1 & 1 & 1\end{pmatrix}$: $x_2=-x_3-x_4$, $x_1=-x_2-x_4=x_3$. With $x_3,x_4$ free: basis $(1,-1,1,0)$, $(0,-1,0,1)$. Check: both are orthogonal to both spanning vectors ✓.
2 $n=(1,1,-1)$, $P=I-\tfrac{nn^T}{3}=\tfrac13\begin{pmatrix}2 & -1 & 1\\ -1 & 2 & 1\\ 1 & 1 & 2\end{pmatrix}$. Distance $\tfrac{|\langle(2,1,0),n\rangle|}{\|n\|}=\tfrac{3}{\sqrt3}=\sqrt3$. ($P$ is symmetric, idempotent, of trace $2$ ✓.)
3 Project $x^2$ onto $P_1$: the residual $x^2-a-bx$ must be orthogonal to $1$ and $x$: $\tfrac13=a+\tfrac b2$ and $\tfrac14=\tfrac a2+\tfrac b3$. Solution $b=1$, $a=-\tfrac16$. Minimum $\int_0^1\left(x^2-x+\tfrac16\right)^2dx=\tfrac1{180}$.
4 $\|x\|^2=\|P_Wx\|^2+\|x-P_Wx\|^2$, so equality holds iff $x=P_Wx$, i.e. iff $x\in W=\operatorname{im}P_W$.
5 $P=\begin{pmatrix}1 & 1\\ 0 & 0\end{pmatrix}$: $P^2=P$, $\operatorname{im}P=\operatorname{span}\{e_1\}$, $\ker P=\operatorname{span}\{(1,-1)\}$, not orthogonal to $e_1$. With $x=\tfrac1{\sqrt2}(1,1)$: $Px=(\sqrt2,0)$, $\|Px\|=\sqrt2\gt1$.
6 $\int_{-\pi}^{\pi}t\sin nt\,dt=\dfrac{2\pi(-1)^{n+1}}{n}$ (integrate by parts), so $\langle f,e_n\rangle=\dfrac{2\sqrt\pi(-1)^{n+1}}{n}$ and $|\langle f,e_n\rangle|^2=\dfrac{4\pi}{n^2}$. Also $\|f\|^2=\int_{-\pi}^{\pi}t^2dt=\tfrac{2\pi^3}{3}$. Bessel: $4\pi\sum\tfrac1{n^2}\le\tfrac{2\pi^3}{3}$, i.e. $\sum\tfrac1{n^2}\le\tfrac{\pi^2}{6}$. Parseval gives equality.
7 $x\perp(U+W)$ iff $x\perp u+w$ for all $u,w$ iff $x\perp U$ and $x\perp W$. For the second, apply the first to $U^\perp,W^\perp$: $(U^\perp+W^\perp)^\perp=U^{\perp\perp}\cap W^{\perp\perp}=U\cap W$, then take $\perp$ and use $M^{\perp\perp}=M$.
8 Every $f$ is $f_e+f_o$ with $f_e(x)=\tfrac{f(x)+f(-x)}2$ even and $f_o$ odd, and $\int_{-1}^1f_eg_o=0$ (odd integrand). So odd functions lie in $W^\perp$; conversely if $h\in W^\perp$ then $0=\langle h,h_e\rangle=\langle h_e,h_e\rangle$, so $h=h_o$. Thus the decomposition is orthogonal and $P_We^x=\cosh x$, $\operatorname{dist}=\|\sinh x\|=\left(\int_{-1}^1\sinh^2x\,dx\right)^{1/2}=\left(\tfrac{\sinh2}{2}-1\right)^{1/2}\approx0.9019$.
9 Let $g\in W^\perp$. By Weierstrass there are polynomials $p_n\to g$ uniformly on $[-1,1]$; then $\|g-p_n\|\le\sqrt2\sup|g-p_n|\to0$ and $\|g\|^2=\langle g,g-p_n\rangle+\langle g,p_n\rangle=\langle g,g-p_n\rangle\le\|g\|\,\|g-p_n\|\to0$. So $g=0$. Hence $(W^\perp)^\perp=\{0\}^\perp=C[-1,1]\ne W$ (e.g. $|x|\notin W$), and $W\oplus W^\perp=W\ne C[-1,1]$: Theorem 10.10 really needs $\dim W\lt\infty$.
IV
Adjoints and least squares.
  • 1 With the standard inner products, find $T^*$ for $T(x,y)=(x+2y,\,3x-y)$ on $\mathbb{R}^2$ and for $T(z,w)=(z+iw,\,2z)$ on $\mathbb{C}^2$.
  • 2 On $M_n(\mathbb{C})$ with $\langle X,Y\rangle=\operatorname{tr}(Y^*X)$, fix $M$ and let $L(X)=MX$ and $\tau(X)=X^T$. Find $L^*$ and $\tau^*$.
  • 3 For a linear map $T$ between finite-dimensional inner product spaces prove $\ker(T^*T)=\ker T$ and $\operatorname{im}(T^*T)=\operatorname{im}T^*$.
  • 4 On $\ell^2$ let $S(x_1,x_2,\dots)=(0,x_1,x_2,\dots)$. Find a map $S^*$ with $\langle Sx,y\rangle=\langle x,S^*y\rangle$, and show $S^*S=I$ but $SS^*\ne I$. Why is this impossible for operators on a finite-dimensional space?
  • 5 Find the least-squares line for the data $(1,1),(2,3),(3,4),(4,4),(5,6)$.
  • 6 Find the least-squares solution of the inconsistent system $x+y=1$, $x-y=1$, $x=3$, and the minimal value of $\|Ax-b\|^2$.
  • 7 For $A=\begin{pmatrix}1 & 1\\ 1 & 1\end{pmatrix}$ and $b=(1,3)^T$, describe all least-squares solutions and find the one of minimum norm.
  • 8 Applied. A materials lab loads a spring with forces $F=1,2,3,4$ N and records extensions $e=0.52,\ 0.98,\ 1.55,\ 2.01$ cm. Fit the model $e=cF$ (a line through the origin) by least squares, and give the spring constant $k=1/c$.
  • 9 Let $A\in M_{m\times n}(\mathbb{R})$ have rank $n$ and $P=A(A^TA)^{-1}A^T$. Verify directly that $P^T=P$, $P^2=P$ and $C(P)=C(A)$, so that $P$ is the orthogonal projection onto $C(A)$. What are $\operatorname{tr}P$ and the eigenvalues of $P$?
1 Real: $A=\begin{pmatrix}1 & 2\\ 3 & -1\end{pmatrix}$, $A^T=\begin{pmatrix}1 & 3\\ 2 & -1\end{pmatrix}$, so $T^*(x,y)=(x+3y,\,2x-y)$. Complex: $A=\begin{pmatrix}1 & i\\ 2 & 0\end{pmatrix}$, $A^*=\begin{pmatrix}1 & 2\\ -i & 0\end{pmatrix}$, so $T^*(z,w)=(z+2w,\,-iz)$. Check: $\langle T(1,0),(0,1)\rangle=\langle(1,2),(0,1)\rangle=2=\langle(1,0),T^*(0,1)\rangle=\langle(1,0),(2,0)\rangle$ ✓.
2 $\langle MX,Y\rangle=\operatorname{tr}(Y^*MX)=\operatorname{tr}\big((M^*Y)^*X\big)=\langle X,M^*Y\rangle$, so $L^*(Y)=M^*Y$. $\langle X^T,Y\rangle=\sum_{i,j}x_{ji}\overline{y_{ij}}=\langle X,Y^T\rangle$, so $\tau^*=\tau$: transposition is self-adjoint.
3 If $T^*Tx=0$ then $\|Tx\|^2=\langle T^*Tx,x\rangle=0$, so $Tx=0$; the converse is clear. Then $\operatorname{im}(T^*T)=(\ker(T^*T)^*)^\perp=(\ker T^*T)^\perp=(\ker T)^\perp=\operatorname{im}T^*$ by Theorem 10.16(2), since $(T^*T)^*=T^*T$.
4 $\langle Sx,y\rangle=\sum_{k\ge1}x_k\overline{y_{k+1}}$, so $S^*y=(y_2,y_3,\dots)$, the left shift. $S^*Sx=x$, but $SS^*y=(0,y_2,y_3,\dots)\ne y$ when $y_1\ne0$. In finite dimension $S^*S=I$ makes $S$ injective, hence invertible with $S^{-1}=S^*$, and then $SS^*=I$ too.
5 $m=5$, $\sum t=15$, $\sum t^2=55$, $\sum y=18$, $\sum ty=65$. Normal equations $5c_0+15c_1=18$, $15c_0+55c_1=65$; determinant $50$; $c_0=\tfrac{990-975}{50}=0.3$, $c_1=\tfrac{325-270}{50}=1.1$. Line $y=0.3+1.1t$.
6 $A=\begin{pmatrix}1 & 1\\ 1 & -1\\ 1 & 0\end{pmatrix}$, $b=(1,1,3)$: $A^TA=\operatorname{diag}(3,2)$, $A^Tb=(5,0)$, so $\hat x=\left(\tfrac53,0\right)$. Residual $\left(-\tfrac23,-\tfrac23,\tfrac43\right)$, $\|r\|^2=\tfrac{24}{9}=\tfrac83$; it is orthogonal to both columns ✓.
7 $A^TA=\begin{pmatrix}2 & 2\\ 2 & 2\end{pmatrix}$, $A^Tb=(4,4)$: the solutions are the line $x+y=2$ (all give $Ax=(2,2)$, the projection of $b$ onto $C(A)=\operatorname{span}\{(1,1)\}$). The minimum-norm one lies in $C(A^T)=\operatorname{span}\{(1,1)\}$: $(1,1)$.
8 For $e=cF$ the single normal equation is $\left(\sum F^2\right)c=\sum Fe$: $30c=0.52+1.96+4.65+8.04=15.17$, so $c\approx0.5057$ cm/N and $k=1/c\approx1.98$ N/cm. The residuals $0.014,-0.031,0.033,-0.013$ cm are small and change sign, so the linear model is adequate.
9 $P^T=A\left((A^TA)^{-1}\right)^TA^T=P$ because $A^TA$ is symmetric. $P^2=A(A^TA)^{-1}(A^TA)(A^TA)^{-1}A^T=P$. $C(P)\subseteq C(A)$, and $PA=A$ gives $C(A)\subseteq C(P)$. By Theorem 10.17 $P$ is the orthogonal projection onto $C(A)$. Its eigenvalues are $1$ (multiplicity $n$) and $0$ (multiplicity $m-n$), and $\operatorname{tr}P=\operatorname{rank}P=n$.
🎯 Exam corner CSIR Part BCSIR Part CGATE MA

E1. CSIR Part B GATE MA Let $P$ be the matrix (in the standard basis) of the orthogonal projection of $\mathbb{R}^3$ onto the line spanned by $(1,2,2)$. The $(2,3)$ entry of $P$ is (A) $\tfrac29$ (B) $\tfrac49$ (C) $\tfrac43$ (D) $\tfrac23$.

Idea Projection onto a line: $P=\dfrac{vv^T}{v^Tv}$ (Definition 10.5).
Work $v^Tv=9$, so $p_{23}=\dfrac{v_2v_3}{9}=\dfrac{4}{9}$. Sanity check: $\operatorname{tr}P=\tfrac{1+4+4}{9}=1=\operatorname{rank}P$. The distractors come from slips: (A) is the $(1,2)$ entry $\tfrac{v_1v_2}{9}$; (C) divides by $\|v\|=3$ instead of $\|v\|^2$; (D) is the third coordinate of the unit vector $v/\|v\|$.
Answer (B).

E2. CSIR Part B GATE MA On $P_2(\mathbb{R})$ with $\langle p,q\rangle=\int_0^1p(x)q(x)\,dx$, which monic quadratic is orthogonal to every polynomial of degree at most $1$? (A) $x^2-\tfrac13$ (B) $x^2-x$ (C) $x^2-x+\tfrac16$ (D) $x^2-\tfrac x2$.

Idea It suffices to test orthogonality to $1$ and $x$; use $\int_0^1x^k=\tfrac1{k+1}$.
Test (A): $\langle\cdot,1\rangle=\tfrac13-\tfrac13=0$ but $\langle\cdot,x\rangle=\tfrac14-\tfrac16=\tfrac1{12}\ne0$. (B): $\langle\cdot,1\rangle=\tfrac13-\tfrac12\ne0$. (D): $\langle\cdot,1\rangle=\tfrac13-\tfrac14\ne0$. (C): $\tfrac13-\tfrac12+\tfrac16=0$ and $\tfrac14-\tfrac13+\tfrac1{12}=0$ ✓. (It is a multiple of the shifted Legendre polynomial $6x^2-6x+1$.)
Answer (C).

E3. CSIR Part B Let $A=\begin{pmatrix}1 & 0\\ 0 & 1\\ 1 & 1\end{pmatrix}$ and $b=(1,1,0)^T$. The minimum of $\|Ax-b\|^2$ over $x\in\mathbb{R}^2$ is (A) $0$ (B) $\tfrac23$ (C) $\tfrac43$ (D) $2$.

Idea Solve the normal equations $A^TA\hat x=A^Tb$ (Theorem 10.18).
Work $A^TA=\begin{pmatrix}2 & 1\\ 1 & 2\end{pmatrix}$, $A^Tb=(1,1)^T$, so $\hat x=\left(\tfrac13,\tfrac13\right)$ and $r=b-A\hat x=\left(\tfrac23,\tfrac23,-\tfrac23\right)$, $\|r\|^2=\tfrac43$. Alternatively $C(A)^\perp$ is spanned by $n=(1,1,-1)$ and $\operatorname{dist}^2=\tfrac{\langle b,n\rangle^2}{\|n\|^2}=\tfrac43$. (A) would need $b\in C(A)$, but $b_3\ne b_1+b_2$.
Answer (C).

E4. CSIR Part C Which of the following define inner products on $\mathbb{R}^2$? (One or more options may be correct.) (A) $x_1y_1+2x_2y_2$ (B) $x_1y_1-x_1y_2-x_2y_1+2x_2y_2$ (C) $x_1y_2+x_2y_1$ (D) $x_1y_1+x_1y_2+2x_2y_2$.

(A) Yes $G=\operatorname{diag}(1,2)$; $\langle x,x\rangle=x_1^2+2x_2^2\gt0$ for $x\ne0$.
(B) Yes $G=\begin{pmatrix}1 & -1\\ -1 & 2\end{pmatrix}$ is symmetric with $a=1\gt0$, $\det G=1\gt0$; indeed $\langle x,x\rangle=(x_1-x_2)^2+x_2^2$.
(C) No $\langle e_1,e_1\rangle=0$ (and $\langle(1,-1),(1,-1)\rangle=-2$).
(D) No Not symmetric: $\langle e_1,e_2\rangle=1$ but $\langle e_2,e_1\rangle=0$ — even though $\langle x,x\rangle=x_1^2+x_1x_2+2x_2^2\gt0$ for $x\ne0$. Positivity of the quadratic form alone does not make an inner product.
Answer (A) and (B).

E5. CSIR Part C Let $V$ be a finite-dimensional complex inner product space and $P,T\in\mathcal{L}(V)$. Which of the following are true? (A) If $P^2=P$ and $\|Px\|\le\|x\|$ for all $x$, then $P^*=P$. (B) If $P^2=P$, then $P^*=P$. (C) $\ker T^*=(\operatorname{im}T)^\perp$. (D) If $\|x+y\|^2=\|x\|^2+\|y\|^2$, then $\langle x,y\rangle=0$.

(A) True Theorem 10.17, (iv)⇒(iii): a contractive idempotent is an orthogonal projection, hence self-adjoint.
(B) False $P=\begin{pmatrix}1 & 1\\ 0 & 0\end{pmatrix}$ on $\mathbb{C}^2$ is idempotent and $P^*=\begin{pmatrix}1 & 0\\ 1 & 0\end{pmatrix}\ne P$.
(C) True Theorem 10.16(2): $y\in\ker T^*\iff\langle Tx,y\rangle=0$ for all $x$.
(D) False The identity only gives $\operatorname{Re}\langle x,y\rangle=0$. In $V=\mathbb{C}$: $x=1$, $y=i$, $\|1+i\|^2=2=1+1$, but $\langle1,i\rangle=\bar i=-i$.
Answer (A) and (C).

E6. CSIR Part C Let $(e_n)_{n\ge1}$ be an orthonormal sequence in an inner product space $V$ and $x\in V$. Which of the following are necessarily true? (A) $\sum_n|\langle x,e_n\rangle|^2\le\|x\|^2$. (B) $\sum_n|\langle x,e_n\rangle|$ converges. (C) $\langle x,e_n\rangle\to0$. (D) $\|e_n-e_m\|=\sqrt2$ for $n\ne m$, so no subsequence of $(e_n)$ converges.

(A) True Bessel's inequality (Theorem 10.12).
(B) False In $\ell^2$ with the standard vectors $e_n$ and $x=(1,\tfrac12,\tfrac13,\dots)\in\ell^2$: $\langle x,e_n\rangle=\tfrac1n$, and $\sum\tfrac1n$ diverges.
(C) True The terms of the convergent series in (A) tend to $0$.
(D) True Pythagoras: $\|e_n-e_m\|^2=1+1=2$. A convergent subsequence would be Cauchy, which is impossible when all mutual distances are $\sqrt2$. (This is why the closed unit ball of an infinite-dimensional inner product space is not compact, §7.4.)
Answer (A), (C) and (D).
🧑‍🏫 For the teacher

Suggested plan — 3 lectures × 60 min.

  1. Inner products and Cauchy–Schwarz. Definition 10.1 with the five standard examples; ask the class why the conjugate is needed (the vector $(1,i)$). Theorem 10.1(d) and the $2\times2$ test via Example 1 (do (b) and (c) as a poll). Prove Cauchy–Schwarz in full — the "subtract the projection" proof previews Gram–Schmidt. Triangle inequality, parallelogram law, and a five-minute account of Theorem 10.4 with the $\|\cdot\|_1$ and $\|\cdot\|_\infty$ counterexamples. Example 2 on the board.
  2. Orthonormal bases, Gram–Schmidt, projections. Theorem 10.5, Gram–Schmidt proved by induction; project Interactive 1 on the first data set, then the dependent one. Example 3 (with $QR$ and the Gram determinant check). Orthogonal complements and Theorem 10.10, best approximation, the strategy box, Example 5. Example 4 (Legendre and $|x|$) as the bridge to function spaces; Bessel and Parseval with the $\pi^2/6$ computation of Exercise III.6.
  3. Adjoints and least squares. Riesz and the adjoint; Example 6 shows why orthonormal bases are required. Theorem 10.16(2) as the "four subspaces" picture; Theorem 10.17. Then least squares: Theorem 10.18 and Example 7, finishing with Interactive 2 (start on Example 7, then the outlier data set). Close with E5 or E6 as a group Part C exercise.

Discussion questions. (1) Which results of this section use only that $W$ is finite-dimensional, and which need $V$ finite-dimensional? (2) Why does the least-squares line pass through the point $(\bar t,\bar y)$ of means? (3) Gram–Schmidt applied to $1,x,x^2,\dots$ gives different polynomials for $\int_{-1}^1$, $\int_0^1$ and $\int_0^\infty e^{-x}$; what stays the same?

Misconceptions to watch for. Forgetting the conjugate (or conjugating the wrong slot) over $\mathbb{C}$; testing only $\langle x,x\rangle\gt0$ and forgetting symmetry (E4(D)); believing every idempotent is an orthogonal projection; taking the transpose of a matrix in a non-orthonormal basis as the adjoint; normalising too early in Gram–Schmidt and drowning in square roots; assuming $W^{\perp\perp}=W$ in infinite dimensions; confusing adjoint with adjugate.

Suggested homework. I(2, 3, 4, 6, 7), II(1, 3, 4, 8), III(2, 3, 5, 6, 9), IV(1, 3, 5, 6, 9); Exam corner E3, E4, E6.

🔬 Research–Teaching NexusResearch-tutored · optional, about 45 minutes

Research question. Gram–Schmidt is exact in exact arithmetic. How much orthogonality does it lose in floating-point arithmetic, and does the order of the subtractions matter?

Why it matters. Least-squares problems are solved millions of times a day: a GPS receiver estimates its position from the ranges to several satellites, weather centres fit models to observations, radio telescopes and MRI scanners reconstruct images, and every statistics package fits regressions. All of them orthogonalise columns, and the columns are often nearly dependent (ill-conditioned). If the computed $Q$ is far from orthogonal, $Q^TQ\ne I$ and every formula of this section that relies on orthonormality silently fails.

What to do.

  1. Implement classical Gram–Schmidt (CGS) exactly as in Theorem 10.6: $u_k=v_k-\sum_{j\lt k}\langle v_k,e_j\rangle e_j$, all inner products taken with the original $v_k$.
  2. Implement modified Gram–Schmidt (MGS): as soon as $e_j$ is computed, subtract its component from all the remaining vectors, so later inner products use the already-updated vectors. Prove that in exact arithmetic CGS and MGS produce the same $e_k$.
  3. For the Hilbert matrices $H_n=\left[\tfrac{1}{i+j-1}\right]$, $n=6,8,10,12$, compute the loss of orthogonality $\|Q^TQ-I\|_2$ for CGS, for MGS and for a library $QR$ routine (Householder reflections). Tabulate it against the condition number of $H_n$.
  4. Try "re-orthogonalisation": run CGS twice. What happens to the loss of orthogonality?

Tools: Python with NumPy (numpy.linalg.qr, numpy.linalg.cond, scipy.linalg.hilbert), or any language with double-precision arithmetic.

Hand in: Your table, a log-scale plot of the loss of orthogonality against $n$ for the three methods, and a paragraph explaining in terms of Theorem 10.6 why subtracting projections of nearly parallel vectors amplifies rounding errors.

Going further. Read about Householder reflections ($I-2uu^T$ with $\|u\|=1$, an orthogonal and self-adjoint matrix — §10.2) and why they are the standard way to compute $QR$; or about the normal equations versus $QR$ for least squares, where forming $A^TA$ squares the condition number.

Instructor note (double precision, spectral norm): for $n=6$, CGS gives about $3\times10^{-4}$, MGS about $4\times10^{-10}$, Householder about $5\times10^{-16}$; for $n=8$ (condition number $\approx1.5\times10^{10}$) CGS has lost orthogonality completely ($\approx1.4$) while MGS gives about $4\times10^{-7}$; for $n=10$, MGS gives about $10^{-4}$ and CGS about $3.5$; for $n=12$ (condition number $\approx1.6\times10^{16}$) even MGS fails ($\approx0.3$) and Householder stays near $10^{-15}$. The loss for MGS grows roughly like machine precision times the condition number, while CGS behaves much worse; running CGS twice restores orthogonality to near machine precision for moderate condition numbers.

🧪 Self-check quiz

0/5 correct

Q1. Which formula defines an inner product on $\mathbb{C}^2$ (linear in the first slot)?

Q2. In an inner product space, $|\langle x,y\rangle|=\|x\|\,\|y\|$ holds if and only if

Q3. For $W=\operatorname{span}\{(1,0,1,0),(0,1,0,1)\}\subseteq\mathbb{R}^4$, $\dim W^\perp$ equals

Q4. Gram–Schmidt is applied to $v_1,v_2,v_3$ where $v_1,v_2$ are independent and $v_3=v_1+2v_2$. Then

Q5. The least-squares solutions of $Ax=b$ ($A\in M_{m\times n}(\mathbb{C})$) are exactly the solutions of