10.1 Inner Product Spaces and Orthonormal Bases
An inner product adds lengths and angles to a vector space. With it come the Cauchy–Schwarz inequality, orthonormal bases built by Gram–Schmidt, orthogonal projections as best approximations, Bessel's inequality and Parseval's identity, least-squares solutions and the adjoint of an operator.
Chapters 8 and 9 studied vector spaces with nothing but addition and scaling. In such a space there is no way to say that two vectors are perpendicular, that one vector is longer than another, or that a point of a subspace is the one closest to a given vector. Yet these are exactly the questions that applications ask: which straight line fits a cloud of measurements best, which polynomial of degree $2$ approximates $|x|$ best on $[-1,1]$, how much of a signal is captured by its first ten Fourier coefficients. All of them are answered by one extra piece of structure, the inner product, which generalises the dot product of $\mathbb{R}^n$ to polynomials, matrices, functions and complex vectors.
This section builds the geometry that an inner product brings: the Cauchy–Schwarz and triangle inequalities, the parallelogram law, orthonormal bases and the Gram–Schmidt process (with the $QR$ factorisation it hides), orthogonal complements and projections, the best-approximation theorem, Bessel's inequality and Parseval's identity, the adjoint $T^*$ of an operator, and least squares. In CSIR NET this material appears directly — "which of these formulas is an inner product", "the projection of $v$ onto $W$", "the distance from $b$ to the column space", "an orthogonal polynomial for $\int_0^1pq$" — and it is the foundation for the spectral theorem (§10.2) and for quadratic forms (§10.3).
Throughout, $\mathbb{F}$ is $\mathbb{R}$ or $\mathbb{C}$, and $\bar a$ is the complex conjugate of $a$ (so $\bar a=a$ when $\mathbb{F}=\mathbb{R}$). For a matrix $A$, $A^*=\bar A^{T}$ is its conjugate transpose, as in §8.3; over $\mathbb{R}$, $A^*=A^T$. Vectors of $\mathbb{F}^n$ are columns.
Inner Products, Norms and the Cauchy–Schwarz Inequality
Let $V$ be a vector space over $\mathbb{F}$. An inner product on $V$ is a function $V\times V\to\mathbb{F}$, $(x,y)\mapsto\langle x,y\rangle$, such that for all $x,y,z\in V$ and $a,b\in\mathbb{F}$:
A vector space with an inner product is an inner product space (a Euclidean space if $\mathbb{F}=\mathbb{R}$, a unitary space if $\mathbb{F}=\mathbb{C}$). Its norm is $\|x\|=\sqrt{\langle x,x\rangle}$ and the distance from $x$ to $y$ is $\|x-y\|$. A vector with $\|x\|=1$ is a unit vector. (Many physics texts make the inner product linear in the second slot instead; every result below holds with the conjugates moved accordingly.)
The standard examples, which you should be able to verify on sight:
- $\mathbb{F}^n$: $\langle x,y\rangle=\sum_{k=1}^nx_k\overline{y_k}=y^*x$. Over $\mathbb{R}$ this is the dot product $x\cdot y=y^Tx$. The conjugate is essential: without it, $(1,i)$ would have "length squared" $1+i^2=0$.
- Weighted inner products on $\mathbb{F}^n$: $\langle x,y\rangle=y^*Gx$, where $G$ is a Hermitian matrix with $x^*Gx\gt0$ for all $x\ne0$ (a positive definite matrix, studied in §10.2 and §10.3). For instance $\langle x,y\rangle=\sum w_kx_k\overline{y_k}$ with all weights $w_k\gt0$.
- Matrices: on $M_{m\times n}(\mathbb{F})$, the Frobenius inner product $\langle A,B\rangle=\operatorname{tr}(B^*A)=\sum_{i,j}a_{ij}\overline{b_{ij}}$; then $\|A\|^2=\sum|a_{ij}|^2$.
- Continuous functions: on $C[a,b]$ (real or complex valued), $\langle f,g\rangle=\int_a^bf(t)\overline{g(t)}\,dt$. Positive definiteness uses continuity: if $f\ne0$ then $|f|^2\gt0$ on some subinterval, so $\int_a^b|f|^2\gt0$. Restricting to polynomials gives inner products on $P_n(\mathbb{F})$, the polynomials of degree at most $n$.
- Square-summable sequences: $\ell^2=\{(x_k):\sum|x_k|^2\lt\infty\}$ with $\langle x,y\rangle=\sum_kx_k\overline{y_k}$; the series converges absolutely because $2|x_k\overline{y_k}|\le|x_k|^2+|y_k|^2$. Its norm is the $\ell^2$ norm of §7.1.
Proof. (a) $\langle x,ay+bz\rangle=\overline{\langle ay+bz,x\rangle}$ $=\overline{a\langle y,x\rangle+b\langle z,x\rangle}$ $=\bar a\langle x,y\rangle+\bar b\langle x,z\rangle$ by (IP2) and (IP1). With $a=b=0$ we get $\langle x,0\rangle=0$, and $\langle0,x\rangle=\overline{\langle x,0\rangle}=0$. (b) Take $y=x$: $\langle x,x\rangle=0$ forces $x=0$ by (IP3). For the second claim apply this to $x-y$. (c) $\|ax\|^2=\langle ax,ax\rangle=a\bar a\langle x,x\rangle=|a|^2\|x\|^2$. (d) Write $x=\sum_jx_je_j$, $y=\sum_iy_ie_i$. By (IP1) and (a), $\langle x,y\rangle=\sum_{i,j}x_j\overline{y_i}\langle e_j,e_i\rangle=\sum_i\overline{y_i}\sum_jg_{ij}x_j=y^*Gx$. Conjugate symmetry gives $\overline{g_{ji}}=\overline{\langle e_i,e_j\rangle}=\langle e_j,e_i\rangle=g_{ij}$, so $G^*=G$, and $x^*Gx=\langle x,x\rangle\gt0$ for $x\ne0$. Conversely, if $G$ is Hermitian with $x^*Gx\gt0$ for $x\ne0$, then $(x,y)\mapsto y^*Gx$ is linear in $x$, satisfies $\overline{y^*Gx}=(y^*Gx)^*=x^*G^*y=x^*Gy$, and is positive definite. ∎
Part (d) turns the question "is this formula an inner product on $\mathbb{R}^n$?" into "is this symmetric matrix positive definite?". For $2\times2$ real matrices there is a quick test, proved in Example 1: $\begin{pmatrix}a & b\\ b & c\end{pmatrix}$ is positive definite if and only if $a\gt0$ and $ac-b^2\gt0$. The general test (leading principal minors) is proved in §10.3.
Proof. If $y=0$ both sides are $0$ and $x,y$ are dependent. Let $y\ne0$, put $t=\dfrac{\langle x,y\rangle}{\|y\|^2}$ and $z=x-ty$. Then $\langle z,y\rangle=\langle x,y\rangle-t\langle y,y\rangle=0$: we have split $x=ty+z$ into a multiple of $y$ and a vector orthogonal to $y$. Expanding with (IP1) and Theorem 10.1(a),
$$\|x\|^2=\langle ty+z,ty+z\rangle=|t|^2\|y\|^2+t\langle y,z\rangle+\bar t\langle z,y\rangle+\|z\|^2=|t|^2\|y\|^2+\|z\|^2,$$because $\langle z,y\rangle=0$ and $\langle y,z\rangle=\overline{\langle z,y\rangle}=0$. Hence $\|x\|^2\ge|t|^2\|y\|^2=\dfrac{|\langle x,y\rangle|^2}{\|y\|^2}$, which is the inequality. Equality holds iff $\|z\|=0$, i.e. iff $x=ty$. If $x=ty$ the vectors are dependent; conversely, if $x,y$ are dependent and $y\ne0$, then $x=sy$ for some scalar $s$, and then $t=\langle sy,y\rangle/\|y\|^2=s$, so $z=0$. ∎
In concrete spaces Cauchy–Schwarz becomes three famous inequalities: $\big|\sum x_k\overline{y_k}\big|^2\le\sum|x_k|^2\sum|y_k|^2$ in $\mathbb{F}^n$; $\big|\int_a^bf\bar g\big|^2\le\int_a^b|f|^2\int_a^b|g|^2$ in $C[a,b]$; and $|\operatorname{tr}(B^*A)|^2\le\operatorname{tr}(A^*A)\operatorname{tr}(B^*B)$ for matrices. Over $\mathbb{R}$ it allows us to define the angle $\theta\in[0,\pi]$ between nonzero vectors by $\cos\theta=\dfrac{\langle x,y\rangle}{\|x\|\,\|y\|}$, a number in $[-1,1]$.
- $\|x+y\|^2=\|x\|^2+2\operatorname{Re}\langle x,y\rangle+\|y\|^2$;
- triangle inequality $\|x+y\|\le\|x\|+\|y\|$, with equality iff one of $x,y$ is a non-negative real multiple of the other; hence $\|\cdot\|$ is a norm in the sense of §7.4;
- parallelogram law $\|x+y\|^2+\|x-y\|^2=2\|x\|^2+2\|y\|^2$;
- polarisation: over $\mathbb{R}$, $\langle x,y\rangle=\tfrac14\left(\|x+y\|^2-\|x-y\|^2\right)$; over $\mathbb{C}$, $\langle x,y\rangle=\tfrac14\sum_{k=0}^{3}i^k\|x+i^ky\|^2$.
Proof. (1) $\langle x+y,x+y\rangle=\|x\|^2+\langle x,y\rangle+\langle y,x\rangle+\|y\|^2$ and $\langle x,y\rangle+\overline{\langle x,y\rangle}=2\operatorname{Re}\langle x,y\rangle$. (2) Since $\operatorname{Re}\langle x,y\rangle\le|\langle x,y\rangle|\le\|x\|\|y\|$ by Cauchy–Schwarz, (1) gives $\|x+y\|^2\le(\|x\|+\|y\|)^2$. Equality forces $\operatorname{Re}\langle x,y\rangle=|\langle x,y\rangle|=\|x\|\|y\|$: by Theorem 10.2, $x,y$ are dependent, say (if $y\ne0$) $x=sy$, and then $\operatorname{Re}(s)\|y\|^2=|s|\|y\|^2$ forces $s$ real and $\ge0$. The other norm axioms are (IP3) and Theorem 10.1(c). (3) Add (1) to (1) with $y$ replaced by $-y$. (4) Subtracting the same two identities gives $\|x+y\|^2-\|x-y\|^2=4\operatorname{Re}\langle x,y\rangle$, which is the real formula. The complex formula is Exercise I.8. ∎
Proof (sketch, real case, all key steps). Necessity is Theorem 10.3(3), and uniqueness follows from polarisation. Conversely, assume the parallelogram law and define $\langle x,y\rangle=\tfrac14(\|x+y\|^2-\|x-y\|^2)$. Then $\langle x,y\rangle=\langle y,x\rangle$ and $\langle x,x\rangle=\tfrac14\|2x\|^2=\|x\|^2$, so only linearity in $x$ needs work. Fix $y$ and write $f(x)=\langle x,y\rangle$. The parallelogram law for the pairs $(x+y,x')$ and $(x-y,x')$ reads
$$\|x+x'+y\|^2+\|x-x'+y\|^2=2\|x+y\|^2+2\|x'\|^2,$$ $$\|x+x'-y\|^2+\|x-x'-y\|^2=2\|x-y\|^2+2\|x'\|^2.$$Subtracting and dividing by $4$ gives $f(x+x')+f(x-x')=2f(x)$ for all $x,x'$. With $x'=x$ (and $f(0)=0$) this says $f(2x)=2f(x)$; substituting $x=\tfrac{a+b}{2}$, $x'=\tfrac{a-b}{2}$ gives $f(a)+f(b)=2f\!\left(\tfrac{a+b}2\right)=f(a+b)$. So $f$ is additive, hence $f(qx)=qf(x)$ for every rational $q$ (first for $q\in\mathbb{N}$ by induction, then for $-1$ and $1/n$). Finally $f$ is continuous, since $\big|\|u\|-\|v\|\big|\le\|u-v\|$, and $\mathbb{Q}$ is dense in $\mathbb{R}$, so $f(tx)=tf(x)$ for every real $t$. In the complex case one defines $\langle x,y\rangle$ by the complex polarisation formula and checks in the same way that $\langle ix,y\rangle=i\langle x,y\rangle$. ∎
Theorem 10.4 settles at once which of the norms of §7.4 are "Euclidean". In $\mathbb{R}^2$ with $x=e_1$, $y=e_2$: for $\|\cdot\|_1$, $\|x+y\|^2+\|x-y\|^2=4+4=8\ne4=2+2$; for $\|\cdot\|_\infty$, $1+1=2\ne4$. So neither comes from an inner product, while $\|\cdot\|_2$ does. The same test shows that the sup norm on $C[0,1]$ is not induced by an inner product (Exercise I.7).
Orthonormal Bases and the Gram–Schmidt Process
- Pythagoras. If $v_1,\dots,v_k$ are pairwise orthogonal, then $\|v_1+\cdots+v_k\|^2=\|v_1\|^2+\cdots+\|v_k\|^2$.
- An orthogonal family of nonzero vectors is linearly independent.
- If $e_1,\dots,e_k$ are orthonormal and $x=\sum_ja_je_j$, then $a_j=\langle x,e_j\rangle$ and $\|x\|^2=\sum_j|a_j|^2$.
- If $e_1,\dots,e_n$ is an orthonormal basis of $V$, then for all $x,y\in V$: $$x=\sum_{j=1}^n\langle x,e_j\rangle e_j,\qquad \langle x,y\rangle=\sum_{j=1}^n\langle x,e_j\rangle\overline{\langle y,e_j\rangle}.$$
Proof. (1) Expanding $\langle\sum v_i,\sum v_j\rangle$ gives $\sum_{i,j}\langle v_i,v_j\rangle$, and only the terms $i=j$ survive. (2) If $\sum_ja_jv_j=0$, take the inner product with $v_i$: $0=\sum_ja_j\langle v_j,v_i\rangle=a_i\|v_i\|^2$, and $\|v_i\|\ne0$, so $a_i=0$. (3) $\langle x,e_i\rangle=\sum_ja_j\langle e_j,e_i\rangle=a_i$; the norm formula is (1) with $v_j=a_je_j$ and $\|a_je_j\|=|a_j|$. (4) The first formula is (3), applied to the expansion of $x$ in the basis. For the second, $\langle x,y\rangle=\big\langle\sum_ja_je_j,\sum_kb_ke_k\big\rangle$ $=\sum_{j,k}a_j\overline{b_k}\delta_{jk}=\sum_ja_j\overline{b_j}$ with $a_j=\langle x,e_j\rangle$, $b_j=\langle y,e_j\rangle$. ∎
Part (4) is the reason orthonormal bases are so convenient: coordinates are computed by inner products instead of by solving a linear system, and in those coordinates every inner product looks like the standard one on $\mathbb{F}^n$. Over $\mathbb{R}$ the converse of Pythagoras holds for two vectors ($\|x+y\|^2=\|x\|^2+\|y\|^2\iff\langle x,y\rangle=0$, by Theorem 10.3(1)); over $\mathbb{C}$ it only gives $\operatorname{Re}\langle x,y\rangle=0$.
Let $v_1,v_2,\dots$ (finitely many or a sequence) be linearly independent vectors in an inner product space. Define recursively
$$u_1=v_1,\qquad u_k=v_k-\sum_{j=1}^{k-1}\langle v_k,e_j\rangle e_j,\qquad e_k=\frac{u_k}{\|u_k\|}.$$Then every $u_k\ne0$, the family $(e_k)$ is orthonormal, and for every $k$: $\operatorname{span}\{e_1,\dots,e_k\}=\operatorname{span}\{v_1,\dots,v_k\}$ and $\langle v_k,e_k\rangle=\|u_k\|\gt0$. Moreover $(e_k)$ is the only orthonormal family with these two properties.
Proof. Induction on $k$. For $k=1$, $u_1=v_1\ne0$ (an independent family contains no zero vector) and $e_1=v_1/\|v_1\|$ has the required span; $\langle v_1,e_1\rangle=\|v_1\|\gt0$. Suppose $e_1,\dots,e_{k-1}$ are orthonormal with $\operatorname{span}\{e_1,\dots,e_{k-1}\}$ $=\operatorname{span}\{v_1,\dots,v_{k-1}\}=:W_{k-1}$.
$u_k\ne0$: the sum $\sum_{j\lt k}\langle v_k,e_j\rangle e_j$ lies in $W_{k-1}$; if $u_k=0$ then $v_k\in W_{k-1}=\operatorname{span}\{v_1,\dots,v_{k-1}\}$, contradicting independence. Orthogonality: for $i\lt k$, $\langle u_k,e_i\rangle=\langle v_k,e_i\rangle-\sum_{j\lt k}\langle v_k,e_j\rangle\langle e_j,e_i\rangle$ $=\langle v_k,e_i\rangle-\langle v_k,e_i\rangle=0$; so $e_k\perp e_1,\dots,e_{k-1}$, and $\|e_k\|=1$. Span: $e_k$ is a combination of $v_k$ and vectors of $W_{k-1}$, so $\operatorname{span}\{e_1,\dots,e_k\}\subseteq\operatorname{span}\{v_1,\dots,v_k\}$; conversely $v_k=u_k+\sum_{j\lt k}\langle v_k,e_j\rangle e_j$ $=\|u_k\|e_k+\sum_{j\lt k}\langle v_k,e_j\rangle e_j$. Both spans have dimension $k$ (by Theorem 10.5(2) the $e$'s are independent), so they are equal. Positivity: taking the inner product of the last display with $e_k$ gives $\langle v_k,e_k\rangle=\|u_k\|\gt0$.
Uniqueness: let $(f_k)$ be another orthonormal family with the two properties. Inductively assume $f_j=e_j$ for $j\lt k$. Then $f_k\in\operatorname{span}\{v_1,\dots,v_k\}=\operatorname{span}\{e_1,\dots,e_k\}$ and $f_k\perp e_1,\dots,e_{k-1}$, so by Theorem 10.5(3) $f_k=\langle f_k,e_k\rangle e_k=ce_k$ with $|c|=\|f_k\|=1$. Finally $0\lt\langle v_k,f_k\rangle=\bar c\langle v_k,e_k\rangle$ with $\langle v_k,e_k\rangle\gt0$ forces $\bar c\gt0$, hence $c=1$. ∎
If the input vectors are dependent, the same recursion produces $u_k=0$ exactly when $v_k\in\operatorname{span}\{v_1,\dots,v_{k-1}\}$; discarding those $v_k$ and continuing yields an orthonormal basis of $\operatorname{span}\{v_1,v_2,\dots\}$. Gram–Schmidt is therefore also a test for linear independence. In hand computations it is often easier to orthogonalise first and normalise at the end, using $u_k=v_k-\sum_{j\lt k}\dfrac{\langle v_k,u_j\rangle}{\langle u_j,u_j\rangle}u_j$ (the same vectors $u_k$, without square roots), and any $u_k$ may be replaced by a convenient nonzero multiple.
Proof. Apply Gram–Schmidt to any basis. For the extension, an orthonormal family $e_1,\dots,e_k$ is independent (Theorem 10.5(2)); extend it to a basis $e_1,\dots,e_k,v_{k+1},\dots,v_n$ (§8.1) and apply Gram–Schmidt. The first $k$ vectors are unchanged, because $u_j=e_j-\sum_{i\lt j}\langle e_j,e_i\rangle e_i=e_j$ has norm $1$. The last statement is Theorem 10.5(4). ∎
Proof. Let $a_1,\dots,a_n$ be the columns and $q_1,\dots,q_n$ their Gram–Schmidt vectors. From the proof of Theorem 10.6, $a_k=\|u_k\|q_k+\sum_{j\lt k}\langle a_k,q_j\rangle q_j$: column $k$ of $A$ is $Q$ times the column vector $(r_{1k},\dots,r_{kk},0,\dots,0)^T$. That is $A=QR$ with $R$ as stated. The $(i,j)$ entry of $Q^*Q$ is $q_i^*q_j=\langle q_j,q_i\rangle=\delta_{ij}$. For uniqueness, if $A=Q'R'$ is another such factorisation, then column $k$ of $A$ is $\sum_{j\le k}r'_{jk}q'_j$, so $\operatorname{span}\{a_1,\dots,a_k\}=\operatorname{span}\{q'_1,\dots,q'_k\}$ ($R'$ is invertible) and $\langle a_k,q'_k\rangle=r'_{kk}\gt0$; the uniqueness part of Theorem 10.6 gives $q'_k=q_k$, and then $R'=Q^*A=R$. A square matrix with $Q^*Q=I$ is unitary (§8.3). ∎
Proof. (a) $\overline{g_{ji}}=\overline{\langle v_i,v_j\rangle}=\langle v_j,v_i\rangle=g_{ij}$. Also $c^*Gc=\sum_{i,j}\overline{c_i}c_j\langle v_j,v_i\rangle=\big\langle\sum_jc_jv_j,\sum_ic_iv_i\big\rangle$. (b), (c) If the $v_j$ are dependent, choose $c\ne0$ with $\sum_jc_jv_j=0$; then $(Gc)_i=\sum_j\langle v_j,v_i\rangle c_j=\big\langle\sum_jc_jv_j,v_i\big\rangle=0$, so $Gc=0$ and $\det G=0$. If they are independent, let $W=\operatorname{span}\{v_j\}$, take the orthonormal basis $q_1,\dots,q_k$ of $W$ produced by Gram–Schmidt, and let $A\in M_k(\mathbb{F})$ be the matrix whose column $j$ is the coordinate vector of $v_j$. By Theorem 10.5(4), $g_{ij}=\langle v_j,v_i\rangle=a_i^*a_j$, i.e. $G=A^*A$. By Theorem 10.8 in coordinates, $A=QR$ with $Q$ unitary and $R$ upper triangular with diagonal $\|u_1\|$, $\dots$, $\|u_k\|$. Then $G=R^*Q^*QR=R^*R$ and $\det G=|\det R|^2=\prod\|u_j\|^2\gt0$. The volume statement is the "base times height" description: $\|u_k\|$ is the height of $v_k$ above the span of the previous vectors. ∎
Orthogonal Complements, Projections, Bessel and Parseval
- Every $x\in V$ can be written uniquely as $x=w+w'$ with $w\in W$ and $w'\in W^\perp$; namely $w=P_Wx:=\sum_{j=1}^k\langle x,e_j\rangle e_j$. Thus $V=W\oplus W^\perp$.
- $P_W$ is linear, $P_W^2=P_W$, $\operatorname{im}P_W=W$, $\ker P_W=W^\perp$, and $\|x\|^2=\|P_Wx\|^2+\|x-P_Wx\|^2$; in particular $\|P_Wx\|\le\|x\|$.
- If $\dim V=n$ is finite, then $\dim W^\perp=n-\dim W$, $(W^\perp)^\perp=W$, and $P_{W^\perp}=I-P_W$.
Proof. (1) Put $w=\sum_j\langle x,e_j\rangle e_j\in W$ and $w'=x-w$. For each $i$, $\langle w',e_i\rangle=\langle x,e_i\rangle-\sum_j\langle x,e_j\rangle\langle e_j,e_i\rangle=0$, so $w'$ is orthogonal to every $e_i$, hence to $W$. If also $x=w_1+w_1'$, then $w-w_1=w_1'-w'\in W\cap W^\perp=\{0\}$. (2) Linearity is visible from the formula. $P_Wx=x$ for $x\in W$ (by uniqueness, $x=x+0$), so $P_W^2=P_W$ and $\operatorname{im}P_W=W$; $P_Wx=0$ iff $x=w'\in W^\perp$. The norm identity is Pythagoras for $x=P_Wx+(x-P_Wx)$. (3) From $V=W\oplus W^\perp$, $\dim W^\perp=n-\dim W$. Every $w\in W$ is orthogonal to every vector of $W^\perp$, so $W\subseteq(W^\perp)^\perp$, and both have dimension $n-(n-\dim W)=\dim W$; hence they are equal. Finally $x=P_Wx+(I-P_W)x$ is the decomposition of $x$ relative to $W^\perp\oplus(W^\perp)^\perp$, so $(I-P_W)x=P_{W^\perp}x$. ∎
Proof. For $w\in W$, $x-w=(x-P_Wx)+(P_Wx-w)$ with $x-P_Wx\in W^\perp$ and $P_Wx-w\in W$. By Pythagoras, $\|x-w\|^2=\|x-P_Wx\|^2+\|P_Wx-w\|^2$, which exceeds $\|x-P_Wx\|^2$ unless $w=P_Wx$. If $x-w_0\perp W$, then $x=w_0+(x-w_0)$ is the decomposition of Theorem 10.10(1), so $w_0=P_Wx$; conversely $x-P_Wx\in W^\perp$. The condition $x-\sum_jc_jv_j\perp v_i$ for each $i$ is exactly $\langle x,v_i\rangle=\sum_jc_j\langle v_j,v_i\rangle$, and $G$ is invertible by Theorem 10.9, so the normal equations have exactly one solution. The distance formula is Theorem 10.10(2) together with $\|P_Wx\|^2=\sum_j|\langle x,e_j\rangle|^2$. ∎
- Orthogonal basis available? Use $P_Wx=\sum_j\dfrac{\langle x,u_j\rangle}{\langle u_j,u_j\rangle}u_j$ — no normalising needed. Otherwise run Gram–Schmidt, or…
- …solve the normal equations $Gc=(\langle x,v_i\rangle)$ with the Gram matrix of any basis. For $W=C(A)\subseteq\mathbb{F}^m$ this is $A^*Ac=A^*x$ (Theorem 10.18), and the projection matrix is $A(A^*A)^{-1}A^*$.
- Is $W^\perp$ smaller? For a hyperplane $W=\{x:\langle x,n\rangle=0\}$ use $P_Wx=x-\dfrac{\langle x,n\rangle}{\langle n,n\rangle}n$ and $\operatorname{dist}(x,W)=\dfrac{|\langle x,n\rangle|}{\|n\|}$. In general $P_Wx=x-P_{W^\perp}x$.
- Check that $x-P_Wx$ is orthogonal to each basis vector of $W$, and that $\|x\|^2=\|P_Wx\|^2+\operatorname{dist}^2$.
Now let the orthonormal family be infinite, as it is for Fourier series. Theorem 10.10 still applies to each finite piece, and that is enough to prove one of the most useful inequalities of analysis.
Proof. Fix $N$ and let $W_N=\operatorname{span}\{e_1,\dots,e_N\}$. By Theorem 10.10(2), $\|x\|^2=\|P_{W_N}x\|^2+\|x-P_{W_N}x\|^2$, where $P_{W_N}x=\sum_{k\le N}\langle x,e_k\rangle e_k$ has $\|P_{W_N}x\|^2=\sum_{k\le N}|\langle x,e_k\rangle|^2$ (Theorem 10.5(3)). This is the last identity, and it gives $\sum_{k\le N}|\langle x,e_k\rangle|^2\le\|x\|^2$. The partial sums of a series of non-negative terms are thus bounded, so the series converges (§2.3) with sum $\le\|x\|^2$, and its terms tend to $0$ (§2.3). ∎
(a) Finite dimension. Let $e_1,\dots,e_m$ be orthonormal in $V$, $\dim V=n$. The following are equivalent: (i) $e_1,\dots,e_m$ is a basis (so $m=n$); (ii) $x=\sum_k\langle x,e_k\rangle e_k$ for every $x$; (iii) $\langle x,y\rangle=\sum_k\langle x,e_k\rangle\overline{\langle y,e_k\rangle}$ for all $x,y$; (iv) Parseval's identity $\|x\|^2=\sum_k|\langle x,e_k\rangle|^2$ for every $x$; (v) the only vector orthogonal to every $e_k$ is $0$.
(b) Orthonormal sequences. For an orthonormal sequence $(e_k)$ in any inner product space and $x\in V$, Parseval's identity $\|x\|^2=\sum_{k=1}^{\infty}|\langle x,e_k\rangle|^2$ holds if and only if the Fourier series converges to $x$ in norm: $\big\|x-\sum_{k\le N}\langle x,e_k\rangle e_k\big\|\to0$. It holds for every $x$ whenever finite linear combinations of the $e_k$ are dense in $V$.
(c) Trigonometric system. For every $f\in C[-\pi,\pi]$ (complex valued), $\displaystyle\int_{-\pi}^{\pi}|f|^2=\sum_{n\in\mathbb{Z}}|\langle f,e_n\rangle|^2$ with $e_n=e^{int}/\sqrt{2\pi}$.
Proof. (a) (i)⇒(ii) and (i)⇒(iii) are Theorem 10.5(4); (iii)⇒(iv) by taking $y=x$; (ii)⇒(iv) by Theorem 10.5(3). (iv)⇒(v): if $x\perp e_k$ for all $k$ then $\|x\|^2=0$. (v)⇒(i): let $W=\operatorname{span}\{e_k\}$. For any $x$, $x-P_Wx$ is orthogonal to every $e_k$, so by (v) $x=P_Wx\in W$. Hence $W=V$, and the $e_k$ are independent (Theorem 10.5(2)), so they form a basis.
(b) The equivalence is the last identity of Theorem 10.12: $\|x-S_N\|^2=\|x\|^2-\sum_{k\le N}|\langle x,e_k\rangle|^2$, where $S_N$ is the $N$-th partial sum. If the span is dense, then given $\varepsilon\gt0$ there is a finite combination $y$ of $e_1,\dots,e_N$ with $\|x-y\|\lt\varepsilon$. By best approximation (Theorem 10.11), $\|x-S_M\|\le\|x-y\|\lt\varepsilon$ for every $M\ge N$, because $y\in W_M$. So $S_M\to x$.
(c) Sketch. By (b) it suffices that trigonometric polynomials are dense in $C[-\pi,\pi]$ for the norm $\|f\|=\left(\int|f|^2\right)^{1/2}$. Given $f$ and $\varepsilon\gt0$, first replace $f$ by a continuous $g$ with $g(-\pi)=g(\pi)$ that agrees with $f$ except on a short interval near $\pi$, where it is linear; this changes $\int|f-g|^2$ by as little as we like. The Stone–Weierstrass theorem (§7.4), applied on the circle, gives a trigonometric polynomial $T$ with $\sup|g-T|$ small, and $\|g-T\|\le\sqrt{2\pi}\sup|g-T|$. ∎
For an orthonormal family that is not "complete", Bessel's inequality is strict for some $x$: in $\mathbb{R}^3$ with $e_1,e_2$ and $x=(1,1,1)$, $\sum|\langle x,e_k\rangle|^2=2\lt3$. Parseval's identity is the infinite-dimensional Pythagoras theorem, and it evaluates series: Exercise III.6 obtains $\sum1/n^2=\pi^2/6$ from $f(t)=t$.
The Adjoint of a Linear Map
Proof. With $y=\sum_k\overline{\varphi(e_k)}e_k$ and Theorem 10.1(a), $\langle x,y\rangle=\sum_k\varphi(e_k)\langle x,e_k\rangle$ $=\varphi\big(\sum_k\langle x,e_k\rangle e_k\big)=\varphi(x)$ by Theorem 10.5(4). If $y'$ also works, $\langle x,y-y'\rangle=0$ for all $x$, so $y=y'$ (Theorem 10.1(b)). The map $y\mapsto\langle\cdot,y\rangle$ is injective by the same argument and surjective by what we just proved; $\langle\cdot,ay\rangle=\bar a\langle\cdot,y\rangle$ makes it conjugate-linear. ∎
So an inner product identifies $V$ with its dual space $V^*$ (§8.2) without choosing a basis — but only in finite dimension: the ⚠️ box gives a functional with no representing vector.
Proof. Fix $y\in W$. The map $x\mapsto\langle Tx,y\rangle$ is a linear functional on $V$, so by Theorem 10.14 there is exactly one vector, which we call $T^*y$, with $\langle Tx,y\rangle=\langle x,T^*y\rangle$ for all $x$. This proves existence and uniqueness. For linearity, $\langle x,T^*(ay+bz)\rangle=\langle Tx,ay+bz\rangle$ $=\bar a\langle Tx,y\rangle+\bar b\langle Tx,z\rangle$ $=\langle x,aT^*y+bT^*z\rangle$ for every $x$, so $T^*(ay+bz)=aT^*y+bT^*z$ by Theorem 10.1(b). For the matrix, let $\mathcal{B}=(e_j)$, $\mathcal{C}=(f_i)$. By Theorem 10.5(4), $Te_j=\sum_i\langle Te_j,f_i\rangle f_i$, so the $(i,j)$ entry of $A=[T]_{\mathcal{B}}^{\mathcal{C}}$ is $a_{ij}=\langle Te_j,f_i\rangle$. Likewise the $(j,i)$ entry of $[T^*]_{\mathcal{C}}^{\mathcal{B}}$ is $\langle T^*f_i,e_j\rangle=\overline{\langle e_j,T^*f_i\rangle}=\overline{\langle Te_j,f_i\rangle}=\overline{a_{ij}}$. ∎
- $(S+T)^*=S^*+T^*$, $(aT)^*=\bar aT^*$, $(T^*)^*=T$, $(RT)^*=T^*R^*$, $I^*=I$; if $T$ is invertible, $(T^{-1})^*=(T^*)^{-1}$.
- $\ker T^*=(\operatorname{im}T)^\perp$, $\operatorname{im}T^*=(\ker T)^\perp$, $\ker T=(\operatorname{im}T^*)^\perp$, $\operatorname{im}T=(\ker T^*)^\perp$. Hence $\operatorname{rank}T^*=\operatorname{rank}T$, and $V=\ker T\oplus\operatorname{im}T^*$ orthogonally.
- For an operator $T\in\mathcal{L}(V)$: $\det T^*=\overline{\det T}$, the characteristic polynomial of $T^*$ is obtained from that of $T$ by conjugating the coefficients, and the eigenvalues of $T^*$ are $\bar\lambda_1,\dots,\bar\lambda_n$ (with the same algebraic multiplicities).
- A subspace $M\subseteq V$ is invariant under $T\in\mathcal{L}(V)$ if and only if $M^\perp$ is invariant under $T^*$.
Proof. (1) Each identity follows from the defining property and uniqueness; for example $\langle RTx,u\rangle=\langle Tx,R^*u\rangle=\langle x,T^*R^*u\rangle$, so $(RT)^*=T^*R^*$; and $\langle T^*y,x\rangle=\overline{\langle x,T^*y\rangle}=\overline{\langle Tx,y\rangle}=\langle y,Tx\rangle$ shows $(T^*)^*=T$. Applying the product rule to $T^{-1}T=I$ and $TT^{-1}=I$ gives $T^*(T^{-1})^*=I=(T^{-1})^*T^*$. (2) $y\in\ker T^*$ $\iff\langle x,T^*y\rangle=0\ \forall x$ $\iff\langle Tx,y\rangle=0\ \forall x$ $\iff y\perp\operatorname{im}T$. Applying this to $T^*$ (whose adjoint is $T$) gives $\ker T=(\operatorname{im}T^*)^\perp$; taking complements and using $M^{\perp\perp}=M$ (Theorem 10.10(3)) gives the other two. Then $\operatorname{rank}T^*=\dim(\ker T)^\perp$ $=\dim V-\dim\ker T=\operatorname{rank}T$ by rank–nullity. (3) In an orthonormal basis $[T^*]=A^*$ with $A=[T]$, and $\det(A^*)=\overline{\det A^T}=\overline{\det A}$ (§8.3). Then $\det(tI-A^*)=\overline{\det(\bar tI-A)}$, so $\lambda$ is a root of $p_{A^*}$ of multiplicity $m$ iff $\bar\lambda$ is a root of $p_A$ of multiplicity $m$. (4) If $TM\subseteq M$, $y\in M^\perp$ and $m\in M$, then $\langle m,T^*y\rangle=\langle Tm,y\rangle=0$, so $T^*y\in M^\perp$. The converse is the same statement for $T^*$ and $M^\perp$, since $T^{**}=T$ and $M^{\perp\perp}=M$. ∎
Part (2) is the "fundamental theorem of linear algebra": for $A\in M_{m\times n}(\mathbb{F})$, $\mathbb{F}^n=N(A)\oplus C(A^*)$ and $\mathbb{F}^m=C(A)\oplus N(A^*)$, both orthogonal. It immediately re-proves $\operatorname{rank}A=\operatorname{rank}A^*$ (row rank equals column rank, §8.3). Part (4) is the engine behind Schur's triangularisation (§9.2) and the spectral theorem (§10.2).
Proof. An idempotent is the projection onto $\operatorname{im}P$ along $\ker P$, and $V=\operatorname{im}P\oplus\ker P$ (§8.2); $P_W$ is the one whose kernel is $W^\perp$. This gives (i)⇔(ii). (ii)⇒(iii): for $x,y\in V$, $y-Py\in\ker P\perp\operatorname{im}P$, so $\langle Px,y\rangle=\langle Px,Py\rangle+\langle Px,y-Py\rangle=\langle Px,Py\rangle$; in the same way $\langle x,Py\rangle=\langle Px,Py\rangle$. Hence $\langle Px,y\rangle=\langle x,Py\rangle$. (iii)⇒(ii): $\ker P=\ker P^*=(\operatorname{im}P)^\perp$ by Theorem 10.16(2). (ii)⇒(iv): Pythagoras for $x=Px+(x-Px)$. (iv)⇒(ii): let $u\in\operatorname{im}P$, $v\in\ker P$ and $t\in\mathbb{F}$. Then $P(u+tv)=u$, so (iv) gives $\|u\|^2\le\|u+tv\|^2$ $=\|u\|^2+2\operatorname{Re}\big(\bar t\langle u,v\rangle\big)+|t|^2\|v\|^2$. Choose $t=-s\langle u,v\rangle$ with $s\gt0$: $0\le-2s|\langle u,v\rangle|^2+s^2|\langle u,v\rangle|^2\|v\|^2$. Dividing by $s$ and letting $s\to0^+$ gives $|\langle u,v\rangle|^2\le0$. So $\ker P\perp\operatorname{im}P$; since $\dim\ker P=\dim V-\dim\operatorname{im}P=\dim(\operatorname{im}P)^\perp$, $\ker P=(\operatorname{im}P)^\perp$. ∎
Least Squares
A system $Ax=b$ with more equations than unknowns is usually inconsistent: the measurements contain noise, and $b$ is not in the column space $C(A)$. Gauss's idea (used in 1801 to recover the orbit of the asteroid Ceres from a few weeks of observations) is to accept the best compromise, the $x$ that makes the residual $b-Ax$ as short as possible. Geometrically, $Ax$ should be the point of $C(A)$ nearest to $b$ — an orthogonal projection.
- $\hat x$ is a least-squares solution of $Ax=b$ if and only if it solves the normal equations $A^*A\hat x=A^*b$, if and only if $A\hat x=P_{C(A)}b$.
- The normal equations are always consistent. Moreover $N(A^*A)=N(A)$ and $\operatorname{rank}(A^*A)=\operatorname{rank}A$.
- The least-squares solution is unique if and only if $\operatorname{rank}A=n$ (independent columns); then $\hat x=(A^*A)^{-1}A^*b$, and the orthogonal projection onto $C(A)$ has matrix $P=A(A^*A)^{-1}A^*$ (equal to $QQ^*$ if $A=QR$).
- Among all least-squares solutions there is exactly one of smallest norm, and it is the one lying in $C(A^*)$.
Proof. (1) $\{Ax:x\in\mathbb{F}^n\}=C(A)$ is a finite-dimensional subspace, so by Theorem 10.11, $\|Ax-b\|$ is minimal iff $Ax=P_{C(A)}b$ iff $b-Ax\perp C(A)$. By Theorem 10.16(2), $C(A)^\perp=N(A^*)$, so this says $A^*(b-Ax)=0$. (2) $P_{C(A)}b$ lies in $C(A)$, so it equals $A\hat x$ for some $\hat x$, which then solves the normal equations. If $A^*Ax=0$, then $\|Ax\|^2=x^*A^*Ax=0$, so $Ax=0$; the reverse inclusion is clear. Rank–nullity then gives $\operatorname{rank}(A^*A)=n-\dim N(A)=\operatorname{rank}A$. (3) The $n\times n$ matrix $A^*A$ is invertible iff its rank is $n$ iff $\operatorname{rank}A=n$; then $P_{C(A)}b=A\hat x=A(A^*A)^{-1}A^*b$. If $A=QR$, then $A^*A=R^*R$ and $A(A^*A)^{-1}A^*=QR(R^*R)^{-1}R^*Q^*=QQ^*$. (4) The least-squares solutions form the coset $\hat x+N(A)$, and $\mathbb{F}^n=C(A^*)\oplus N(A)$ orthogonally (Theorem 10.16(2)). Write $\hat x=x_0+k$ with $x_0\in C(A^*)$, $k\in N(A)$; then $x_0$ is also a solution, and every solution is $x_0+k'$ with $k'\in N(A)$, of norm $\sqrt{\|x_0\|^2+\|k'\|^2}\ge\|x_0\|$, with equality only for $k'=0$. ∎
The minimum-norm solution in (4) is $A^+b$, where $A^+$ is the Moore–Penrose pseudoinverse, most easily described through the singular value decomposition (§10.2). For straight-line fitting $y\approx c_0+c_1t$ through data $(t_i,y_i)$, $i=1,\dots,m$, the matrix $A$ has columns $\mathbf 1=(1,\dots,1)$ and $t=(t_1,\dots,t_m)$, and the normal equations read $\begin{pmatrix}m & \sum t_i\\ \sum t_i & \sum t_i^2\end{pmatrix}\begin{pmatrix}c_0\\ c_1\end{pmatrix}=\begin{pmatrix}\sum y_i\\ \sum t_iy_i\end{pmatrix}$. They say precisely that the residual vector is orthogonal to $\mathbf 1$ and to $t$: the residuals add up to $0$, and so do the residuals weighted by $t_i$.
- Symmetric and bilinear is not enough: $x_1y_1-x_2y_2$ on $\mathbb{R}^2$ (the Minkowski form) has $\langle e_2,e_2\rangle=-1$; $x_1y_2+x_2y_1$ has $\langle e_1,e_1\rangle=0$. Positive definiteness must be checked.
- No conjugate, no inner product over $\mathbb{C}$: $z_1w_1+z_2w_2$ gives $\langle(1,i),(1,i)\rangle=0$. And $x^TAy$ with $A$ not symmetric (e.g. $x_1y_1+x_1y_2+2x_2y_2$) fails symmetry: $\langle e_1,e_2\rangle=1\ne0=\langle e_2,e_1\rangle$.
- Evaluation "inner products" need enough points: $p(0)q(0)+p(1)q(1)$ is an inner product on $P_1$ but not on $P_2$ ($p=x(x-1)$).
- Not every norm is Euclidean: $\|\cdot\|_1$, $\|\cdot\|_\infty$ on $\mathbb{R}^n$ $(n\ge2)$ and the sup norm on $C[0,1]$ violate the parallelogram law.
- Orthogonal does not imply independent if $0$ is allowed, and orthogonality is not transitive: $e_1\perp e_2$ and $e_2\perp e_1$, but $e_1\not\perp e_1$.
- Pythagoras' converse fails over $\mathbb{C}$: in $\mathbb{C}$, $x=1$, $y=i$ satisfy $\|x+y\|^2=2=\|x\|^2+\|y\|^2$, yet $\langle x,y\rangle=\bar i=-i\ne0$.
- Gram–Schmidt on dependent vectors produces $u_k=0$; normalising it is impossible. Order matters too: the output depends on the order of the input vectors.
- $P^2=P$ does not make $P$ an orthogonal projection: $P=\begin{pmatrix}1 & 1\\ 0 & 0\end{pmatrix}$ is idempotent, $P\ne P^T$, and $x=\tfrac1{\sqrt2}(1,1)$ has $\|Px\|=\sqrt2\gt1=\|x\|$.
- "Adjoint = conjugate transpose" only in orthonormal bases: Example 6 gives $[D^*]_{\{1,x\}}\ne[D]_{\{1,x\}}^T$. Also, the adjoint (a map $W\to V$) has nothing to do with the adjugate $\operatorname{adj}A$ of §8.3.
- In infinite dimensions, $V\ne W\oplus W^\perp$ is possible: in $C[-1,1]$ with $\int f\bar g$, the polynomials $W$ are dense, so $W^\perp=\{0\}$ and $(W^\perp)^\perp=C[-1,1]\ne W$ (Exercise III.9).
- Riesz and adjoints can fail in infinite dimensions: on $c_{00}$ (finitely supported sequences, $\ell^2$ inner product), $\varphi(x)=\sum_nx_n/n$ is linear but equals $\langle x,y\rangle$ only for $y=(1/n)\notin c_{00}$; so $T(x)=\varphi(x)e_1$ has no adjoint.
- Bessel is not Parseval: for an incomplete orthonormal family the inequality can be strict ($e_1,e_2$ in $\mathbb{R}^3$ and $x=(1,1,1)$: $2\lt3$). And an orthonormal "basis" of $\ell^2$ (the $e_n$) is not a Hamel basis: $(1/n)\in\ell^2$ is not a finite combination of the $e_n$.
- Square-summable is not summable: Fourier coefficients satisfy $\sum|\langle x,e_n\rangle|^2\lt\infty$ but not necessarily $\sum|\langle x,e_n\rangle|\lt\infty$: $x=(1/n)\in\ell^2$.
- Over $\mathbb{C}$ use $A^*$, not $A^T$: the normal equations are $A^*A\hat x=A^*b$; $A^TA$ can even be $0$ for $A\ne0$, e.g. $A=(1\ \ i)^T$.
- Least-squares solutions always exist but need not be unique: for dependent columns there is a whole affine subspace of them; only the minimum-norm one is unique.
Exercises
- 1 Show that $\langle A,B\rangle=\operatorname{tr}(B^TA)$ is an inner product on $M_n(\mathbb{R})$ and that it equals $\sum_{i,j}a_{ij}b_{ij}$. Find $\|I_n\|$, and show that symmetric and skew-symmetric matrices are orthogonal to each other.
- 2 For which real $k$ is $\langle x,y\rangle=x_1y_1+kx_1y_2+kx_2y_1+4x_2y_2$ an inner product on $\mathbb{R}^2$?
- 3 Is $\langle p,q\rangle=p(0)q(0)+p(1)q(1)+p(2)q(2)$ an inner product on $P_2(\mathbb{R})$? On $P_3(\mathbb{R})$?
- 4 In $\mathbb{C}^2$ with the standard inner product let $x=(1+i,2)$, $y=(i,1-i)$. Compute $\langle x,y\rangle$, $\langle y,x\rangle$, $\|x\|$, $\|y\|$ and verify the Cauchy–Schwarz inequality.
- 5 Find the maximum of $3x+4y-12z$ on the unit sphere of $\mathbb{R}^3$ and the point where it is attained.
- 6 Prove that $(a_1+\cdots+a_n)\left(\tfrac1{a_1}+\cdots+\tfrac1{a_n}\right)\ge n^2$ for positive reals $a_i$, with equality iff all $a_i$ are equal.
- 7 Show that the sup norm on $C[0,1]$ is not induced by any inner product.
- 8 Prove the complex polarisation identity $\langle x,y\rangle=\tfrac14\sum_{k=0}^3i^k\|x+i^ky\|^2$ (with our convention: linear in the first slot).
- 1 Apply Gram–Schmidt to $(1,2,2)$, $(4,1,1)$ in $\mathbb{R}^3$, and extend the result to an orthonormal basis of $\mathbb{R}^3$.
- 2 Find an orthonormal basis of $W=\{x\in\mathbb{R}^4:x_1+x_2+x_3+x_4=0\}$.
- 3 Find the $QR$ factorisation of the $4\times2$ matrix with columns $(1,1,1,1)$ and $(1,2,3,4)$.
- 4 Use a Gram determinant to find the area of the parallelogram in $\mathbb{R}^3$ spanned by $(1,2,3)$ and $(1,0,1)$, and check it with the cross product.
- 5 Show that $\tfrac1{\sqrt{2\pi}},\ \tfrac{\cos t}{\sqrt\pi},\ \tfrac{\sin t}{\sqrt\pi},\ \tfrac{\cos2t}{\sqrt\pi},\ \tfrac{\sin2t}{\sqrt\pi},\dots$ is an orthonormal family in $C[-\pi,\pi]$ with $\langle f,g\rangle=\int_{-\pi}^{\pi}fg$.
- 6 Prove or disprove: if $x\perp y$ and $y\perp z$, then $x\perp z$.
- 7 Let $e_1,\dots,e_n$ be an orthonormal basis of a real inner product space and $T$ an operator. Show that the $(i,j)$ entry of $[T]$ is $\langle Te_j,e_i\rangle$ and that $\operatorname{tr}T=\sum_i\langle Te_i,e_i\rangle$.
- 8 Prove Hadamard's inequality: for $A\in M_n(\mathbb{C})$ with columns $a_1,\dots,a_n$, $|\det A|\le\|a_1\|\cdots\|a_n\|$, with equality iff some $a_j=0$ or the columns are pairwise orthogonal.
- 1 Find a basis of $W^\perp$ for $W=\operatorname{span}\{(1,1,0,1),(0,1,1,1)\}\subseteq\mathbb{R}^4$.
- 2 Find the matrix of the orthogonal projection of $\mathbb{R}^3$ onto the plane $x+y-z=0$, and the distance from $(2,1,0)$ to this plane.
- 3 Find $a,b$ minimising $\int_0^1(x^2-a-bx)^2dx$ and the minimum value.
- 4 Prove: for a finite-dimensional subspace $W$, $\|P_Wx\|=\|x\|$ if and only if $x\in W$.
- 5 Give an idempotent $2\times2$ real matrix $P$ which is not an orthogonal projection, and a vector $x$ with $\|Px\|\gt\|x\|$. Identify $\operatorname{im}P$ and $\ker P$.
- 6 In $C[-\pi,\pi]$ with $\langle f,g\rangle=\int_{-\pi}^{\pi}fg$, compute the Fourier coefficients of $f(t)=t$ with respect to the orthonormal family $\sin nt/\sqrt\pi$ $(n\ge1)$. What does Bessel's inequality give? Using Parseval's identity (Theorem 10.13(c); the cosine coefficients vanish because $f$ is odd), deduce $\sum_{n\ge1}1/n^2=\pi^2/6$.
- 7 Prove that in a finite-dimensional inner product space $(U+W)^\perp=U^\perp\cap W^\perp$ and $(U\cap W)^\perp=U^\perp+W^\perp$.
- 8 In real $C[-1,1]$ with $\langle f,g\rangle=\int_{-1}^1fg$, let $W$ be the subspace of even functions. Show that $W^\perp$ is the subspace of odd functions and find the distance from $e^x$ to $W$.
- 9 Let $W$ be the subspace of polynomials in real $C[-1,1]$ (same inner product). Use the Weierstrass approximation theorem (§5.2) to show $W^\perp=\{0\}$. Conclude that $(W^\perp)^\perp\ne W$ and $C[-1,1]\ne W\oplus W^\perp$.
- 1 With the standard inner products, find $T^*$ for $T(x,y)=(x+2y,\,3x-y)$ on $\mathbb{R}^2$ and for $T(z,w)=(z+iw,\,2z)$ on $\mathbb{C}^2$.
- 2 On $M_n(\mathbb{C})$ with $\langle X,Y\rangle=\operatorname{tr}(Y^*X)$, fix $M$ and let $L(X)=MX$ and $\tau(X)=X^T$. Find $L^*$ and $\tau^*$.
- 3 For a linear map $T$ between finite-dimensional inner product spaces prove $\ker(T^*T)=\ker T$ and $\operatorname{im}(T^*T)=\operatorname{im}T^*$.
- 4 On $\ell^2$ let $S(x_1,x_2,\dots)=(0,x_1,x_2,\dots)$. Find a map $S^*$ with $\langle Sx,y\rangle=\langle x,S^*y\rangle$, and show $S^*S=I$ but $SS^*\ne I$. Why is this impossible for operators on a finite-dimensional space?
- 5 Find the least-squares line for the data $(1,1),(2,3),(3,4),(4,4),(5,6)$.
- 6 Find the least-squares solution of the inconsistent system $x+y=1$, $x-y=1$, $x=3$, and the minimal value of $\|Ax-b\|^2$.
- 7 For $A=\begin{pmatrix}1 & 1\\ 1 & 1\end{pmatrix}$ and $b=(1,3)^T$, describe all least-squares solutions and find the one of minimum norm.
- 8 Applied. A materials lab loads a spring with forces $F=1,2,3,4$ N and records extensions $e=0.52,\ 0.98,\ 1.55,\ 2.01$ cm. Fit the model $e=cF$ (a line through the origin) by least squares, and give the spring constant $k=1/c$.
- 9 Let $A\in M_{m\times n}(\mathbb{R})$ have rank $n$ and $P=A(A^TA)^{-1}A^T$. Verify directly that $P^T=P$, $P^2=P$ and $C(P)=C(A)$, so that $P$ is the orthogonal projection onto $C(A)$. What are $\operatorname{tr}P$ and the eigenvalues of $P$?
E1. CSIR Part B GATE MA Let $P$ be the matrix (in the standard basis) of the orthogonal projection of $\mathbb{R}^3$ onto the line spanned by $(1,2,2)$. The $(2,3)$ entry of $P$ is (A) $\tfrac29$ (B) $\tfrac49$ (C) $\tfrac43$ (D) $\tfrac23$.
E2. CSIR Part B GATE MA On $P_2(\mathbb{R})$ with $\langle p,q\rangle=\int_0^1p(x)q(x)\,dx$, which monic quadratic is orthogonal to every polynomial of degree at most $1$? (A) $x^2-\tfrac13$ (B) $x^2-x$ (C) $x^2-x+\tfrac16$ (D) $x^2-\tfrac x2$.
E3. CSIR Part B Let $A=\begin{pmatrix}1 & 0\\ 0 & 1\\ 1 & 1\end{pmatrix}$ and $b=(1,1,0)^T$. The minimum of $\|Ax-b\|^2$ over $x\in\mathbb{R}^2$ is (A) $0$ (B) $\tfrac23$ (C) $\tfrac43$ (D) $2$.
E4. CSIR Part C Which of the following define inner products on $\mathbb{R}^2$? (One or more options may be correct.) (A) $x_1y_1+2x_2y_2$ (B) $x_1y_1-x_1y_2-x_2y_1+2x_2y_2$ (C) $x_1y_2+x_2y_1$ (D) $x_1y_1+x_1y_2+2x_2y_2$.
E5. CSIR Part C Let $V$ be a finite-dimensional complex inner product space and $P,T\in\mathcal{L}(V)$. Which of the following are true? (A) If $P^2=P$ and $\|Px\|\le\|x\|$ for all $x$, then $P^*=P$. (B) If $P^2=P$, then $P^*=P$. (C) $\ker T^*=(\operatorname{im}T)^\perp$. (D) If $\|x+y\|^2=\|x\|^2+\|y\|^2$, then $\langle x,y\rangle=0$.
E6. CSIR Part C Let $(e_n)_{n\ge1}$ be an orthonormal sequence in an inner product space $V$ and $x\in V$. Which of the following are necessarily true? (A) $\sum_n|\langle x,e_n\rangle|^2\le\|x\|^2$. (B) $\sum_n|\langle x,e_n\rangle|$ converges. (C) $\langle x,e_n\rangle\to0$. (D) $\|e_n-e_m\|=\sqrt2$ for $n\ne m$, so no subsequence of $(e_n)$ converges.
Suggested plan — 3 lectures × 60 min.
- Inner products and Cauchy–Schwarz. Definition 10.1 with the five standard examples; ask the class why the conjugate is needed (the vector $(1,i)$). Theorem 10.1(d) and the $2\times2$ test via Example 1 (do (b) and (c) as a poll). Prove Cauchy–Schwarz in full — the "subtract the projection" proof previews Gram–Schmidt. Triangle inequality, parallelogram law, and a five-minute account of Theorem 10.4 with the $\|\cdot\|_1$ and $\|\cdot\|_\infty$ counterexamples. Example 2 on the board.
- Orthonormal bases, Gram–Schmidt, projections. Theorem 10.5, Gram–Schmidt proved by induction; project Interactive 1 on the first data set, then the dependent one. Example 3 (with $QR$ and the Gram determinant check). Orthogonal complements and Theorem 10.10, best approximation, the strategy box, Example 5. Example 4 (Legendre and $|x|$) as the bridge to function spaces; Bessel and Parseval with the $\pi^2/6$ computation of Exercise III.6.
- Adjoints and least squares. Riesz and the adjoint; Example 6 shows why orthonormal bases are required. Theorem 10.16(2) as the "four subspaces" picture; Theorem 10.17. Then least squares: Theorem 10.18 and Example 7, finishing with Interactive 2 (start on Example 7, then the outlier data set). Close with E5 or E6 as a group Part C exercise.
Discussion questions. (1) Which results of this section use only that $W$ is finite-dimensional, and which need $V$ finite-dimensional? (2) Why does the least-squares line pass through the point $(\bar t,\bar y)$ of means? (3) Gram–Schmidt applied to $1,x,x^2,\dots$ gives different polynomials for $\int_{-1}^1$, $\int_0^1$ and $\int_0^\infty e^{-x}$; what stays the same?
Misconceptions to watch for. Forgetting the conjugate (or conjugating the wrong slot) over $\mathbb{C}$; testing only $\langle x,x\rangle\gt0$ and forgetting symmetry (E4(D)); believing every idempotent is an orthogonal projection; taking the transpose of a matrix in a non-orthonormal basis as the adjoint; normalising too early in Gram–Schmidt and drowning in square roots; assuming $W^{\perp\perp}=W$ in infinite dimensions; confusing adjoint with adjugate.
Suggested homework. I(2, 3, 4, 6, 7), II(1, 3, 4, 8), III(2, 3, 5, 6, 9), IV(1, 3, 5, 6, 9); Exam corner E3, E4, E6.
Research question. Gram–Schmidt is exact in exact arithmetic. How much orthogonality does it lose in floating-point arithmetic, and does the order of the subtractions matter?
Why it matters. Least-squares problems are solved millions of times a day: a GPS receiver estimates its position from the ranges to several satellites, weather centres fit models to observations, radio telescopes and MRI scanners reconstruct images, and every statistics package fits regressions. All of them orthogonalise columns, and the columns are often nearly dependent (ill-conditioned). If the computed $Q$ is far from orthogonal, $Q^TQ\ne I$ and every formula of this section that relies on orthonormality silently fails.
What to do.
- Implement classical Gram–Schmidt (CGS) exactly as in Theorem 10.6: $u_k=v_k-\sum_{j\lt k}\langle v_k,e_j\rangle e_j$, all inner products taken with the original $v_k$.
- Implement modified Gram–Schmidt (MGS): as soon as $e_j$ is computed, subtract its component from all the remaining vectors, so later inner products use the already-updated vectors. Prove that in exact arithmetic CGS and MGS produce the same $e_k$.
- For the Hilbert matrices $H_n=\left[\tfrac{1}{i+j-1}\right]$, $n=6,8,10,12$, compute the loss of orthogonality $\|Q^TQ-I\|_2$ for CGS, for MGS and for a library $QR$ routine (Householder reflections). Tabulate it against the condition number of $H_n$.
- Try "re-orthogonalisation": run CGS twice. What happens to the loss of orthogonality?
Tools: Python with NumPy (numpy.linalg.qr, numpy.linalg.cond, scipy.linalg.hilbert), or any language with double-precision arithmetic.
Hand in: Your table, a log-scale plot of the loss of orthogonality against $n$ for the three methods, and a paragraph explaining in terms of Theorem 10.6 why subtracting projections of nearly parallel vectors amplifies rounding errors.
Going further. Read about Householder reflections ($I-2uu^T$ with $\|u\|=1$, an orthogonal and self-adjoint matrix — §10.2) and why they are the standard way to compute $QR$; or about the normal equations versus $QR$ for least squares, where forming $A^TA$ squares the condition number.
🧪 Self-check quiz
Q1. Which formula defines an inner product on $\mathbb{C}^2$ (linear in the first slot)?
Q2. In an inner product space, $|\langle x,y\rangle|=\|x\|\,\|y\|$ holds if and only if
Q3. For $W=\operatorname{span}\{(1,0,1,0),(0,1,0,1)\}\subseteq\mathbb{R}^4$, $\dim W^\perp$ equals
Q4. Gram–Schmidt is applied to $v_1,v_2,v_3$ where $v_1,v_2$ are independent and $v_3=v_1+2v_2$. Then
Q5. The least-squares solutions of $Ax=b$ ($A\in M_{m\times n}(\mathbb{C})$) are exactly the solutions of