Chapter 10 · Inner Product Spaces and Quadratic Forms

10.2 Self-Adjoint, Normal and Unitary Operators

Normal operators — those that commute with their adjoint — are exactly the operators that an orthonormal basis diagonalises. Self-adjoint, unitary and positive operators are the normal operators whose spectra are real, unimodular and non-negative; and every matrix, normal or not, becomes diagonal under two unitary changes of basis.

Chapter 9 asked when an operator can be diagonalised, and answered: exactly when its minimal polynomial splits into distinct linear factors. But a diagonalising basis found that way can be badly skewed — two eigenvectors may be almost parallel, and then the change of basis $P$ and its inverse have huge entries, and errors are magnified in every computation. An inner product lets us ask for much more: a basis of eigenvectors that is orthonormal. Then $P$ is unitary, $P^{-1}=P^*$ costs nothing to compute, lengths and angles are preserved, and the operator is literally a list of independent stretchings along perpendicular axes. The question of this section is: which operators admit such a basis? The answer — the spectral theorem — is strikingly clean: over $\mathbb{C}$ exactly the normal operators ($T^*T=TT^*$), over $\mathbb{R}$ exactly the self-adjoint ones.

The spectral theorem is the workhorse behind quantum mechanics (observables are self-adjoint, so they have real eigenvalues and orthogonal eigenstates), vibration analysis (normal modes), principal component analysis in statistics, and the classification of quadratic forms in §10.3. From it we get positive operators and their square roots, the polar decomposition, and the singular value decomposition, which diagonalises every matrix with two unitary matrices. CSIR NET examines this material relentlessly: "which of the following matrices are normal / unitarily diagonalisable / positive definite", eigenvalues of Hermitian and unitary matrices, $\|A\|$ versus the largest eigenvalue, square roots, and Gershgorin-type eigenvalue bounds.

Conventions. Throughout, $V$ is a finite-dimensional inner product space over $\mathbb{F}=\mathbb{R}$ or $\mathbb{C}$ (§10.1), with the inner product linear in the first variable and conjugate-linear in the second; on $\mathbb{F}^n$, $\langle x,y\rangle=y^*x=\sum_ix_i\overline{y_i}$. For $T\in \mathcal{L}(V)$ the adjoint $T^*$ is the unique operator with $\langle Tx,y\rangle=\langle x,T^*y\rangle$ for all $x,y$ (§10.1). We use freely: $(S+T)^*=S^*+T^*$, $(cT)^*=\bar cT^*$, $(ST)^*=T^*S^*$, $T^{**}=T$, $\ker T^*=(\operatorname{im}T)^\perp$, and, in an orthonormal basis, $[T^*]=[T]^*=\overline{[T]}^{\,T}$. For a matrix $A\in M_n(\mathbb{F})$ we use the operator $x\mapsto Ax$ on $\mathbb{F}^n$, whose adjoint is $x\mapsto A^*x$.

Self-Adjoint, Unitary and Normal Operators

📘 Definition 10.9 — Self-adjoint, skew-adjoint, unitary and normal operators

An operator $T\in \mathcal{L}(V)$ is

  • self-adjoint (or Hermitian) if $T^*=T$; for matrices: $A^*=A$ ($a_{ji}=\overline{a_{ij}}$), and for real matrices $A^T=A$ (symmetric);
  • skew-adjoint (skew-Hermitian) if $T^*=-T$; for real matrices $A^T=-A$ (skew-symmetric);
  • unitary if $T^*T=TT^*=I$; for matrices $U^*U=I$; a real unitary matrix ($Q^TQ=I$) is called orthogonal;
  • normal if $T^*T=TT^*$, i.e. $T$ commutes with its adjoint.

Self-adjoint, skew-adjoint and unitary operators are all normal (in each case $T^*$ is $\pm T$ or $T^{-1}$, which commute with $T$). Diagonal matrices are normal, and so is every matrix $aI+bK$ with $K$ skew-symmetric and $a,b$ scalars, e.g. $\begin{pmatrix}a&-b\\b&a\end{pmatrix}$.

Think of the analogy with complex numbers: $A\mapsto A^*$ plays the role of $z\mapsto\bar z$. Hermitian matrices are the "real numbers", skew-Hermitian ones the "imaginary numbers", unitary ones the "numbers of modulus one", and every matrix splits as $A=H+iK$ with $H=\tfrac12(A+A^*)$ and $K=\tfrac1{2i}(A-A^*)$ both Hermitian (Exercise I.2). The analogy is exact for normal matrices and breaks down for the others — that is the content of the whole section.

📐 Theorem 10.19 — Characterisations of unitary operators
For $T\in \mathcal{L}(V)$ the following are equivalent:
  1. $T^*T=I$;
  2. $\langle Tx,Ty\rangle=\langle x,y\rangle$ for all $x,y\in V$ ($T$ preserves inner products);
  3. $\|Tx\|=\|x\|$ for all $x\in V$ ($T$ is an isometry);
  4. $T$ maps every orthonormal basis of $V$ to an orthonormal basis;
  5. $T$ maps some orthonormal basis of $V$ to an orthonormal basis.
In that case $T$ is invertible, $T^{-1}=T^*$, $TT^*=I$, so $T$ is unitary; the matrix of $T$ in any orthonormal basis is unitary; and $|\det T|=1$.

Proof. (1)⇒(2): $\langle Tx,Ty\rangle=\langle x,T^*Ty\rangle=\langle x,y\rangle$. (2)⇒(3): take $y=x$. (3)⇒(2): the inner product is recovered from norms by the polarisation identities. Over $\mathbb{R}$, $\langle x,y\rangle=\tfrac14\left(\|x+y\|^2-\|x-y\|^2\right)$. Over $\mathbb{C}$, $$\langle x,y\rangle=\frac14\sum_{k=0}^{3}i^k\,\|x+i^ky\|^2 .$$ To check the complex one, expand $\|x+i^ky\|^2=\|x\|^2+\|y\|^2$ $+\overline{i^k}\langle x,y\rangle+i^k\langle y,x\rangle$ and multiply by $i^k$: the result is $i^k(\|x\|^2+\|y\|^2)+\langle x,y\rangle+i^{2k}\langle y,x\rangle$. Summing over $k=0,1,2,3$ kills the first and last terms (since $\sum i^k=\sum(-1)^k=0$) and leaves $4\langle x,y\rangle$. As $T$ preserves every norm on the right-hand side, it preserves $\langle x,y\rangle$. (2)⇒(4): $\langle Te_i,Te_j\rangle=\langle e_i,e_j\rangle=\delta_{ij}$, and $n$ orthonormal vectors in an $n$-dimensional space form a basis. (4)⇒(5) is immediate. (5)⇒(1): if $(e_i)$ and $(Te_i)$ are orthonormal, then for $x=\sum a_ie_i$ and $y=\sum b_je_j$, $$\langle Tx,Ty\rangle=\sum_{i,j}a_i\overline{b_j}\,\langle Te_i,Te_j\rangle=\sum_ia_i\overline{b_i}=\langle x,y\rangle,$$ so $\langle T^*Tx-x,y\rangle=0$ for all $y$, whence $T^*Tx=x$ for every $x$. Finally $T^*T=I$ makes $T$ injective, hence invertible ($V$ is finite-dimensional), and then $T^{-1}=T^*$ and $TT^*=I$. In an orthonormal basis $[T]^*[T]=[T^*T]=I$, and $|\det T|^2=\overline{\det T}\det T=\det(T^*T)=1$. ∎

📐 Theorem 10.20 — The quadratic-form test $\langle Tx,x\rangle$
Let $T\in \mathcal{L}(V)$.
  1. If $\mathbb{F}=\mathbb{C}$ and $\langle Tx,x\rangle=0$ for every $x\in V$, then $T=0$. Hence two operators on a complex space with $\langle Sx,x\rangle=\langle Tx,x\rangle$ for all $x$ are equal.
  2. If $\mathbb{F}=\mathbb{C}$, then $T$ is self-adjoint if and only if $\langle Tx,x\rangle\in\mathbb{R}$ for every $x$.
  3. If $\mathbb{F}=\mathbb{R}$, statement 1 fails (rotation of $\mathbb{R}^2$ through $\tfrac\pi2$ has $\langle Tx,x\rangle=0$ for all $x$), but it holds for self-adjoint $T$: if $T^*=T$ and $\langle Tx,x\rangle=0$ for all $x$, then $T=0$.

Proof. (1) The operator version of polarisation is $$4\langle Tx,y\rangle=\sum_{k=0}^{3}i^k\,\big\langle T(x+i^ky),\,x+i^ky\big\rangle,$$ verified exactly as above: the $k$-th term equals $i^k\big(\langle Tx,x\rangle+\langle Ty,y\rangle\big)$ plus $\langle Tx,y\rangle$ plus $i^{2k}\langle Ty,x\rangle$, and the sum over $k$ leaves $4\langle Tx,y\rangle$. If every $\langle Tz,z\rangle$ vanishes, then $\langle Tx,y\rangle=0$ for all $x,y$; with $y=Tx$ this gives $\|Tx\|^2=0$. Apply this to $S-T$ for the second sentence. (2) For every $x$, $$\langle Tx,x\rangle-\overline{\langle Tx,x\rangle}=\langle Tx,x\rangle-\langle x,Tx\rangle=\langle (T-T^*)x,x\rangle .$$ So all $\langle Tx,x\rangle$ are real iff $\langle(T-T^*)x,x\rangle=0$ for all $x$, iff $T=T^*$ by (1). (3) For the rotation $T(x_1,x_2)=(-x_2,x_1)$, $\langle Tx,x\rangle=-x_2x_1+x_1x_2=0$. If $T$ is self-adjoint on a real space, $\langle Ty,x\rangle=\langle y,Tx\rangle=\langle Tx,y\rangle$, so $$\langle T(x+y),x+y\rangle-\langle T(x-y),x-y\rangle=2\langle Tx,y\rangle+2\langle Ty,x\rangle=4\langle Tx,y\rangle,$$ and we finish as in (1). ∎

📐 Theorem 10.21 — Spectra of self-adjoint, skew-adjoint and unitary operators
Let $T\in \mathcal{L}(V)$, $Tx=\lambda x$ and $Ty=\mu y$ with $x,y\ne0$.
  1. If $T$ is self-adjoint, $\lambda\in\mathbb{R}$; and if $\lambda\ne\mu$ then $x\perp y$.
  2. If $T$ is skew-adjoint, $\lambda\in i\mathbb{R}$ (purely imaginary or $0$).
  3. If $T$ is unitary, $|\lambda|=1$; and if $\lambda\ne\mu$ then $x\perp y$.
  4. Every Hermitian matrix — in particular every real symmetric matrix — has a characteristic polynomial that splits over $\mathbb{R}$ into real linear factors. Hence every self-adjoint operator on a real inner product space of dimension $\ge1$ has an eigenvector.

Proof. (1) $\lambda\|x\|^2=\langle Tx,x\rangle=\langle x,Tx\rangle=\bar\lambda\|x\|^2$, so $\lambda=\bar\lambda$. Then $\mu$ is real too, and $\lambda\langle x,y\rangle=\langle Tx,y\rangle=\langle x,Ty\rangle=\bar\mu\langle x,y\rangle=\mu\langle x,y\rangle$, so $(\lambda-\mu)\langle x,y\rangle=0$. (2) $\lambda\|x\|^2=\langle Tx,x\rangle=-\langle x,Tx\rangle=-\bar\lambda\|x\|^2$, so $\lambda+\bar\lambda=2\operatorname{Re}\lambda=0$. (3) $\|x\|=\|Tx\|$ $=|\lambda|\,\|x\|$ by Theorem 10.19. Next $\langle x,y\rangle=\langle Tx,Ty\rangle=\lambda\bar\mu\langle x,y\rangle$; since $|\mu|=1$ we have $\bar\mu=1/\mu$, so $\lambda\bar\mu=\lambda/\mu\ne1$ when $\lambda\ne\mu$, forcing $\langle x,y\rangle=0$. (4) Regard a Hermitian $A\in M_n(\mathbb{C})$ (or a real symmetric one, for which $A^*=A^T=A$) as an operator on $\mathbb{C}^n$. By the fundamental theorem of algebra, $p_A(t)=\prod_{i=1}^n(t-\lambda_i)$ over $\mathbb{C}$, and each $\lambda_i$ is an eigenvalue of the Hermitian $A$, hence real by (1). So $p_A$ is a product of real linear factors. If $A$ is real and $\lambda$ is one of its (real) roots, then $\det(A-\lambda I)=0$ in $M_n(\mathbb{R})$, so a real eigenvector exists. For a self-adjoint operator on a real space apply this to its matrix in an orthonormal basis, which is symmetric because $[T]^T=[T^*]=[T]$. ∎

📐 Theorem 10.22 — Basic properties of normal operators
Let $T\in \mathcal{L}(V)$.
  1. $T$ is normal if and only if $\|Tx\|=\|T^*x\|$ for every $x\in V$.
  2. If $T$ is normal, $\ker T=\ker T^*$, and $T-\lambda I$ is normal for every scalar $\lambda$.
  3. If $T$ is normal, then $Tx=\lambda x\iff T^*x=\bar\lambda x$: $T$ and $T^*$ have the same eigenvectors, with conjugate eigenvalues.
  4. If $T$ is normal, eigenvectors for distinct eigenvalues are orthogonal.
  5. (Any $T$.) If a subspace $W$ is $T$-invariant, then $W^\perp$ is $T^*$-invariant. If $T$ is normal and $E_\lambda$ is an eigenspace, then $E_\lambda^\perp$ is invariant under both $T$ and $T^*$.

Proof. (1) Put $N=T^*T-TT^*$. Then $\|Tx\|^2-\|T^*x\|^2$ equals $\langle T^*Tx,x\rangle-\langle TT^*x,x\rangle=\langle Nx,x\rangle$. If $T$ is normal, $N=0$. Conversely, $N$ is self-adjoint, so $\langle Nx,x\rangle=0$ for all $x$ forces $N=0$ by Theorem 10.20 (parts 1 and 3 — this works over $\mathbb{R}$ and $\mathbb{C}$). (2) The first claim is (1). Next $(T-\lambda I)^*=T^*-\bar\lambda I$, and both products $(T-\lambda I)(T^*-\bar\lambda I)$ and $(T^*-\bar\lambda I)(T-\lambda I)$ expand to $TT^*-\bar\lambda T-\lambda T^*+|\lambda|^2I$ (using $TT^*=T^*T$). (3) Apply $\ker S=\ker S^*$ to the normal operator $S=T-\lambda I$. (4) If $Tx=\lambda x$ and $Ty=\mu y$, then by (3) $T^*y=\bar\mu y$ and $\lambda\langle x,y\rangle=\langle Tx,y\rangle=\langle x,T^*y\rangle=\langle x,\bar\mu y\rangle=\mu\langle x,y\rangle$. (5) For $u\in W^\perp$ and $w\in W$: $\langle T^*u,w\rangle=\langle u,Tw\rangle=0$ since $Tw\in W$. For normal $T$, $E_\lambda$ is invariant under $T$ and, by (3), under $T^*$ ($T^*x=\bar\lambda x$); applying the first statement to $T$ and to $T^*$ shows that $E_\lambda^\perp$ is invariant under $T^*$ and under $T^{**}=T$. ∎

✏️ Example 1 — Sorting matrices into classes
Classify each matrix as Hermitian, skew-Hermitian, unitary, normal (or none), and find its eigenvalues: $$A=\begin{pmatrix}2&1+i\\1-i&3\end{pmatrix},\quad B=\begin{pmatrix}1&-1\\1&1\end{pmatrix},\quad C=\begin{pmatrix}1&1\\0&1\end{pmatrix},$$ $$U=\frac1{\sqrt2}\begin{pmatrix}1&i\\i&1\end{pmatrix},\qquad K=\begin{pmatrix}i&2\\-2&0\end{pmatrix}.$$
$A$ $A^*=\begin{pmatrix}2&\overline{1-i}\\\overline{1+i}&3\end{pmatrix}=A$: Hermitian. $\operatorname{tr}A=5$, $\det A=6-(1+i)(1-i)=6-2=4$, so $p_A(t)=t^2-5t+4=(t-1)(t-4)$: eigenvalues $1,4$ — real, as Theorem 10.21 predicts.
$B$ $B=I+\begin{pmatrix}0&-1\\1&0\end{pmatrix}$ is "scalar plus skew-symmetric", hence normal: $B^TB=BB^T=2I$. It is not symmetric, not skew-symmetric, and not orthogonal ($B^TB=2I\ne I$). Eigenvalues: $p_B(t)=t^2-2t+2$, so $1\pm i$, with $|1\pm i|=\sqrt2$.
$C$ $C^TC=\begin{pmatrix}1&1\\1&2\end{pmatrix}$ but $CC^T=\begin{pmatrix}2&1\\1&1\end{pmatrix}$: not normal. Its only eigenvalue is $1$ — real and of modulus $1$ — yet $C$ is neither symmetric nor orthogonal. Eigenvalues alone never decide the class of a non-normal matrix.
$U$ $U^*=\tfrac1{\sqrt2}\begin{pmatrix}1&-i\\-i&1\end{pmatrix}$ and $U^*U=\tfrac12\begin{pmatrix}1+1&i-i\\-i+i&1+1\end{pmatrix}=I$: unitary (so normal). Eigenvalues: $p_U(t)=t^2-\sqrt2\,t+1$, so $t=\tfrac{1\pm i}{\sqrt2}=e^{\pm i\pi/4}$, both of modulus $1$.
$K$ $K^*=\begin{pmatrix}-i&-2\\2&0\end{pmatrix}=-K$: skew-Hermitian. $p_K(t)=t^2-it+4$, so $t=\tfrac{i\pm\sqrt{-1-16}}{2}=\tfrac{1\pm\sqrt{17}}{2}\,i$ — purely imaginary. ✓

The Spectral Theorem

📐 Theorem 10.23 — The complex spectral theorem
Let $V$ be a finite-dimensional complex inner product space and $T\in \mathcal{L}(V)$. The following are equivalent:
  1. $T$ is normal;
  2. $V$ has an orthonormal basis consisting of eigenvectors of $T$;
  3. (matrix form) $A\in M_n(\mathbb{C})$ is unitarily diagonalisable: $A=UDU^*$ with $U$ unitary and $D$ diagonal.

Proof. (2)⇒(1): in an orthonormal eigenbasis $[T]=D$ is diagonal, so $[T^*]=D^*=\overline D$ is diagonal too, and diagonal matrices commute: $[T^*T]=\overline DD=D\overline D=[TT^*]$. (2)⇔(3): the columns of $U$ are the orthonormal eigenvectors, and $U^{-1}=U^*$.

(1)⇒(2). By Schur's unitary triangularisation (§9.2) there is an orthonormal basis $e_1,\dots,e_n$ in which $R=[T]=(r_{ij})$ is upper triangular. Because the basis is orthonormal, $[T^*]=R^*$ and $R^*R=RR^*$: the matrix $R$ is normal. We show that a normal upper triangular matrix is diagonal. Compare the $(1,1)$ entries of $R^*R$ and $RR^*$, i.e. $\|Re_1\|^2$ and $\|R^*e_1\|^2$ (Theorem 10.22(1)): $$\|Re_1\|^2=\sum_i|r_{i1}|^2=|r_{11}|^2,\qquad\|R^*e_1\|^2=\sum_j|r_{1j}|^2=|r_{11}|^2+\sum_{j\ge2}|r_{1j}|^2,$$ since the first column of $R$ has only one nonzero entry. Equality forces $r_{12}=\cdots=r_{1n}=0$. Then $R=\begin{pmatrix}r_{11}&0\\0&R'\end{pmatrix}$ with $R'$ upper triangular, and $R^*R=RR^*$ gives $R'^*R'=R'R'^*$; by induction on $n$, $R'$ is diagonal. So $R$ is diagonal and $e_1,\dots,e_n$ are eigenvectors. ∎

A second proof avoids Schur: take any eigenvector $e_1$ (one exists over $\mathbb{C}$), normalise it; by Theorem 10.22(5), $W=\{e_1\}^\perp$ is invariant under $T$ and $T^*$, the restriction $T|_W$ is normal (its adjoint is $T^*|_W$), and induction on $\dim V$ gives an orthonormal eigenbasis of $W$.

📐 Theorem 10.24 — The real spectral theorem (principal axis theorem)
Let $V$ be a finite-dimensional real inner product space and $T\in \mathcal{L}(V)$. Then $T$ is self-adjoint if and only if $V$ has an orthonormal basis of eigenvectors of $T$. In matrix form: $A\in M_n(\mathbb{R})$ is symmetric if and only if $$A=QDQ^T$$ with $Q$ real orthogonal and $D$ real diagonal ($A$ is orthogonally diagonalisable).

Proof. (⇐) In an orthonormal eigenbasis $[T]=D$ is diagonal, hence symmetric, so $[T^*]=D^T=D=[T]$. For matrices, $(QDQ^T)^T=QD^TQ^T=QDQ^T$.

(⇒) Induction on $n=\dim V$; $n=1$ is clear. By Theorem 10.21(4), the self-adjoint $T$ has a real eigenvalue $\lambda_1$ and a unit eigenvector $e_1$. The line $W=\operatorname{span}(e_1)$ is $T$-invariant, so $W^\perp$ is $T^*$-invariant (Theorem 10.22(5)), i.e. $T$-invariant. The restriction $S=T|_{W^\perp}$ is self-adjoint on the $(n-1)$-dimensional inner product space $W^\perp$, because $\langle Sx,y\rangle=\langle Tx,y\rangle=\langle x,Ty\rangle=\langle x,Sy\rangle$ for $x,y\in W^\perp$. By induction $W^\perp$ has an orthonormal basis $e_2,\dots,e_n$ of eigenvectors of $S$, hence of $T$, and $e_1,e_2,\dots,e_n$ is the required basis of $V=W\oplus W^\perp$. ∎

Over $\mathbb{R}$ "normal" is not enough: the rotation $\begin{pmatrix}0&-1\\1&0\end{pmatrix}$ is orthogonal, hence normal, but has no real eigenvalue at all. The step that fails is the very first one — the existence of a real eigenvector — and Theorem 10.21(4) is exactly what supplies it for symmetric matrices.

📐 Theorem 10.25 — Spectral decomposition and functional calculus
Let $T$ be normal (if $\mathbb{F}=\mathbb{C}$) or self-adjoint (if $\mathbb{F}=\mathbb{R}$), with distinct eigenvalues $\lambda_1,\dots,\lambda_k$, eigenspaces $E_j=\ker(T-\lambda_jI)$, and $P_j$ the orthogonal projection onto $E_j$. Then
  1. $V=E_1\oplus\cdots\oplus E_k$, the summands being mutually orthogonal;
  2. $P_j^2=P_j=P_j^*$, $P_iP_j=0$ for $i\ne j$, and $P_1+\cdots+P_k=I$;
  3. $T=\sum_j\lambda_jP_j$, $T^*=\sum_j\bar\lambda_jP_j$, and $f(T)=\sum_jf(\lambda_j)P_j$ for every polynomial $f$;
  4. $P_j=\ell_j(T)$, where $\ell_j(t)=\prod_{i\ne j}\dfrac{t-\lambda_i}{\lambda_j-\lambda_i}$ — each projection is a polynomial in $T$;
  5. conversely, if $T=\sum_j\mu_jQ_j$ with distinct scalars $\mu_j$ and nonzero orthogonal projections $Q_j$ satisfying $Q_iQ_j=0$ ($i\ne j$) and $\sum Q_j=I$, then the $\mu_j$ are exactly the eigenvalues of $T$ and $Q_j$ is the projection onto the eigenspace of $\mu_j$.
For any function $f$ defined on $\{\lambda_1,\dots,\lambda_k\}$ we define $f(T)=\sum_jf(\lambda_j)P_j$ — this gives $e^T$, $|T|$, $\sqrt T$ (when all $\lambda_j\ge0$), and agrees with the polynomial value when $f$ is a polynomial.

Proof. (1) By Theorems 10.23 and 10.24 the eigenvectors span $V$; eigenspaces for different eigenvalues are orthogonal (Theorem 10.22(4)). So each $x$ is uniquely $x=x_1+\cdots+x_k$ with $x_j\in E_j$, and these components are mutually orthogonal. (2) $P_jx=x_j$; the identities follow at once, and orthogonal projections are self-adjoint (§10.1). (3) $Tx=\sum_j\lambda_jx_j=\sum_j\lambda_jP_jx$; by induction $T^mx=\sum_j\lambda_j^mx_j$, hence $f(T)x=\sum_jf(\lambda_j)x_j$. For the adjoint, $(\sum\lambda_jP_j)^*=\sum\bar\lambda_jP_j^*=\sum\bar\lambda_jP_j$. (4) $\ell_j(\lambda_i)=\delta_{ij}$, so by (3) $\ell_j(T)=\sum_i\delta_{ij}P_i=P_j$. (5) For $x\in\operatorname{im}Q_j$, $Tx=\sum_i\mu_iQ_iQ_jx=\mu_jx$; so the $\operatorname{im}Q_j$ are eigenspaces, they are orthogonal and span $V$, and an eigenvector for any other value $\mu$ would be orthogonal to all of them, hence $0$. ∎

📐 Theorem 10.26 — Reading the class of a normal operator from its spectrum
Let $T$ be a normal operator on a complex inner product space, with spectrum $\sigma(T)$. Then
  1. $T$ is self-adjoint $\iff\sigma(T)\subseteq\mathbb{R}$;
  2. $T$ is skew-adjoint $\iff\sigma(T)\subseteq i\mathbb{R}$;
  3. $T$ is unitary $\iff\sigma(T)\subseteq\{z:|z|=1\}$;
  4. $T$ is an orthogonal projection $\iff\sigma(T)\subseteq\{0,1\}$;
  5. $T$ is nilpotent $\iff T=0$; more generally $\operatorname{rank}T^m=\operatorname{rank}T$ for all $m\ge1$;
  6. the operator norm $\|T\|=\max_{\|x\|=1}\|Tx\|$ equals the spectral radius $\rho(T)=\max_j|\lambda_j|$.

Proof. Write $T=\sum\lambda_jP_j$ and $T^*=\sum\bar\lambda_jP_j$ (Theorem 10.25). By the uniqueness in Theorem 10.25(5), $T^*=T$ iff $\bar\lambda_j=\lambda_j$ for every $j$, and $T^*=-T$ iff $\bar\lambda_j=-\lambda_j$. Next $T^*T=\sum|\lambda_j|^2P_j$, which equals $I=\sum P_j$ iff every $|\lambda_j|=1$. Also $T^2=\sum\lambda_j^2P_j$ equals $T$ iff $\lambda_j^2=\lambda_j$; such a $T$ is then self-adjoint by (1), and a self-adjoint idempotent is an orthogonal projection. For (5), $T^m=\sum\lambda_j^mP_j$ has the same nonzero eigenvalues (and the same eigenspaces) as $T$, so the same rank; if $T^m=0$ every $\lambda_j=0$ and $T=0$. For (6), by Pythagoras $\|Tx\|^2=\sum_j|\lambda_j|^2\|x_j\|^2$ $\le\rho^2\sum_j\|x_j\|^2=\rho^2\|x\|^2$, with equality for an eigenvector of an eigenvalue of largest modulus. ∎

Every implication "⇐" fails without normality: $C=\begin{pmatrix}1&1\\0&1\end{pmatrix}$ has $\sigma(C)=\{1\}$, which lies in $\mathbb{R}$, on the unit circle and in $\{0,1\}$, yet $C$ is not Hermitian, unitary or idempotent, and $\|C\|=\tfrac{1+\sqrt5}{2}\gt1=\rho(C)$ (Exercise III.4). Likewise $\begin{pmatrix}0&1\\0&0\end{pmatrix}$ is nilpotent but not $0$.

📐 Theorem 10.27 — Real normal and real orthogonal matrices
If $A\in M_n(\mathbb{R})$ is normal, there is a real orthogonal $Q$ such that $Q^TAQ$ is block diagonal with $1\times1$ blocks $(\lambda)$ ($\lambda$ a real eigenvalue) and $2\times2$ blocks $\begin{pmatrix}a&-b\\b&a\end{pmatrix}$ with $b\ne0$ (one for each conjugate pair $a\pm ib$ of non-real eigenvalues). Consequently:
  1. a real orthogonal matrix is orthogonally similar to $\operatorname{diag}\big(\pm1,\dots,\pm1,R_{\theta_1},\dots,R_{\theta_s}\big)$, where $R_\theta=\begin{pmatrix}\cos\theta&-\sin\theta\\\sin\theta&\cos\theta\end{pmatrix}$;
  2. every $3\times3$ real orthogonal matrix with determinant $1$ is a rotation through some angle $\theta$ about an axis $\mathbb{R}u$ with $Qu=u$, and $\operatorname{tr}Q=1+2\cos\theta$;
  3. a real skew-symmetric matrix is orthogonally similar to $\operatorname{diag}\!\left(0,\dots,0,\begin{pmatrix}0&-b_1\\b_1&0\end{pmatrix},\dots\right)$; its rank is even and its nonzero eigenvalues are $\pm ib_j$.

Proof (sketch with all key steps). Real eigenvalues are handled as in Theorem 10.24: a unit eigenvector spans a line whose orthogonal complement is invariant under $A$ and $A^T$ (Theorem 10.22(5)). For a non-real eigenvalue $\lambda=a+ib$ ($b\ne0$) choose an eigenvector $z=x+iy\in\mathbb{C}^n$ with $x,y\in\mathbb{R}^n$. Since $A$ is real, $\bar z$ is an eigenvector for $\bar\lambda\ne\lambda$, so $\langle z,\bar z\rangle=0$ by Theorem 10.22(4). Expanding, $$\langle x+iy,\,x-iy\rangle=\|x\|^2-\|y\|^2+2i\langle x,y\rangle=0,$$ so $\|x\|=\|y\|\ne0$ and $x\perp y$. Comparing real and imaginary parts of $A(x+iy)=(a+ib)(x+iy)$ gives $Ax=ax-by$ and $Ay=bx+ay$; so in the orthonormal pair $\big(\tfrac{x}{\|x\|},-\tfrac{y}{\|y\|}\big)$ the restriction of $A$ has matrix $\begin{pmatrix}a&-b\\b&a\end{pmatrix}$. By Theorem 10.22(3), $A^Tz=\bar\lambda z$, so the plane $\operatorname{span}(x,y)$ is invariant under $A^T$ as well; its orthogonal complement is invariant under $A$ and $A^T$, and induction finishes. For (1), orthogonal blocks have $|\lambda|=1$, i.e. $\lambda=\pm1$ or $a^2+b^2=1$. For (2), with $n=3$ the normal form is either $\operatorname{diag}(\varepsilon,R_\theta)$ with $\varepsilon=\pm1$ — and $\det=\varepsilon\det R_\theta=\varepsilon$ forces $\varepsilon=1$ — or $\operatorname{diag}(\varepsilon_1,\varepsilon_2,\varepsilon_3)$ with product $1$, i.e. $I$ or (up to order) $\operatorname{diag}(1,-1,-1)=\operatorname{diag}(1,R_\pi)$. In every case it is $\operatorname{diag}(1,R_\theta)$, whose trace is $1+2\cos\theta$. For (3), skew-symmetric blocks have $a=0$ and real eigenvalues $0$. ∎

🧭 Problem-solving strategy — orthogonally (unitarily) diagonalising a matrix
  1. Check the hypothesis. Real: is $A^T=A$? Complex: is $A^*A=AA^*$? (Hermitian, skew-Hermitian, unitary and circulant matrices are normal; for anything else compute both products.)
  2. Eigenvalues. Use $\operatorname{tr}$, $\det$, obvious eigenvectors (row sums, $(1,1,\dots,1)$), and the fact that the spectrum must be real / imaginary / unimodular for the special classes.
  3. Eigenvectors. Eigenvectors for different eigenvalues are automatically orthogonal — only normalise them. Inside an eigenspace of dimension $\ge2$, run Gram–Schmidt (§10.1); a non-orthogonal basis of one eigenspace is the only place where work is needed.
  4. Assemble and check. $U$ (or $Q$) has the orthonormal eigenvectors as columns; check $U^*AU=D$ on one column. For functions of $A$, use $f(A)=\sum f(\lambda_j)P_j$ with $P_j=\sum_{\text{basis of }E_j}u\,u^*$.
✏️ Example 2 — Orthogonal diagonalisation, powers and a square root
Let $A=\begin{pmatrix}2&1&1\\1&2&1\\1&1&2\end{pmatrix}$. (a) Find an orthogonal $Q$ with $Q^TAQ$ diagonal. (b) Find the spectral decomposition of $A$ and a closed formula for $A^n$. (c) Find the positive square root of $A$.
Eigenvalues $A=I+J$, where $J$ is the all-ones matrix. $J$ has eigenvalue $3$ on $(1,1,1)^T$ and $0$ on the plane $x_1+x_2+x_3=0$. So $A$ has eigenvalue $4$ (once) and $1$ (twice). Check: $\operatorname{tr}A=6=4+1+1$.
Eigenvectors $E_4$ is spanned by $q_1=\tfrac1{\sqrt3}(1,1,1)^T$. $E_1=\{x_1+x_2+x_3=0\}$ has the non-orthogonal basis $(1,-1,0)^T,(1,0,-1)^T$; Gram–Schmidt gives $q_2=\tfrac1{\sqrt2}(1,-1,0)^T$ and then $(1,0,-1)-\tfrac12(1,-1,0)=\tfrac12(1,1,-2)$, so $q_3=\tfrac1{\sqrt6}(1,1,-2)^T$. Both are automatically orthogonal to $q_1$.
(a) $Q=[\,q_1\ q_2\ q_3\,]=\begin{pmatrix}1/\sqrt3&1/\sqrt2&1/\sqrt6\\1/\sqrt3&-1/\sqrt2&1/\sqrt6\\1/\sqrt3&0&-2/\sqrt6\end{pmatrix}$ and $Q^TAQ=\operatorname{diag}(4,1,1)$.
(b) $P_4=q_1q_1^T=\tfrac13J$ and $P_1=I-P_4=I-\tfrac13J$. So $A=4\cdot\tfrac13J+1\cdot\left(I-\tfrac13J\right)$ and, by Theorem 10.25, $$A^n=\frac{4^n}{3}J+\Big(I-\frac13J\Big)=I+\frac{4^n-1}{3}J .$$ Check $n=1$: $I+J=A$ ✓; $n=2$: $A^2=I+5J$, whose diagonal entries are $6=2^2+1^2+1^2$ ✓.
(c) $\sqrt A=\sqrt4\,P_4+\sqrt1\,P_1=\tfrac23J+I-\tfrac13J=I+\tfrac13J=\tfrac13\begin{pmatrix}4&1&1\\1&4&1\\1&1&4\end{pmatrix}$. Check: $(I+\tfrac13J)^2=I+\tfrac23J+\tfrac19J^2=I+\tfrac23J+\tfrac13J=I+J$ (using $J^2=3J$). ✓
✏️ Example 3 — Circulant matrices are normal
Let $P=\begin{pmatrix}0&1&0\\0&0&1\\1&0&0\end{pmatrix}$ and $C=2I+P=\begin{pmatrix}2&1&0\\0&2&1\\1&0&2\end{pmatrix}$. Show that $C$ is normal but not symmetric, find an orthonormal eigenbasis of $\mathbb{C}^3$, and find the eigenvalues of $C$ and $\|C\|$.
Normal $P$ is a permutation matrix, hence orthogonal: $P^TP=I$, and $P^T=P^{-1}=P^2$. Every polynomial in a normal matrix is normal, so $C=2I+P$ is normal; explicitly $C^TC=(2I+P^2)(2I+P)=4I+2P+2P^2+I=CC^T$. $C$ is visibly not symmetric.
Eigenvectors Let $\omega=e^{2\pi i/3}$. For $f_k=\tfrac1{\sqrt3}(1,\omega^k,\omega^{2k})^T$ ($k=0,1,2$), $Pf_k=\tfrac1{\sqrt3}(\omega^k,\omega^{2k},1)^T=\omega^kf_k$. These are eigenvectors of $P$ for three distinct eigenvalues of a unitary matrix, so they are orthonormal (Theorem 10.21(3)); directly, $\langle f_j,f_k\rangle=\tfrac13\sum_m\omega^{m(j-k)}=\delta_{jk}$.
Eigenvalues $Cf_k=(2+\omega^k)f_k$: the eigenvalues are $3$, $2+\omega=\tfrac32+\tfrac{\sqrt3}2i$ and $2+\omega^2=\tfrac32-\tfrac{\sqrt3}2i$. Check: their sum is $6=\operatorname{tr}C$ and their product is $3\cdot\left(\tfrac94+\tfrac34\right)=9=\det C$ ✓.
Norm $C$ is normal, so $\|C\|=\rho(C)=\max(3,\sqrt3,\sqrt3)=3$ (Theorem 10.26(6)), attained at $f_0$. The same Fourier vectors diagonalise every $3\times3$ circulant $aI+bP+cP^2$ simultaneously, with eigenvalues $a+b\omega^k+c\omega^{2k}$.

Positive Operators, Square Roots, Polar and Singular Value Decompositions

📘 Definition 10.10 — Positive semidefinite and positive definite operators

An operator $T\in \mathcal{L}(V)$ is positive semidefinite, written $T\succeq0$, if $T$ is self-adjoint and $\langle Tx,x\rangle\ge0$ for all $x\in V$; it is positive definite, $T\succ0$, if moreover $\langle Tx,x\rangle\gt0$ for every $x\ne0$. A matrix $A\in M_n(\mathbb{F})$ is positive (semi)definite if $A=A^*$ and $x^*Ax\gt0$ for all $x\ne0$ ($\ge0$ for all $x$). $T$ is negative (semi)definite if $-T$ is positive (semi)definite, and an indefinite self-adjoint operator is one that is neither positive nor negative semidefinite. We write $S\succeq T$ when $S-T\succeq0$.

Over $\mathbb{C}$ the self-adjointness requirement is automatic (Theorem 10.20(2)); over $\mathbb{R}$ it is part of the definition: $\begin{pmatrix}1&1\\-1&1\end{pmatrix}$ satisfies $x^TAx=x_1^2+x_2^2\gt0$ but is not symmetric, and is not called positive definite. Quadratic forms (§10.3) are the matrix side of the same notion.

📐 Theorem 10.28 — Characterisations of positive (semi)definite operators
For a self-adjoint $T\in \mathcal{L}(V)$ the following are equivalent:
  1. $T\succeq0$;
  2. every eigenvalue of $T$ is $\ge0$;
  3. $T=S^*S$ for some $S\in \mathcal{L}(V)$ (for matrices: $A=B^*B$, i.e. $A$ is the Gram matrix $a_{ij}=\langle b_j,b_i\rangle$ of the columns $b_j$ of $B$);
  4. $T=R^2$ for some self-adjoint $R\succeq0$.
The same holds for $T\succ0$ with "$\gt0$" in (2) and $S$, $R$ invertible in (3), (4); and $T\succ0$ if and only if $(x,y)\mapsto\langle Tx,y\rangle$ is again an inner product on $V$. If $A\in M_n(\mathbb{F})$ is positive definite, then $a_{ii}\gt0$, $\operatorname{tr}A\gt0$, $\det A\gt0$, every principal submatrix of $A$ is positive definite, $|a_{ij}|^2\lt a_{ii}a_{jj}$ for $i\ne j$, and $A^{-1}\succ0$.

Proof. (1)⇒(2): if $Tx=\lambda x$ with $x\ne0$, then $\lambda\|x\|^2=\langle Tx,x\rangle\ge0$. (2)⇒(4): write $T=\sum\lambda_jP_j$ (Theorem 10.25) with all $\lambda_j\ge0$ and put $R=\sum\sqrt{\lambda_j}\,P_j$. Then $R^*=R$, $R^2=\sum\lambda_jP_j=T$, and $\langle Rx,x\rangle=\sum_j\sqrt{\lambda_j}\,\|x_j\|^2\ge0$. (4)⇒(3): take $S=R$. (3)⇒(1): $(S^*S)^*=S^*S$ and $\langle S^*Sx,x\rangle=\langle Sx,Sx\rangle=\|Sx\|^2\ge0$. In the definite case each inequality becomes strict, and $\|Sx\|\gt0$ for all $x\ne0$ means exactly that $S$ is injective, i.e. invertible. The form $\langle Tx,y\rangle$ is linear in $x$, conjugate symmetric because $T=T^*$, and positive definite iff $T\succ0$. For the matrix facts: $a_{ii}=e_i^*Ae_i\gt0$; $\operatorname{tr}A$ and $\det A$ are the sum and product of the positive eigenvalues; for an index set $I$ and $x$ supported on $I$, $x^*Ax=x_I^*A_Ix_I$, so the principal submatrix $A_I$ is positive definite, and for $I=\{i,j\}$ its determinant $a_{ii}a_{jj}-|a_{ij}|^2$ is positive; finally $A^{-1}$ is Hermitian with eigenvalues $1/\lambda_j\gt0$. ∎

📐 Theorem 10.29 — The unique positive square root
For every $T\succeq0$ there is exactly one $R\succeq0$ with $R^2=T$; it is denoted $\sqrt T$ or $T^{1/2}$. If $T=\sum\lambda_jP_j$, then $\sqrt T=\sum\sqrt{\lambda_j}\,P_j$; $\sqrt T$ is a polynomial in $T$, so it commutes with every operator that commutes with $T$; and $\sqrt T\succ0$ iff $T\succ0$.

Proof. Existence is (2)⇒(4) above, and $\sqrt T=\sum\sqrt{\lambda_j}\,\ell_j(T)$ is a polynomial in $T$ by Theorem 10.25(4). Uniqueness: let $R\succeq0$ with $R^2=T$. As $R$ is self-adjoint, $R=\sum_i\mu_iQ_i$ with distinct eigenvalues $\mu_i\ge0$ and orthogonal eigenprojections $Q_i$, and then $T=R^2=\sum_i\mu_i^2Q_i$. Because $\mu\mapsto\mu^2$ is injective on $[0,\infty)$, the numbers $\mu_i^2$ are distinct, so by Theorem 10.25(5) they are the eigenvalues $\lambda_j$ of $T$ and the $Q_i$ are its eigenprojections $P_j$. Thus $\mu_i=\sqrt{\lambda_j}$ and $R=\sum\sqrt{\lambda_j}P_j$ — there was no choice. The last statement holds because $\sqrt{\lambda}\gt0\iff\lambda\gt0$. ∎

Without positivity, square roots are far from unique: every reflection $\begin{pmatrix}\cos\theta&\sin\theta\\\sin\theta&-\cos\theta\end{pmatrix}$ squares to $I$, so $I_2$ has infinitely many square roots, and $\begin{pmatrix}0&1\\0&0\end{pmatrix}$ has none at all (a square root $X$ would satisfy $X^4=0$, hence $X^2=0$ for a $2\times2$ matrix).

✏️ Example 4 — All square roots of a positive definite matrix
Let $A=\begin{pmatrix}5&4\\4&5\end{pmatrix}$. Show that $A\succ0$, find $\sqrt A$, and find all real matrices $X$ with $X^2=A$.
Spectrum $A(1,1)^T=9(1,1)^T$ and $A(1,-1)^T=(1,-1)^T$: eigenvalues $9,1$, both positive, so $A\succ0$. The eigenprojections are $P_9=\tfrac12\begin{pmatrix}1&1\\1&1\end{pmatrix}$ and $P_1=\tfrac12\begin{pmatrix}1&-1\\-1&1\end{pmatrix}$.
$\sqrt A$ $\sqrt A=3P_9+1\cdot P_1=\begin{pmatrix}2&1\\1&2\end{pmatrix}$. Check: $\begin{pmatrix}2&1\\1&2\end{pmatrix}^2=\begin{pmatrix}5&4\\4&5\end{pmatrix}$ ✓.
Every root commutes with $A$ If $X^2=A$ then $XA=X^3=AX$. So $X$ maps each eigenspace of $A$ into itself: if $Av=9v$ then $A(Xv)=X(Av)=9Xv$. Both eigenspaces are lines, so $(1,1)^T$ and $(1,-1)^T$ are eigenvectors of $X$, say with eigenvalues $\alpha,\beta$, and $\alpha^2=9$, $\beta^2=1$.
Answer Exactly four real square roots, $X=\pm3P_9\pm P_1$: $$\begin{pmatrix}2&1\\1&2\end{pmatrix},\ \begin{pmatrix}1&2\\2&1\end{pmatrix},\ \begin{pmatrix}-1&-2\\-2&-1\end{pmatrix},\ \begin{pmatrix}-2&-1\\-1&-2\end{pmatrix}.$$ All are symmetric, but only the first is positive semidefinite — exactly as Theorem 10.29 says. (Distinct eigenvalues made the count finite; for $A=I$ there are infinitely many.)
📘 Definition 10.11 — Singular values
For $A\in M_{m\times n}(\mathbb{F})$ the matrix $A^*A\in M_n(\mathbb{F})$ is positive semidefinite (Theorem 10.28(3)). Its eigenvalues are $\sigma_1^2\ge\sigma_2^2\ge\cdots\ge\sigma_n^2\ge0$, and the numbers $\sigma_i\ge0$ are the singular values of $A$. The number of nonzero singular values equals $\operatorname{rank}A$, because $\ker A^*A=\ker A$ (if $A^*Ax=0$ then $\|Ax\|^2=\langle A^*Ax,x\rangle=0$).
📐 Theorem 10.30 — The singular value decomposition (SVD)
Let $A\in M_{m\times n}(\mathbb{F})$ have rank $r$ and singular values $\sigma_1\ge\cdots\ge\sigma_r\gt0$. There are unitary (real orthogonal if $\mathbb{F}=\mathbb{R}$) matrices $U\in M_m$, $V\in M_n$ such that $$A=U\Sigma V^*,\qquad\Sigma=\begin{pmatrix}\operatorname{diag}(\sigma_1,\dots,\sigma_r)&0\\0&0\end{pmatrix}\in M_{m\times n}.$$ Equivalently $A=\sum_{i=1}^r\sigma_iu_iv_i^*$ with orthonormal columns $u_i$ of $U$ and $v_i$ of $V$. Moreover:
  1. $Av_i=\sigma_iu_i$ and $A^*u_i=\sigma_iv_i$; $v_1,\dots,v_r$ is an orthonormal basis of $\operatorname{im}A^*$, $v_{r+1},\dots,v_n$ of $\ker A$, $u_1,\dots,u_r$ of $\operatorname{im}A$, and $u_{r+1},\dots,u_m$ of $\ker A^*$;
  2. $\|A\|=\max_{\|x\|=1}\|Ax\|=\sigma_1$ and $\|A\|_F^2=\sum_{i,j}|a_{ij}|^2=\operatorname{tr}(A^*A)=\sum_i\sigma_i^2$;
  3. the nonzero eigenvalues of $AA^*$ are also $\sigma_1^2,\dots,\sigma_r^2$; if $m=n$, $|\det A|=\sigma_1\sigma_2\cdots\sigma_n$;
  4. if $A$ is normal with eigenvalues $\lambda_i$, its singular values are the $|\lambda_i|$;
  5. (Eckart–Young, statement) for $k\lt r$, $A_k=\sum_{i\le k}\sigma_iu_iv_i^*$ is a closest matrix of rank $\le k$ to $A$: $\min_{\operatorname{rank}B\le k}\|A-B\|=\|A-A_k\|=\sigma_{k+1}$.

Proof. By the spectral theorem applied to the positive semidefinite $A^*A$ there is an orthonormal basis $v_1,\dots,v_n$ of $\mathbb{F}^n$ with $A^*Av_i=\sigma_i^2v_i$, ordered so that $\sigma_1\ge\cdots\ge\sigma_n$; exactly $r$ of the $\sigma_i$ are positive (Definition 10.11). Then $$\langle Av_i,Av_j\rangle=\langle A^*Av_i,v_j\rangle=\sigma_i^2\langle v_i,v_j\rangle=\sigma_i^2\delta_{ij},$$ so $u_i=Av_i/\sigma_i$ ($i\le r$) are orthonormal, while $Av_i=0$ for $i\gt r$. Extend $u_1,\dots,u_r$ to an orthonormal basis $u_1,\dots,u_m$ of $\mathbb{F}^m$ (Gram–Schmidt, §10.1). Then $AV=U\Sigma$ column by column, i.e. $A=U\Sigma V^*$. (1) $A^*u_i=A^*Av_i/\sigma_i=\sigma_iv_i$; the subspace statements follow since $\operatorname{im}A=\operatorname{span}(Av_1,\dots,Av_n)$, which is $\operatorname{span}(u_1,\dots,u_r)$, while $\ker A=\operatorname{span}(v_{r+1},\dots,v_n)$, and the other two are orthogonal complements ($\ker A^*=(\operatorname{im}A)^\perp$, $\operatorname{im}A^*=(\ker A)^\perp$). (2) For $x=\sum c_iv_i$, $\|Ax\|^2=\sum_i\sigma_i^2|c_i|^2\le\sigma_1^2\|x\|^2$, with equality at $x=v_1$; and $\operatorname{tr}(A^*A)$ is the sum of its eigenvalues. (3) $AA^*=U\Sigma\Sigma^TU^*$ has eigenvalues $\sigma_i^2$ and zeros; $|\det A|=|\det\Sigma|=\prod\sigma_i$ because $|\det U|=|\det V|=1$. (4) If $A=WDW^*$ with $W$ unitary then $A^*A=W\overline DDW^*=W|D|^2W^*$. (5) is quoted without proof. ∎

📐 Theorem 10.31 — The polar decomposition
Every $A\in M_n(\mathbb{F})$ can be written $A=WP$ with $W$ unitary (orthogonal if $\mathbb{F}=\mathbb{R}$) and $P=\sqrt{A^*A}\succeq0$; $P$ is always unique, and $W$ is unique when $A$ is invertible ($W=AP^{-1}$). Similarly $A=P'W$ with $P'=\sqrt{AA^*}$. This is the matrix analogue of $z=e^{i\theta}|z|$: $P$ is "$|A|$".

Proof. From the SVD, $A=U\Sigma V^*=(UV^*)(V\Sigma V^*)$. Here $W=UV^*$ is unitary, and $P=V\Sigma V^*$ is self-adjoint with eigenvalues $\sigma_i\ge0$ and $P^2=V\Sigma^2V^*=A^*A$; by Theorem 10.29 it is the positive square root of $A^*A$. If $A=WP$ is any such factorisation then $A^*A=PW^*WP=P^2$, which forces $P=\sqrt{A^*A}$. If $A$ is invertible, so is $P$, and $W=AP^{-1}$. Likewise $A=(U\Sigma U^*)(UV^*)$ gives $A=P'W$. ∎

✏️ Example 5 — An SVD and a polar decomposition by hand
Let $A=\begin{pmatrix}3&0\\4&5\end{pmatrix}$. Find the singular values, an SVD $A=U\Sigma V^T$, $\|A\|$, and the polar decomposition $A=WP$. Compare $\|A\|$ with the eigenvalues of $A$.
$A^TA$ $A^TA=\begin{pmatrix}25&20\\20&25\end{pmatrix}$, with eigenvalues $45$ on $(1,1)^T$ and $5$ on $(1,-1)^T$. So $\sigma_1=\sqrt{45}=3\sqrt5$, $\sigma_2=\sqrt5$, and $v_1=\tfrac1{\sqrt2}(1,1)^T$, $v_2=\tfrac1{\sqrt2}(1,-1)^T$.
$u_i=Av_i/\sigma_i$ $Av_1=\tfrac1{\sqrt2}(3,9)^T$, so $u_1=\tfrac{1}{\sqrt2\cdot3\sqrt5}(3,9)^T=\tfrac1{\sqrt{10}}(1,3)^T$; $Av_2=\tfrac1{\sqrt2}(3,-1)^T$, so $u_2=\tfrac1{\sqrt{10}}(3,-1)^T$. Indeed $u_1\perp u_2$.
SVD $$A=\underbrace{\tfrac1{\sqrt{10}}\begin{pmatrix}1&3\\3&-1\end{pmatrix}}_{U}\begin{pmatrix}3\sqrt5&0\\0&\sqrt5\end{pmatrix}\underbrace{\tfrac1{\sqrt2}\begin{pmatrix}1&1\\1&-1\end{pmatrix}}_{V^T}.$$ Check: $\sigma_1\sigma_2=15=|\det A|$ ✓ and $\sigma_1^2+\sigma_2^2=50=9+16+25=\|A\|_F^2$ ✓.
Norm vs eigenvalues $\|A\|=\sigma_1=3\sqrt5\approx6.708$, while the eigenvalues of the triangular $A$ are $3$ and $5$. So $\|A\|\gt\rho(A)=5$: $A$ is not normal, and its singular values are not the moduli of its eigenvalues.
Polar $P=V\Sigma V^T=\sqrt5\begin{pmatrix}2&1\\1&2\end{pmatrix}$ (check: $P^2=5\begin{pmatrix}5&4\\4&5\end{pmatrix}=A^TA$ ✓) and $W=UV^T=\tfrac1{\sqrt5}\begin{pmatrix}2&-1\\1&2\end{pmatrix}$, a rotation through $\arctan\tfrac12$. Check: $WP=\begin{pmatrix}2&-1\\1&2\end{pmatrix}\begin{pmatrix}2&1\\1&2\end{pmatrix}=\begin{pmatrix}3&0\\4&5\end{pmatrix}$ ✓.
Interactive 1 — $A=U\Sigma V^T$: Rotate, Stretch, Rotate

The unit circle and two orthonormal vectors $v_1$ (red) and $v_2$ (blue) are carried through the three factors of the SVD: first $V^T$ turns $v_1,v_2$ onto the axes, then $\Sigma$ stretches the axes by $\sigma_1,\sigma_2$, then $U$ turns them onto $u_1,u_2$. The result is the image ellipse of $A$ (dashed), whose semi-axes are $\sigma_1u_1$ and $\sigma_2u_2$. The green lines are the real eigen-directions of $A$. For a symmetric matrix they are the axes of the ellipse (spectral theorem); for a non-normal one they are not, and $\|A\|=\sigma_1$ exceeds every $|\lambda|$.

σ₁, σ₂—
eigenvalues—
‖A‖ vs ρ(A)—
normal?—

Locating Eigenvalues: Rayleigh Quotients and Gershgorin Discs

📐 Theorem 10.32 — Rayleigh quotients and the Courant–Fischer theorem
Let $T$ be self-adjoint with eigenvalues $\lambda_1\ge\lambda_2\ge\cdots\ge\lambda_n$ (repeated by multiplicity) and orthonormal eigenvectors $e_1,\dots,e_n$. For $x\ne0$ let $R(x)=\dfrac{\langle Tx,x\rangle}{\langle x,x\rangle}$ (the Rayleigh quotient).
  1. $\lambda_n\le R(x)\le\lambda_1$ for all $x\ne0$; $R(x)=\lambda_1$ iff $x$ is an eigenvector for $\lambda_1$ (similarly for $\lambda_n$). Thus $\lambda_1=\max_{\|x\|=1}\langle Tx,x\rangle$ and $\lambda_n=\min_{\|x\|=1}\langle Tx,x\rangle$.
  2. The set $\{\langle Tx,x\rangle:\|x\|=1\}$ (the numerical range) is the whole interval $[\lambda_n,\lambda_1]$. In particular every diagonal entry $a_{ii}=e_i^*Ae_i$ of a Hermitian matrix lies in $[\lambda_{\min},\lambda_{\max}]$.
  3. (Courant–Fischer) $\displaystyle\lambda_k=\max_{\dim W=k}\ \min_{x\in W\setminus\{0\}}R(x)$ $\displaystyle{}=\min_{\dim W=n-k+1}\ \max_{x\in W\setminus\{0\}}R(x).$

Proof. (1) Write $x=\sum c_ie_i$. Then $\langle Tx,x\rangle=\sum_i\lambda_i|c_i|^2$ and $\langle x,x\rangle=\sum|c_i|^2$, so $R(x)$ is a weighted average of the $\lambda_i$ with weights $|c_i|^2/\|x\|^2$; it lies between the smallest and the largest, and equals $\lambda_1$ iff all the weight sits on indices with $\lambda_i=\lambda_1$, i.e. $x\in E_{\lambda_1}$. (2) For $x_t=\cos t\,e_1+\sin t\,e_n$ ($0\le t\le\frac\pi2$), $\|x_t\|=1$ and $\langle Tx_t,x_t\rangle=\lambda_1\cos^2t+\lambda_n\sin^2t$ runs continuously from $\lambda_1$ to $\lambda_n$. (3) For $W_0=\operatorname{span}(e_1,\dots,e_k)$, every $x\in W_0$ has weights only on $\lambda_1,\dots,\lambda_k$, so $R(x)\ge\lambda_k$, with equality at $e_k$. For any $W$ with $\dim W=k$, the subspace $Z=\operatorname{span}(e_k,\dots,e_n)$ has dimension $n-k+1$, so $\dim(W\cap Z)\ge k+(n-k+1)-n=1$; a nonzero $x\in W\cap Z$ has $R(x)\le\lambda_k$. Hence the maximum over $W$ of the minimum is exactly $\lambda_k$. The min–max form is the same argument with the roles exchanged. ∎

📐 Theorem 10.33 — Gershgorin's circle theorem
Let $A=(a_{ij})\in M_n(\mathbb{C})$, $R_i=\sum_{j\ne i}|a_{ij}|$ and $D_i=\{z\in\mathbb{C}:|z-a_{ii}|\le R_i\}$ (the Gershgorin discs).
  1. Every eigenvalue of $A$ lies in $D_1\cup D_2\cup\cdots\cup D_n$.
  2. The same holds for the column discs $\{|z-a_{jj}|\le\sum_{i\ne j}|a_{ij}|\}$, since $A^T$ has the same eigenvalues; so the eigenvalues lie in the intersection of the two unions.
  3. If the union of $k$ of the discs is disjoint from the union of the remaining $n-k$, it contains exactly $k$ eigenvalues, counted with algebraic multiplicity.

Proof. (1) Let $Ax=\lambda x$, $x\ne0$, and choose $i$ with $|x_i|=\max_j|x_j|\gt0$. Row $i$ of $Ax=\lambda x$ reads $(\lambda-a_{ii})x_i=\sum_{j\ne i}a_{ij}x_j$, so $$|\lambda-a_{ii}|\le\sum_{j\ne i}|a_{ij}|\,\frac{|x_j|}{|x_i|}\le\sum_{j\ne i}|a_{ij}|=R_i ,$$ i.e. $\lambda\in D_i$. (2) Apply (1) to $A^T$. (3) (Sketch.) Let $D=\operatorname{diag}(a_{11},\dots,a_{nn})$ and $A(t)=D+t(A-D)$ for $0\le t\le1$. The discs of $A(t)$ have the same centres and radii $tR_i$, so they stay inside the $D_i$: if $\mathcal U$ is the union of the chosen $k$ discs and $\mathcal V$ the union of the others, every eigenvalue of every $A(t)$ lies in $\mathcal U\cup\mathcal V$, and $\mathcal U\cap\mathcal V=\varnothing$. The roots of $p_{A(t)}$ depend continuously on $t$ — e.g. the number of roots inside a closed curve $\Gamma$ around $\mathcal U$ that avoids $\mathcal U\cup\mathcal V$ is $\frac1{2\pi i}\oint_\Gamma p_{A(t)}'/p_{A(t)}\,dz$, a continuous integer-valued function of $t$, hence constant. At $t=0$ the eigenvalues are the $a_{ii}$, and exactly $k$ of them (the centres of the chosen discs) lie in $\mathcal U$. ∎

📐 Theorem 10.34 — Diagonal dominance
Let $A\in M_n(\mathbb{C})$ be strictly diagonally dominant: $|a_{ii}|\gt R_i$ for every $i$.
  1. $A$ is invertible (Lévy–Desplanques).
  2. If moreover $A$ is Hermitian with every $a_{ii}\gt0$, then $A\succ0$.
Also: if $A$ is a real matrix and a Gershgorin disc is disjoint from all the other discs, then that disc contains exactly one eigenvalue, and it is real.

Proof. (1) $|0-a_{ii}|\gt R_i$ for each $i$, so $0$ lies in no disc and is not an eigenvalue. (2) The eigenvalues are real (Theorem 10.21) and lie in $\bigcup[a_{ii}-R_i,a_{ii}+R_i]\subseteq(0,\infty)$; apply Theorem 10.28. For the last claim, the isolated disc contains exactly one eigenvalue $\lambda$ by Theorem 10.33(3). Its centre $a_{ii}$ is real, so the disc is symmetric about $\mathbb{R}$, and $\bar\lambda$ — also an eigenvalue, since $p_A$ has real coefficients — lies in it too. Hence $\bar\lambda=\lambda$. ∎

Weak dominance is not enough: $\begin{pmatrix}1&1\\1&1\end{pmatrix}$ has $|a_{ii}|=R_i$ and is singular.

✏️ Example 6 — Three real eigenvalues without solving a cubic
Let $A=\begin{pmatrix}8&1&-1\\1&-3&1\\0&1&2\end{pmatrix}$. Without computing the characteristic polynomial, show that $A$ has three real eigenvalues, one in each of $[6,10]$, $[-5,-1]$ and $[1,3]$, and decide the sign of $\det A$. Then sharpen the bound on the largest eigenvalue with the column discs.
Row discs $R_1=|1|+|-1|=2$, $R_2=1+1=2$, $R_3=0+1=1$: $D_1=\{|z-8|\le2\}$, $D_2=\{|z+3|\le2\}$, $D_3=\{|z-2|\le1\}$. Their real cross-sections are $[6,10]$, $[-5,-1]$, $[1,3]$, and the discs are pairwise disjoint (the distances between centres, $11$, $6$, $5$, exceed the sums of radii $4$, $3$, $3$).
One eigenvalue each, real By Theorem 10.33(3) each disc contains exactly one eigenvalue; by Theorem 10.34 (real matrix, isolated discs centred on $\mathbb{R}$) each is real. So the eigenvalues satisfy $\lambda_1\in[6,10]$, $\lambda_2\in[-5,-1]$, $\lambda_3\in[1,3]$.
Sign of $\det A$ $\det A=\lambda_1\lambda_2\lambda_3$ is (positive)(negative)(positive) $\lt0$. Direct check: $\det A=8(-6-1)-1(2-0)+(-1)(1-0)=-56-2-1=-59$ ✓.
Columns The column radii are $1+0=1$, $1+1=2$, $1+1=2$, so the first column disc $\{|z-8|\le1\}$ is disjoint from the others ($\{|z+3|\le2\}$, $\{|z-2|\le2\}$): $\lambda_1\in[7,9]$. Numerically $\lambda_1\approx8.077$, $\lambda_2\approx-3.294$, $\lambda_3\approx2.218$ — all inside their discs.
Interactive 2 — Gershgorin Discs: Switching on the Off-Diagonal Entries

The matrix is $A(t)=D+t(A-D)$, where $D$ is the diagonal of $A$. At $t=0$ the eigenvalues (red dots) sit at the centres $a_{ii}$; as $t$ grows the discs (blue for rows, amber dashed for columns) inflate to radius $tR_i$, and the eigenvalues move along the purple paths — but they can never leave the union of the discs. An isolated group of $k$ discs always holds exactly $k$ eigenvalues; the last preset shows a single disc that holds none.

t—
eigenvalues of A(t)—
disc groups → eigenvalues—
✏️ Example 7 — Prove or disprove
Let $A,B\in M_n(\mathbb{C})$. Prove or disprove: (a) if $A$ and $B$ are Hermitian, so is $AB$; (b) if $A$ is normal and $A^3=A^2$, then $A^2=A$; (c) if $A$ is real, normal and has only real eigenvalues, then $A$ is symmetric; (d) if every eigenvalue of $A$ has modulus $1$ and $A$ is diagonalisable, then $A$ is unitary; (e) if $A$ is Hermitian and $A^5=I$, then $A=I$.
(a) False $(AB)^*=B^*A^*=BA$, so $AB$ is Hermitian iff $AB=BA$. Counterexample: $\begin{pmatrix}1&0\\0&-1\end{pmatrix}\begin{pmatrix}0&1\\1&0\end{pmatrix}=\begin{pmatrix}0&1\\-1&0\end{pmatrix}$, which is skew-symmetric.
(b) True $A=\sum\lambda_jP_j$ gives $\sum(\lambda_j^3-\lambda_j^2)P_j=0$, so each $\lambda_j\in\{0,1\}$ (uniqueness in Theorem 10.25(5)); then $A^2=\sum\lambda_j^2P_j=\sum\lambda_jP_j=A$. Without normality it fails: $N=\begin{pmatrix}0&1\\0&0\end{pmatrix}$ has $N^3=N^2=0$, but $N^2=0\ne N$.
(c) True Over $\mathbb{C}$, a normal matrix with real spectrum is Hermitian (Theorem 10.26(1)); a real Hermitian matrix is symmetric.
(d) False $A=\begin{pmatrix}1&1\\0&-1\end{pmatrix}$ has distinct eigenvalues $\pm1$, so it is diagonalisable, but its columns are not orthonormal ($A^*A=\begin{pmatrix}1&1\\1&2\end{pmatrix}\ne I$). Unitarity needs an orthonormal eigenbasis, i.e. normality.
(e) True The eigenvalues are real with $\lambda^5=1$, so $\lambda=1$; a Hermitian matrix is diagonalisable, so $A=U\,I\,U^*=I$.
⚠️ Counterexamples and pitfalls
  1. Real eigenvalues do not make a matrix symmetric: $\begin{pmatrix}1&1\\0&2\end{pmatrix}$. Only for normal matrices does the spectrum determine the class (Theorem 10.26).
  2. $|\lambda|=1$ for all eigenvalues does not make a matrix unitary: $\begin{pmatrix}1&1\\0&1\end{pmatrix}$ — not normal, $\|A\|=\tfrac{1+\sqrt5}2$.
  3. Normal over $\mathbb{R}$ is not orthogonally diagonalisable over $\mathbb{R}$: the rotation $\begin{pmatrix}0&-1\\1&0\end{pmatrix}$ has no real eigenvalue. Over $\mathbb{R}$ the spectral theorem needs symmetric.
  4. Diagonalisable is not unitarily diagonalisable: $\begin{pmatrix}1&1\\0&2\end{pmatrix}$ has eigenvectors $(1,0)$ and $(1,1)$, which are not orthogonal. Similar does not imply unitarily similar ($\operatorname{tr}A^*A=6$ vs $5$ for $\operatorname{diag}(1,2)$).
  5. Products and sums: the product of two Hermitian matrices is Hermitian only if they commute (Example 7(a)); the sum of two normal matrices need not be normal: $\begin{pmatrix}0&1\\1&0\end{pmatrix}+\begin{pmatrix}0&1\\-1&0\end{pmatrix}=\begin{pmatrix}0&2\\0&0\end{pmatrix}$.
  6. "$x^TAx\gt0$ for all $x\ne0$" is not positive definiteness for non-symmetric real $A$: $\begin{pmatrix}1&1\\-1&1\end{pmatrix}$ has eigenvalues $1\pm i$.
  7. Positive entries $\ne$ positive definite: $\begin{pmatrix}1&2\\2&1\end{pmatrix}$ has eigenvalue $-1$; and $\begin{pmatrix}2&-1\\-1&2\end{pmatrix}\succ0$ has negative entries. What a positive definite matrix must have is a positive diagonal and $|a_{ij}|^2\lt a_{ii}a_{jj}$.
  8. $\|A\|\ne\rho(A)$ unless $A$ is normal: $\begin{pmatrix}0&1\\0&0\end{pmatrix}$ has $\rho=0$ and $\|A\|=1$; Example 5 has $\rho=5\lt3\sqrt5$.
  9. Singular values are not $|\text{eigenvalues}|$ in general: $\{3\sqrt5,\sqrt5\}$ versus $\{5,3\}$ in Example 5. They agree exactly for normal matrices.
  10. Square roots: $I_2$ has infinitely many square roots (all reflections), only one of them positive; $\begin{pmatrix}0&1\\0&0\end{pmatrix}$ has none.
  11. A single Gershgorin disc may contain no eigenvalue: for $\begin{pmatrix}0.5&1\\4&-0.5\end{pmatrix}$ the disc $|z-0.5|\le1$ misses both eigenvalues $\pm\sqrt{4.25}$; only the union (or an isolated group) is guaranteed.
  12. Weak diagonal dominance does not give invertibility: $\begin{pmatrix}1&1\\1&1\end{pmatrix}$.
  13. Unitary matrices are diagonalisable over $\mathbb{C}$, but real orthogonal ones need not be over $\mathbb{R}$ (rotations); and a real orthogonal matrix need not have $\det=1$ (reflections).

Exercises

I
Self-adjoint, unitary and normal operators.
  • 1 Show that $A=\begin{pmatrix}1&2i\\-2i&5\end{pmatrix}$ is Hermitian and find its eigenvalues.
  • 2 Prove that every $A\in M_n(\mathbb{C})$ can be written uniquely as $A=H+iK$ with $H,K$ Hermitian, and that $A$ is normal if and only if $HK=KH$.
  • 3 Find all $a,b\in\mathbb{C}$ for which $\begin{pmatrix}a&b\\-\bar b&\bar a\end{pmatrix}$ is unitary.
  • 4 Prove that a normal nilpotent operator is $0$, and that a normal operator $T$ satisfies $\ker T^2=\ker T$.
  • 5 Let $T$ be self-adjoint with $T^k=I$ for some $k\ge1$. Prove that $T^2=I$.
  • 6 Let $T$ be normal. Prove that $\operatorname{im}T=\operatorname{im}T^*$ and $V=\ker T\oplus\operatorname{im}T$ with $\ker T\perp\operatorname{im}T$. Give a $2\times2$ matrix for which $\ker A\cap\operatorname{im}A\ne\{0\}$.
  • 7 Prove that an upper triangular unitary matrix is diagonal, with diagonal entries of modulus $1$.
  • 8 (Cayley transform.) Let $K$ be skew-Hermitian. Prove that $I+K$ is invertible and that $U=(I-K)(I+K)^{-1}$ is unitary.
1 $\overline{a_{12}}=\overline{2i}=-2i=a_{21}$ and the diagonal is real: $A^*=A$. $\operatorname{tr}A=6$, $\det A=5-(2i)(-2i)=5-4=1$, so $\lambda=3\pm2\sqrt2$ (both real and positive, so in fact $A\succ0$).
2 $H=\tfrac12(A+A^*)$, $K=\tfrac1{2i}(A-A^*)$; then $K^*=-\tfrac1{2i}(A^*-A)=K$. If $H+iK=H'+iK'$, then $H-H'=i(K'-K)$ is both Hermitian and skew-Hermitian, hence $0$. Finally $A^*A-AA^*=(H-iK)(H+iK)-(H+iK)(H-iK)=2i(HK-KH)$.
3 With $U$ the matrix, $U^*U=\begin{pmatrix}\bar a&-b\\\bar b&a\end{pmatrix}\begin{pmatrix}a&b\\-\bar b&\bar a\end{pmatrix}=\begin{pmatrix}|a|^2+|b|^2&\bar ab-b\bar a\\\bar ba-a\bar b&|a|^2+|b|^2\end{pmatrix}=(|a|^2+|b|^2)I$. Unitary iff $|a|^2+|b|^2=1$. (These are the matrices of $SU(2)$.)
4 If $T$ is normal and $T^m=0$, all eigenvalues are $0$ and $T=\sum0\cdot P_j=0$ (Theorem 10.26(5)). For the kernel: if $T^2x=0$ then $Tx\in\ker T=\ker T^*$, so $\|Tx\|^2=\langle T^*Tx,x\rangle=0$.
5 The eigenvalues are real with $\lambda^k=1$, so $\lambda=\pm1$; $T=\sum\lambda_jP_j$ gives $T^2=\sum\lambda_j^2P_j=\sum P_j=I$.
6 $\operatorname{im}T=(\ker T^*)^\perp=(\ker T)^\perp=\operatorname{im}T^*$, using $\ker T=\ker T^*$. Hence $V=\ker T\oplus(\ker T)^\perp=\ker T\oplus\operatorname{im}T$, orthogonally. For $A=\begin{pmatrix}0&1\\0&0\end{pmatrix}$, $\ker A=\operatorname{im}A=\operatorname{span}(e_1)$.
7 A unitary matrix is normal, and a normal upper triangular matrix is diagonal (proof of Theorem 10.23). A diagonal unitary matrix has $|d_i|^2=1$.
8 The eigenvalues of $K$ are purely imaginary (Theorem 10.21(2)), so $-1$ is not one and $I+K$ is invertible. Using $(I+K)^*=I-K$, $(I-K)^*=I+K$ and the fact that $I-K$ and $I+K$ commute: $U^*U=(I-K)^{-1}(I+K)(I-K)(I+K)^{-1}=(I-K)^{-1}(I-K)(I+K)(I+K)^{-1}=I$.
II
The spectral theorem in practice.
  • 1 Orthogonally diagonalise $\begin{pmatrix}3&1\\1&3\end{pmatrix}$.
  • 2 Orthogonally diagonalise $\begin{pmatrix}1&2&2\\2&1&2\\2&2&1\end{pmatrix}$.
  • 3 Unitarily diagonalise $\begin{pmatrix}2&i\\-i&2\end{pmatrix}$.
  • 4 For $B=\begin{pmatrix}0&1\\1&0\end{pmatrix}$ find the spectral projections and compute $e^{tB}$ from $e^{tB}=\sum e^{t\lambda_j}P_j$.
  • 5 Find the eigenvalues of the circulant $\begin{pmatrix}1&2&3&4\\4&1&2&3\\3&4&1&2\\2&3&4&1\end{pmatrix}$ and its norm $\|C\|$.
  • 6 Prove that every $3\times3$ real orthogonal $Q$ with $\det Q=1$ has $1$ as an eigenvalue. Find the axis and angle of the rotation $Q=\begin{pmatrix}0&0&1\\1&0&0\\0&1&0\end{pmatrix}$.
  • 7 Prove that a real normal $2\times2$ matrix is either symmetric or of the form $\begin{pmatrix}a&b\\-b&a\end{pmatrix}$ with $b\ne0$.
  • 8 Prove that two commuting normal operators $S,T$ on a complex space have a common orthonormal eigenbasis.
1 Eigenvalues $4$ on $(1,1)$ and $2$ on $(1,-1)$: $Q=\tfrac1{\sqrt2}\begin{pmatrix}1&1\\1&-1\end{pmatrix}$, $Q^TAQ=\operatorname{diag}(4,2)$.
2 $A=2J-I$: eigenvalues $5$ on $(1,1,1)$ and $-1$ (twice) on $x_1+x_2+x_3=0$. $Q=\big[\tfrac1{\sqrt3}(1,1,1)^T,\ \tfrac1{\sqrt2}(1,-1,0)^T,\ \tfrac1{\sqrt6}(1,1,-2)^T\big]$ and $Q^TAQ=\operatorname{diag}(5,-1,-1)$.
3 Hermitian with $\operatorname{tr}=4$, $\det=4-1=3$: eigenvalues $1,3$. For $3$: $-x+iy=0$, $u_3=\tfrac1{\sqrt2}(1,-i)^T$; for $1$: $x+iy=0$, $u_1=\tfrac1{\sqrt2}(1,i)^T$. $U=\tfrac1{\sqrt2}\begin{pmatrix}1&1\\i&-i\end{pmatrix}$ gives $U^*AU=\operatorname{diag}(1,3)$; check $\langle u_1,u_3\rangle=\tfrac12(1+i\cdot\bar{(-i)})=\tfrac12(1-1)=0$.
4 $\lambda=\pm1$ with $P_\pm=\tfrac12(I\pm B)$. $e^{tB}=e^tP_++e^{-t}P_-=\begin{pmatrix}\cosh t&\sinh t\\\sinh t&\cosh t\end{pmatrix}$.
5 With $\omega=i$ and $f_k=(1,i^k,i^{2k},i^{3k})$, the eigenvalues are $1+2i^k+3i^{2k}+4i^{3k}$: $k=0$: $10$; $k=1$: $1+2i-3-4i=-2-2i$; $k=2$: $1-2+3-4=-2$; $k=3$: $-2+2i$. Circulants are normal, so $\|C\|=\rho(C)=10$.
6 By Theorem 10.27(2), $Q\sim\operatorname{diag}(1,R_\theta)$; directly, $\det(Q-I)=\det(Q-QQ^T)=\det Q\det(I-Q^T)=\det(I-Q)=(-1)^3\det(Q-I)$, so $\det(Q-I)=0$. Here $Q(1,1,1)^T=(1,1,1)^T$: the axis is $\mathbb{R}(1,1,1)$; $\operatorname{tr}Q=0=1+2\cos\theta$ gives $\theta=\tfrac{2\pi}3$ (a cyclic permutation of the axes).
7 For $A=\begin{pmatrix}a&b\\c&d\end{pmatrix}$, $A^TA=AA^T$ means $c^2=b^2$ and $ab+cd=ac+bd$. If $c=b$, $A$ is symmetric. If $c=-b\ne0$, the second equation becomes $2b(a-d)=0$, so $d=a$.
8 $T^*=\sum\bar\lambda_jP_j$ is a polynomial in $T$ (Theorem 10.25(4)), so $S$ commutes with $T^*$ too. Each eigenspace $E$ of $S$ is therefore invariant under $T$ and $T^*$, so $T|_E$ is normal and $E$ has an orthonormal basis of eigenvectors of $T$ (Theorem 10.23) — all of them eigenvectors of $S$. The eigenspaces of $S$ are mutually orthogonal and span $V$; put the bases together.
III
Positive operators, square roots and singular values.
  • 1 Decide whether $\begin{pmatrix}2&-1&0\\-1&2&-1\\0&-1&2\end{pmatrix}$ is positive definite by finding its eigenvalues.
  • 2 Find the positive square root of $\begin{pmatrix}10&6\\6&10\end{pmatrix}$.
  • 3 Prove that $A^*A\succeq0$ for every $A\in M_{m\times n}(\mathbb{C})$, and that $A^*A\succ0$ iff the columns of $A$ are linearly independent.
  • 4 Find the singular values and $\|A\|$ for $A=\begin{pmatrix}1&1\\0&1\end{pmatrix}$, and compare with $\rho(A)$.
  • 5 Find the polar decomposition of $\begin{pmatrix}0&-2\\1&0\end{pmatrix}$.
  • 6 Let $A\succ0$ and $B\succeq0$. Prove that $AB$ is diagonalisable with real non-negative eigenvalues, though $AB$ need not be Hermitian.
  • 7 Prove that $\operatorname{tr}(AB)\ge0$ for $A,B\succeq0$, and that $A\succeq0$ with $\operatorname{tr}A=0$ implies $A=0$.
  • 8 Prove that a positive semidefinite matrix with $a_{ii}=0$ has its whole $i$-th row and column equal to $0$.
1 The eigenvalues of this tridiagonal matrix are $2-2\cos\tfrac{k\pi}{4}$, $k=1,2,3$: $2-\sqrt2$, $2$, $2+\sqrt2$ (check: sum $6=\operatorname{tr}$, product $(4-2)\cdot2=4=\det$). All positive: positive definite.
2 Eigenvalues $16$ on $(1,1)$, $4$ on $(1,-1)$: $\sqrt A=4P_{16}+2P_4=\begin{pmatrix}3&1\\1&3\end{pmatrix}$; check $\begin{pmatrix}3&1\\1&3\end{pmatrix}^2=\begin{pmatrix}10&6\\6&10\end{pmatrix}$.
3 $x^*A^*Ax=\|Ax\|^2\ge0$ and $A^*A$ is Hermitian. It is $\gt0$ for all $x\ne0$ iff $Ax\ne0$ for all $x\ne0$, i.e. iff $\ker A=0$, i.e. the columns are independent.
4 $A^TA=\begin{pmatrix}1&1\\1&2\end{pmatrix}$, eigenvalues $\tfrac{3\pm\sqrt5}2=\left(\tfrac{1\pm\sqrt5}{2}\right)^2$. So $\sigma_1=\tfrac{1+\sqrt5}2\approx1.618$ (the golden ratio) and $\sigma_2=\tfrac{\sqrt5-1}2\approx0.618$; $\|A\|=\sigma_1\gt1=\rho(A)$. Check $\sigma_1\sigma_2=1=\det A$.
5 $A^TA=\operatorname{diag}(1,4)$, so $P=\operatorname{diag}(1,2)$ and $W=AP^{-1}=\begin{pmatrix}0&-1\\1&0\end{pmatrix}$ (rotation by $\tfrac\pi2$). $A=WP$: first stretch the $y$-axis by $2$, then rotate.
6 $AB=A^{1/2}\big(A^{1/2}BA^{1/2}\big)A^{-1/2}$ is similar to $C=A^{1/2}BA^{1/2}=(B^{1/2}A^{1/2})^*(B^{1/2}A^{1/2})\succeq0$, which is unitarily diagonalisable with eigenvalues $\ge0$. Example: $A=\begin{pmatrix}2&1\\1&1\end{pmatrix}$, $B=\operatorname{diag}(1,2)$: $AB=\begin{pmatrix}2&2\\1&2\end{pmatrix}$ is not symmetric, with eigenvalues $2\pm\sqrt2$.
7 $\operatorname{tr}(AB)=\operatorname{tr}(A^{1/2}A^{1/2}B)=\operatorname{tr}(A^{1/2}BA^{1/2})\ge0$, the trace of a positive semidefinite matrix being the sum of its non-negative eigenvalues. If $\operatorname{tr}A=0$ then all eigenvalues are $0$, and the Hermitian (diagonalisable) $A$ is $0$.
8 The principal submatrix on $\{i,j\}$ is $\begin{pmatrix}0&a_{ij}\\\overline{a_{ij}}&a_{jj}\end{pmatrix}\succeq0$, so its determinant $-|a_{ij}|^2$ is $\ge0$: $a_{ij}=0$. (Alternatively take $x=te_i+e_j$ and let $t\to\pm\infty$.)
IV
Locating eigenvalues; applications.
  • 1 Use Gershgorin discs to locate the eigenvalues of $\begin{pmatrix}5&1&0\\1&0&1\\0&1&-5\end{pmatrix}$. Can you decide from the discs whether the matrix is invertible?
  • 2 Show without computing eigenvalues that $\begin{pmatrix}4&1&1\\1&5&2\\1&2&6\end{pmatrix}$ is positive definite.
  • 3 For the matrix in 2, use the Rayleigh quotient with $x=(1,1,1)^T$ to give a lower bound for $\lambda_{\max}$, and compare with a Gershgorin upper bound.
  • 4 Prove $\|A\|=\rho(A)$ for normal $A$, and give a non-normal $A$ with $\|A\|\gt\rho(A)$ and $\rho(A)=0$.
  • 5 Find the numerical range $\{\langle Ax,x\rangle:\|x\|=1\}$ of $\begin{pmatrix}2&1\\1&2\end{pmatrix}$.
  • 6 Applied (vibrations). Two carts of mass $1\ \text{kg}$ are joined to two walls and to each other by three springs of stiffness $1\ \text{N/m}$, so $x''=-Kx$ with $K=\begin{pmatrix}2&-1\\-1&2\end{pmatrix}$. Find the normal-mode frequencies and mode shapes, and explain why the mode shapes are orthogonal.
  • 7 Applied (statistics). Two measurements have covariance matrix $\Sigma=\begin{pmatrix}5&2\\2&2\end{pmatrix}$ (in $\text{cm}^2$). Find the first principal direction and the fraction of the total variance it explains.
1 The matrix is symmetric, so its eigenvalues are real and lie in $[4,6]\cup[-2,2]\cup[-6,-4]$; the three discs are disjoint, so there is one eigenvalue in each. The middle disc contains $0$, so the discs cannot decide invertibility — and in fact $\det=5(0-1)-1(-5-0)=0$: the eigenvalues are $\pm\sqrt{27}$ and $0$.
2 Symmetric, positive diagonal, and strictly diagonally dominant ($4\gt2$, $5\gt3$, $6\gt3$): positive definite by Theorem 10.34(2). (Eigenvalues $\approx3.308,\ 3.643,\ 8.049$.)
3 $R(x)=\dfrac{x^TAx}{x^Tx}=\dfrac{\text{sum of all entries}}{3}=\dfrac{23}{3}\approx7.667\le\lambda_{\max}$. Gershgorin: $\lambda_{\max}\le\max(4+2,5+3,6+3)=9$. So $7.667\le\lambda_{\max}\le9$; the true value is $\approx8.049$.
4 Theorem 10.26(6). Non-normal: $\begin{pmatrix}0&1\\0&0\end{pmatrix}$, with $\rho=0$ and $\|A\|=\|Ae_2\|=1$.
5 Hermitian with eigenvalues $1,3$: by Theorem 10.32(2) the numerical range is $[1,3]$ (for real $x=(\cos t,\sin t)$, $\langle Ax,x\rangle=2+\sin2t$).
6 Trying $x=ve^{i\omega t}$ gives $Kv=\omega^2v$. $K$ has eigenvalues $1$ on $(1,1)$ and $3$ on $(1,-1)$, so $\omega_1=1\ \text{rad/s}$ (carts in phase) and $\omega_2=\sqrt3\approx1.732\ \text{rad/s}$ (carts in antiphase). $K$ is symmetric, so eigenvectors for distinct eigenvalues are orthogonal (Theorem 10.21(1)).
7 Eigenvalues $6$ on $(2,1)$ and $1$ on $(1,-2)$. The first principal direction is $\tfrac1{\sqrt5}(2,1)$; it carries variance $6\ \text{cm}^2$ out of $\operatorname{tr}\Sigma=7\ \text{cm}^2$, i.e. $6/7\approx85.7\%$.
🎯 Exam corner CSIR Part BCSIR Part CGATE MA

E1. CSIR Part B GATE MA A real symmetric $3\times3$ matrix $A$ has eigenvalues $2,2,5$, and $(1,1,1)^T$ is an eigenvector for the eigenvalue $5$. With $J$ the all-ones matrix, $A$ equals (A) $2I+J$; (B) $5I-J$; (C) $2I+3J$; (D) $3I+J$.

Idea By the spectral theorem the eigenspace of $2$ is the orthogonal complement of $(1,1,1)$, so $A=5P+2(I-P)$ with $P=\tfrac13J$.
Work $A=2I+3P=2I+J$. Checking the others on $(1,1,1)$: (B) gives $5-3=2$, (C) gives $2+9=11$, (D) gives $3+3=6$ — wrong.
Answer (A).

E2. CSIR Part B Which matrix is normal but neither Hermitian, skew-Hermitian nor unitary? (A) $\begin{pmatrix}1&1\\0&1\end{pmatrix}$; (B) $\begin{pmatrix}1&-2\\2&1\end{pmatrix}$; (C) $\begin{pmatrix}0&i\\i&0\end{pmatrix}$; (D) $\begin{pmatrix}2&i\\-i&2\end{pmatrix}$.

Eliminate (A) is not normal. (C) satisfies $C^*=-C$ (skew-Hermitian) and $C^*C=-C^2=I$ (also unitary). (D) is Hermitian.
Why (B) $B=I+2K$ with $K=\begin{pmatrix}0&-1\\1&0\end{pmatrix}$, so $B^TB=BB^T=5I$: normal, not orthogonal, not symmetric or skew. Eigenvalues $1\pm2i$ — neither real, imaginary nor unimodular, consistent with Theorem 10.26.
Answer (B).

E3. CSIR Part B GATE MA For $A=\begin{pmatrix}1&2\\0&1\end{pmatrix}$, the operator norm $\|A\|=\max_{\|x\|_2=1}\|Ax\|_2$ is (A) $1$; (B) $\sqrt5$; (C) $1+\sqrt2$; (D) $3$.

Idea $\|A\|=\sigma_1=\sqrt{\lambda_{\max}(A^TA)}$; the eigenvalue $1$ of the non-normal $A$ is irrelevant.
Work $A^TA=\begin{pmatrix}1&2\\2&5\end{pmatrix}$ has $\operatorname{tr}=6$, $\det=1$, so $\lambda_{\max}=3+2\sqrt2=(1+\sqrt2)^2$. Sanity check: $\|A\|\le\|A\|_F=\sqrt6\approx2.449$ and $1+\sqrt2\approx2.414$.
Answer (C).

E4. CSIR Part C Let $A\in M_n(\mathbb{C})$ be normal. Which of the following are true? (One or more options may be correct.) (A) If all eigenvalues of $A$ are real, then $A$ is Hermitian. (B) Every eigenvector of $A$ is an eigenvector of $A^*$. (C) $\operatorname{rank}A=\operatorname{rank}A^2$. (D) If $A$ has real entries, then $A=QDQ^T$ for some real orthogonal $Q$ and real diagonal $D$.

(A) True Theorem 10.26(1).
(B) True $Ax=\lambda x\iff A^*x=\bar\lambda x$ (Theorem 10.22(3)).
(C) True $A=UDU^*$ and $A^2=UD^2U^*$; $D$ and $D^2$ have the same number of nonzero entries (Theorem 10.26(5)).
(D) False $\begin{pmatrix}0&-1\\1&0\end{pmatrix}$ is real and normal but has eigenvalues $\pm i$; $QDQ^T$ with real $D$ is symmetric.
Answer (A), (B) and (C).

E5. CSIR Part C Let $A$ be a real $n\times n$ positive definite matrix, $n\ge2$. Which of the following are true? (A) If $B$ is also positive definite, then $AB$ is positive definite. (B) $A+A^{-1}-2I$ is positive semidefinite. (C) The entry of $A$ of largest absolute value lies on the diagonal. (D) There is exactly one real symmetric matrix $S$ with $S^2=A$.

(A) False $AB$ need not be symmetric: $A=\begin{pmatrix}2&1\\1&1\end{pmatrix}$, $B=\operatorname{diag}(1,2)$ give $AB=\begin{pmatrix}2&2\\1&2\end{pmatrix}$. (Its eigenvalues $2\pm\sqrt2$ are positive — Exercise III.6 — but positive definiteness requires symmetry.)
(B) True It is symmetric with eigenvalues $\lambda+\tfrac1\lambda-2=\left(\sqrt\lambda-\tfrac1{\sqrt\lambda}\right)^2\ge0$.
(C) True $|a_{ij}|^2\lt a_{ii}a_{jj}\le\max(a_{ii},a_{jj})^2$ for $i\ne j$ (Theorem 10.28).
(D) False Both $\sqrt A$ and $-\sqrt A$ are symmetric square roots (and there are more when eigenvalues repeat). Uniqueness holds only among positive semidefinite roots.
Answer (B) and (C).

E6. CSIR Part C Let $A=\begin{pmatrix}10&1&2\\1&4&1\\2&0&-6\end{pmatrix}$. Which of the following are true? (A) $A$ is invertible. (B) Exactly two eigenvalues of $A$ have positive real part. (C) $A$ has a pair of non-real eigenvalues. (D) $\det A\lt0$.

Discs $R_1=3$, $R_2=2$, $R_3=2$: discs $|z-10|\le3$, $|z-4|\le2$, $|z+6|\le2$, with real sections $[7,13]$, $[2,6]$, $[-8,-4]$ — pairwise disjoint. Each holds exactly one eigenvalue, which is real (Theorem 10.34).
(A) True $0$ lies in no disc.
(B) True; (C) False The eigenvalues are real, two of them positive (in $[7,13]$ and $[2,6]$) and one negative.
(D) True $\det A=(+)(+)(-)\lt0$. Directly: $10(-24)-1(-6-2)+2(0-8)=-240+8-16=-248$. (Eigenvalues $\approx10.418,\ 3.817,\ -6.236$.)
Answer (A), (B) and (D).
🧑‍🏫 For the teacher

Suggested plan — 3 lectures × 60 min.

  1. The special classes. Definition 10.9 with the complex-number analogy; Theorem 10.19 (prove the polarisation identity on the board — it is used twice); Theorem 10.20 with the rotation counterexample; Theorem 10.21 in full; Theorem 10.22(1)–(4). Example 1 as a class activity (each row of students takes one matrix).
  2. The spectral theorem. Theorem 10.23 via Schur (the "first row" argument is the heart — do it slowly), Theorem 10.24 by induction, then Theorem 10.25 and Theorem 10.26 with the warning examples. Example 2 on the board; Example 3 to show a normal matrix that is not Hermitian. Project Interactive 1 with the two symmetric presets: the green eigen-lines are the axes of the ellipse.
  3. Positivity, SVD and eigenvalue location. Theorems 10.28–10.29 with Example 4; the SVD (Theorem 10.30) and Example 5, then Interactive 1 again with "Example 5" and "shear" to contrast singular values with eigenvalues. Gershgorin (Theorem 10.33) with Example 6 and Interactive 2 — let students predict where the eigenvalues end before pressing Play. Finish with E4 or E6.

Discussion questions. (1) Over $\mathbb{R}$, why is "normal" not enough for orthogonal diagonalisation, and what replaces diagonal form (Theorem 10.27)? (2) If you know only the eigenvalues of a matrix, what can you say about $\|A\|$? What if you also know $A$ is normal? (3) Why does a positive definite matrix have a unique positive square root, while $I$ has infinitely many square roots?

Misconceptions to watch for. Deciding "Hermitian" or "unitary" from the eigenvalues of a non-normal matrix; forgetting to orthonormalise inside a repeated eigenspace (and orthonormalising across different eigenspaces, which is unnecessary); confusing singular values with $|\lambda|$; calling a non-symmetric real matrix positive definite because $x^TAx\gt0$; believing that every Gershgorin disc contains an eigenvalue; forgetting conjugates in $\langle x,y\rangle$ on $\mathbb{C}^n$.

Suggested homework. I(1, 2, 4, 8), II(2, 3, 5, 6, 8), III(2, 4, 5, 6), IV(1, 3, 6); Exam corner E2, E4, E5.

🔬 Research–Teaching NexusResearch-tutored · optional, about 60 minutes

Research question. How few numbers does it take to store an image, a data table or a climate record "almost exactly" — and why is the truncated singular value decomposition the best possible answer?

Why it matters. A grey-scale image, a table of patients × gene expressions, or a grid of sea-surface temperatures × months is a matrix $A$. Truncating its SVD to $A_k=\sum_{i\le k}\sigma_iu_iv_i^*$ is the idea behind principal component analysis in statistics, "empirical orthogonal functions" in climate science, latent-factor models in recommender systems and noise reduction in medical imaging. The Eckart–Young theorem (Theorem 10.30(5)) says no rank-$k$ matrix does better.

What to do.

  1. By hand: find the SVD of $A=\begin{pmatrix}2&0&1\\0&2&1\end{pmatrix}$ (start from $AA^T$), the best rank-one approximation $A_1$, and $\|A-A_1\|$ and $\|A-A_1\|_F$. Check that they equal $\sigma_2$ and $\sqrt{\sum_{i\gt1}\sigma_i^2}$.
  2. In Python (numpy.linalg.svd), load any grey-scale image as a matrix, plot $\log\sigma_i$ against $i$, and display $A_k$ for $k=5,20,50$. Record the relative error $\|A-A_k\|_F/\|A\|_F$ and the storage $k(m+n+1)$ compared with $mn$.
  3. Test Eckart–Young numerically: generate 1000 random rank-$k$ matrices $B$ (as products of random $m\times k$ and $k\times n$ matrices) and check that $\|A-B\|$ never beats $\sigma_{k+1}$.
  4. Add Gaussian noise to the image and compare $A_k$ of the noisy image with the original. For which $k$ is the error smallest?

Tools: Python with NumPy and Matplotlib (or any language with an SVD routine).

Hand in: A two-page report with your hand computation, the singular-value plot, the reconstructions, a table of errors and storage, and one paragraph explaining why the singular values of a "natural" image decay quickly.

Going further. Read about randomized SVD algorithms for huge matrices, or about the nuclear-norm relaxation used in matrix completion (filling in the missing entries of a ratings table).

Instructor note: $AA^T=\begin{pmatrix}5&1\\1&5\end{pmatrix}$ has eigenvalues $6,4$, so $\sigma_1=\sqrt6$, $\sigma_2=2$, $u_1=\tfrac1{\sqrt2}(1,1)$, $v_1=A^Tu_1/\sigma_1=\tfrac1{\sqrt{12}}(2,2,2)=\tfrac1{\sqrt3}(1,1,1)$, and $A_1=\sigma_1u_1v_1^T=\begin{pmatrix}1&1&1\\1&1&1\end{pmatrix}$. Then $A-A_1=\begin{pmatrix}1&-1&0\\-1&1&0\end{pmatrix}$ has $\|A-A_1\|=\|A-A_1\|_F=2=\sigma_2$. Typical photographs keep 95–99% of $\|A\|_F^2$ with $k$ a few percent of $\min(m,n)$; with noise, the error is usually smallest for a moderate $k$, where the remaining singular values are at the level of the noise.

🧪 Self-check quiz

0/5 correct

Q1. Every eigenvalue of a Hermitian matrix is

Q2. Which matrix is not normal?

Q3. A real square matrix $A$ can be written $QDQ^T$ with $Q$ real orthogonal and $D$ real diagonal if and only if

Q4. The positive semidefinite square root of $\begin{pmatrix}5&4\\4&5\end{pmatrix}$ is

Q5. For $A=\begin{pmatrix}3&0\\4&5\end{pmatrix}$, the operator norm $\|A\|=\max_{\|x\|=1}\|Ax\|$ is