A symmetric matrix $A^{T}=A$ always has real eigenvalues and perpendicular eigenvectors, so $A=PDP^{T}$: it simply stretches space along perpendicular axes. This explains ellipses, quadratic forms and the SVD.
In Chapter 7 diagonalisation could go wrong in several ways: the eigenvalues might be complex, there might not be enough eigenvectors, and even when $A=PDP^{-1}$ worked, the eigenvectors in $P$ usually met at awkward angles. There is one large family of matrices for which none of this happens: the symmetric matrices, those equal to their own transpose. They are everywhere — the stiffness matrix of a bridge, the covariance matrix of a data set, the matrix of distances between cities, the Hessian of second derivatives in optimisation, the inertia tensor of a spinning satellite.
The headline result of this section, the Spectral Theorem, says that a symmetric matrix can always be diagonalised by an orthogonal matrix. Geometrically, a symmetric matrix does nothing more than stretch (or flip) space along a set of mutually perpendicular axes. That one fact lets us rotate any quadratic expression like $6x^2-4xy+9y^2$ into a sum of squares, identify tilted ellipses and hyperbolas, decide whether a quadratic is always positive, and — looking ahead — understand how a photograph can be compressed with the singular value decomposition.
Symmetric Matrices and Orthogonal Diagonalisation
📘 Definition 8.9 — Symmetric matrix
A square matrix $A$ is symmetric if $A^{T}=A$, that is $a_{ij}=a_{ji}$ for all $i,j$: the entries are mirrored across the main diagonal. For example $\begin{bmatrix}6&-2\\-2&9\end{bmatrix}$ and $\begin{bmatrix}2&1&1\\1&2&1\\1&1&2\end{bmatrix}$ are symmetric; $\begin{bmatrix}1&2\\0&1\end{bmatrix}$ is not.
📐 Theorem 8.15 — Eigenvalues and eigenvectors of a symmetric matrix
Let $A$ be a real symmetric $n\times n$ matrix. Then
every eigenvalue of $A$ is a real number;
eigenvectors that correspond to distinct eigenvalues are orthogonal.
Why (b) is true. Let $A\mathbf{v}_1=\lambda_1\mathbf{v}_1$ and $A\mathbf{v}_2=\lambda_2\mathbf{v}_2$ with $\lambda_1\ne\lambda_2$. Using $\mathbf{u}\cdot\mathbf{w}=\mathbf{u}^{T}\mathbf{w}$ and $A^{T}=A$,
$$\lambda_1(\mathbf{v}_1\cdot\mathbf{v}_2)=(A\mathbf{v}_1)^{T}\mathbf{v}_2=\mathbf{v}_1^{T}A^{T}\mathbf{v}_2=\mathbf{v}_1^{T}(A\mathbf{v}_2)=\lambda_2(\mathbf{v}_1\cdot\mathbf{v}_2).$$
So $(\lambda_1-\lambda_2)(\mathbf{v}_1\cdot\mathbf{v}_2)=0$, and since $\lambda_1\ne\lambda_2$ we get $\mathbf{v}_1\cdot\mathbf{v}_2=0$. Why (a) is true for $2\times2$ matrices. For $A=\begin{bmatrix}a&b\\b&d\end{bmatrix}$ the characteristic polynomial is $\lambda^2-(a+d)\lambda+(ad-b^2)$, whose discriminant is $(a+d)^2-4(ad-b^2)=(a-d)^2+4b^2\ge0$. So the roots are real. (The general proof uses complex numbers and is left to a later course.) Compare the rotation matrix $\begin{bmatrix}0&-1\\1&0\end{bmatrix}$ of Section 7.1 — not symmetric — whose eigenvalues are $\pm i$.
Recall from Section 8.1 that a square matrix is orthogonal if its columns are orthonormal; then $P^{T}P=I$, so $P^{-1}=P^{T}$. (Section 8.1 called such a matrix $U$, and Section 8.2 wrote $Q$ for a matrix with orthonormal columns; here we call it $P$, matching $A=PDP^{-1}$ from Chapter 7.) When the eigenvectors of $A$ can be chosen orthonormal, the troublesome inverse in $A=PDP^{-1}$ becomes a simple transpose.
📘 Definition 8.10 — Orthogonally diagonalisable
An $n\times n$ matrix $A$ is orthogonally diagonalisable if there are an orthogonal matrix $P$ (so $P^{-1}=P^{T}$) and a diagonal matrix $D$ with
$$A=PDP^{T}=PDP^{-1}.$$
The columns of $P$ are then an orthonormal basis of $\mathbb{R}^n$ consisting of eigenvectors of $A$, and the diagonal entries of $D$ are the corresponding eigenvalues.
✏️ Example 1 — Orthogonally diagonalising a $2\times2$ matrix
Eigenvalues $\det(A-\lambda I)=(6-\lambda)(9-\lambda)-4=\lambda^2-15\lambda+50=(\lambda-5)(\lambda-10)$, so $\lambda_1=5$, $\lambda_2=10$ — real, as Theorem 8.15 promises.
Eigenvectors $\lambda=5$: $A-5I=\begin{bmatrix}1&-2\\-2&4\end{bmatrix}$, so $x_1=2x_2$ and $\mathbf{v}_1=\begin{bmatrix}2\\1\end{bmatrix}$. $\lambda=10$: $A-10I=\begin{bmatrix}-4&-2\\-2&-1\end{bmatrix}$, so $x_2=-2x_1$ and $\mathbf{v}_2=\begin{bmatrix}-1\\2\end{bmatrix}$. Indeed $\mathbf{v}_1\cdot\mathbf{v}_2=-2+2=0$.
Normalise Both have length $\sqrt5$: $$P=\frac{1}{\sqrt5}\begin{bmatrix}2&-1\\1&2\end{bmatrix},\qquad D=\begin{bmatrix}5&0\\0&10\end{bmatrix},\qquad A=PDP^{T}.$$
Check $PD=\tfrac1{\sqrt5}\begin{bmatrix}10&-10\\5&20\end{bmatrix}$ and $PDP^{T}=\tfrac15\begin{bmatrix}10&-10\\5&20\end{bmatrix}\begin{bmatrix}2&1\\-1&2\end{bmatrix}=\tfrac15\begin{bmatrix}30&-10\\-10&45\end{bmatrix}=\begin{bmatrix}6&-2\\-2&9\end{bmatrix}$. ✓
✏️ Example 2 — A repeated eigenvalue needs Gram–Schmidt
Eigenvalues Expanding (or noting $A=I+J$ with $J$ the all-ones matrix), $\det(A-\lambda I)=(4-\lambda)(1-\lambda)^2$. So $\lambda=4$ (once) and $\lambda=1$ (twice).
$E_4$ $A-4I=\begin{bmatrix}-2&1&1\\1&-2&1\\1&1&-2\end{bmatrix}$ row-reduces to $x_1=x_3$, $x_2=x_3$: $E_4=\operatorname{span}\{(1,1,1)\}$.
$E_1$ $A-I$ has every row equal to $(1,1,1)$, so $E_1$ is the plane $x_1+x_2+x_3=0$, with basis $\mathbf{w}_1=(1,-1,0)$, $\mathbf{w}_2=(1,0,-1)$. These are eigenvectors but not orthogonal ($\mathbf{w}_1\cdot\mathbf{w}_2=1$). Gram–Schmidt inside $E_1$: $$\mathbf{w}_2-\frac{\mathbf{w}_2\cdot\mathbf{w}_1}{\mathbf{w}_1\cdot\mathbf{w}_1}\mathbf{w}_1=(1,0,-1)-\tfrac12(1,-1,0)=\left(\tfrac12,\tfrac12,-1\right),$$ which we scale to $(1,1,-2)$. It is still in $E_1$, so still an eigenvector for $\lambda=1$.
Assemble $(1,1,1)$ is automatically orthogonal to both (Theorem 8.15). Normalising, $$P=\begin{bmatrix}\tfrac1{\sqrt3}&\tfrac1{\sqrt2}&\tfrac1{\sqrt6}\\\tfrac1{\sqrt3}&-\tfrac1{\sqrt2}&\tfrac1{\sqrt6}\\\tfrac1{\sqrt3}&0&-\tfrac2{\sqrt6}\end{bmatrix},\qquad D=\begin{bmatrix}4&0&0\\0&1&0\\0&0&1\end{bmatrix},\qquad A=PDP^{T}.$$
Check The columns of $P$ are unit vectors with zero dot products, so $P^{T}P=I$, and $A\mathbf{p}_1=4\mathbf{p}_1$, $A\mathbf{p}_2=\mathbf{p}_2$, $A\mathbf{p}_3=\mathbf{p}_3$ (e.g. $A(1,1,-2)=(1,1,-2)$). ✓
📐 Theorem 8.16 — The Spectral Theorem for symmetric matrices
Let $A$ be a real $n\times n$ symmetric matrix. Then:
$A$ has $n$ real eigenvalues, counted with multiplicity;
for each eigenvalue $\lambda$, the dimension of the eigenspace $E_\lambda$ equals the multiplicity of $\lambda$ as a root of $\det(A-\lambda I)$;
the eigenspaces are mutually orthogonal: eigenvectors from different eigenspaces are orthogonal;
$A$ is orthogonally diagonalisable: $A=PDP^{T}$ with $P$ orthogonal and $D$ diagonal.
Conversely, every orthogonally diagonalisable matrix is symmetric, because $(PDP^{T})^{T}=(P^{T})^{T}D^{T}P^{T}=PDP^{T}$. So for real matrices, symmetric and orthogonally diagonalisable mean exactly the same thing.
Check that $A$ is symmetric (otherwise it cannot be orthogonally diagonalised).
Find the eigenvalues from $\det(A-\lambda I)=0$.
For each eigenvalue find a basis of $E_\lambda$. If $\dim E_\lambda\ge2$, apply Gram–Schmidt within $E_\lambda$ to make that basis orthogonal. (Vectors from different eigenspaces are already orthogonal.)
Normalise every vector, place them as the columns of $P$, and put the matching eigenvalues on the diagonal of $D$ in the same order.
Check: $P^{T}P=I$ and $AP=PD$.
Interactive 1 — A Symmetric Matrix Turns a Circle into an Ellipse
The dashed circle is the set of unit vectors $\mathbf{x}$; the blue curve is the set of images $A\mathbf{x}$ for $A=\begin{bmatrix}a&b\\b&d\end{bmatrix}$. Press Play: the red vector $\mathbf{x}$ runs round the circle and the blue vector $A\mathbf{x}$ follows. Only along the two purple eigenvector lines do $\mathbf{x}$ and $A\mathbf{x}$ point the same way (or exactly opposite) — and those lines are always perpendicular and are the axes of the ellipse, with semi-axes $|\lambda_1|$ and $|\lambda_2|$.
$A$—
eigenvalues $\lambda_1,\lambda_2$—
unit eigenvectors $\mathbf{u}_1,\mathbf{u}_2$—
$\mathbf{u}_1\cdot\mathbf{u}_2$—
angle from $\mathbf{x}$ to $A\mathbf{x}$—
The Spectral Decomposition
Write $P=\begin{bmatrix}\mathbf{u}_1&\cdots&\mathbf{u}_n\end{bmatrix}$. Multiplying out $A=PDP^{T}$ column by row gives a second, very useful, way of writing a symmetric matrix:
$$A=\begin{bmatrix}\lambda_1\mathbf{u}_1&\cdots&\lambda_n\mathbf{u}_n\end{bmatrix}\begin{bmatrix}\mathbf{u}_1^{T}\\\vdots\\\mathbf{u}_n^{T}\end{bmatrix}=\lambda_1\mathbf{u}_1\mathbf{u}_1^{T}+\lambda_2\mathbf{u}_2\mathbf{u}_2^{T}+\cdots+\lambda_n\mathbf{u}_n\mathbf{u}_n^{T}.$$
📐 Theorem 8.17 — Spectral decomposition
If $A$ is a real symmetric $n\times n$ matrix with orthogonal diagonalisation $A=PDP^{T}$, where $P$ has orthonormal columns $\mathbf{u}_1,\dots,\mathbf{u}_n$ and $D=\operatorname{diag}(\lambda_1,\dots,\lambda_n)$, then
$$A=\sum_{i=1}^{n}\lambda_i\,\mathbf{u}_i\mathbf{u}_i^{T}.$$
Each $\mathbf{u}_i\mathbf{u}_i^{T}$ is an $n\times n$ matrix of rank $1$; it is the projection matrix onto the line $\operatorname{span}\{\mathbf{u}_i\}$ (Theorem 8.11 with $U=\mathbf{u}_i$). So $A$ projects onto each eigen-direction, scales by the eigenvalue, and adds up.
✏️ Example 3 — Spectral decomposition and powers
Write the spectral decomposition of $A=\begin{bmatrix}6&-2\\-2&9\end{bmatrix}$ from Example 1, and use it to find $A^3$.
💡 Show solution▶
Rank-one pieces With $\mathbf{u}_1=\tfrac1{\sqrt5}(2,1)$ and $\mathbf{u}_2=\tfrac1{\sqrt5}(-1,2)$: $$\mathbf{u}_1\mathbf{u}_1^{T}=\frac15\begin{bmatrix}4&2\\2&1\end{bmatrix},\qquad\mathbf{u}_2\mathbf{u}_2^{T}=\frac15\begin{bmatrix}1&-2\\-2&4\end{bmatrix}.$$
Powers Because $\mathbf{u}_i^{T}\mathbf{u}_j=0$ for $i\ne j$ and $=1$ for $i=j$, cross terms vanish when we multiply, and $A^k=\lambda_1^k\mathbf{u}_1\mathbf{u}_1^{T}+\lambda_2^k\mathbf{u}_2\mathbf{u}_2^{T}$. So $$A^3=\frac{125}{5}\begin{bmatrix}4&2\\2&1\end{bmatrix}+\frac{1000}{5}\begin{bmatrix}1&-2\\-2&4\end{bmatrix}=\begin{bmatrix}100&50\\50&25\end{bmatrix}+\begin{bmatrix}200&-400\\-400&800\end{bmatrix}=\begin{bmatrix}300&-350\\-350&825\end{bmatrix}.$$
Check $A^2=\begin{bmatrix}40&-30\\-30&85\end{bmatrix}$ and $A^3=A^2A=\begin{bmatrix}240+60&-80-270\\-180-170&60+765\end{bmatrix}=\begin{bmatrix}300&-350\\-350&825\end{bmatrix}$. ✓
Quadratic Forms and the Principal Axes Theorem
Expressions such as $6x_1^2-4x_1x_2+9x_2^2$ — every term of degree exactly two — appear as energies in physics, as variances in statistics and as the second-order terms of a function near a critical point. Each one can be written with a symmetric matrix.
📘 Definition 8.11 — Quadratic form
A quadratic form on $\mathbb{R}^n$ is a function $Q(\mathbf{x})=\mathbf{x}^{T}A\mathbf{x}$, where $A$ is a symmetric $n\times n$ matrix, called the matrix of the form. The coefficient of $x_i^2$ goes on the diagonal as $a_{ii}$, and the coefficient of the cross term $x_ix_j$ ($i\ne j$) is split in half between $a_{ij}$ and $a_{ji}$. For example
$$6x_1^2-4x_1x_2+9x_2^2=\begin{bmatrix}x_1&x_2\end{bmatrix}\begin{bmatrix}6&-2\\-2&9\end{bmatrix}\begin{bmatrix}x_1\\x_2\end{bmatrix}.$$
The cross term $x_1x_2$ is what makes a quadratic form hard to understand. Change variables by $\mathbf{x}=P\mathbf{y}$, where $P$ orthogonally diagonalises $A$. Then
$$\mathbf{x}^{T}A\mathbf{x}=(P\mathbf{y})^{T}A(P\mathbf{y})=\mathbf{y}^{T}\left(P^{T}AP\right)\mathbf{y}=\mathbf{y}^{T}D\mathbf{y}=\lambda_1y_1^2+\lambda_2y_2^2+\cdots+\lambda_ny_n^2,$$
and the cross terms have disappeared. Since $P$ is orthogonal, the new coordinate axes (the columns of $P$) are just the old axes rotated (and possibly reflected): lengths and angles are unchanged.
📐 Theorem 8.18 — The Principal Axes Theorem
Let $A$ be a real symmetric $n\times n$ matrix. Then there is an orthogonal change of variable $\mathbf{x}=P\mathbf{y}$ that transforms the quadratic form $\mathbf{x}^{T}A\mathbf{x}$ into a form with no cross terms:
$$\mathbf{x}^{T}A\mathbf{x}=\mathbf{y}^{T}D\mathbf{y}=\lambda_1y_1^2+\cdots+\lambda_ny_n^2,$$
where $\lambda_1,\dots,\lambda_n$ are the eigenvalues of $A$ and the columns of $P$ are corresponding orthonormal eigenvectors. These columns are called the principal axes of the form.
✏️ Example 4 — A tilted ellipse
Identify the curve $6x_1^2-4x_1x_2+9x_2^2=20$, find its principal axes and its semi-axis lengths.
💡 Show solution▶
Matrix The left side is $\mathbf{x}^{T}A\mathbf{x}$ with $A=\begin{bmatrix}6&-2\\-2&9\end{bmatrix}$ — the matrix of Example 1, with eigenvalues $5,10$ and $P=\tfrac1{\sqrt5}\begin{bmatrix}2&-1\\1&2\end{bmatrix}$.
Rotate With $\mathbf{x}=P\mathbf{y}$ the equation becomes $5y_1^2+10y_2^2=20$, i.e. $$\frac{y_1^2}{4}+\frac{y_2^2}{2}=1 .$$ This is an ellipse.
Axes The $y_1$-axis points along $\mathbf{u}_1=\tfrac1{\sqrt5}(2,1)$, at angle $\tan^{-1}\tfrac12\approx26.6^\circ$ to the $x_1$-axis; the semi-axis there is $2$ (major axis). Along $\mathbf{u}_2=\tfrac1{\sqrt5}(-1,2)$ the semi-axis is $\sqrt2\approx1.414$ (minor axis).
Check The end of the major axis is $\mathbf{x}=2\mathbf{u}_1=\tfrac1{\sqrt5}(4,2)$: $6\cdot\tfrac{16}{5}-4\cdot\tfrac85+9\cdot\tfrac45=\tfrac{96-32+36}{5}=20$. ✓ Choose Example 4 in Interactive 2 to see the ellipse with its axes.
✏️ Example 5 — A tilted hyperbola
Identify the curve $x_1^2+4x_1x_2-2x_2^2=6$ and find its principal axes.
💡 Show solution▶
Matrix Split the cross coefficient $4$ in half: $A=\begin{bmatrix}1&2\\2&-2\end{bmatrix}$.
Eigen $\det(A-\lambda I)=(1-\lambda)(-2-\lambda)-4=\lambda^2+\lambda-6=(\lambda-2)(\lambda+3)$. For $\lambda=2$: $A-2I=\begin{bmatrix}-1&2\\2&-4\end{bmatrix}$, $\mathbf{u}_1=\tfrac1{\sqrt5}(2,1)$. For $\lambda=-3$: $A+3I=\begin{bmatrix}4&2\\2&1\end{bmatrix}$, $\mathbf{u}_2=\tfrac1{\sqrt5}(-1,2)$.
Rotate With $\mathbf{x}=P\mathbf{y}$: $2y_1^2-3y_2^2=6$, i.e. $\dfrac{y_1^2}{3}-\dfrac{y_2^2}{2}=1$ — a hyperbola opening along the $y_1$-axis (direction $(2,1)$), with vertices at $y_1=\pm\sqrt3$, that is $\mathbf{x}=\pm\sqrt3\,\mathbf{u}_1=\pm\sqrt{\tfrac35}\,(2,1)$.
Check At $\mathbf{x}=\sqrt{\tfrac35}(2,1)$: $x_1^2+4x_1x_2-2x_2^2=\tfrac35(4+8-2)=6$. ✓ One positive and one negative eigenvalue always gives a hyperbola.
Interactive 2 — The Conic $\mathbf{x}^{T}A\mathbf{x}=1$ and Its Principal Axes
Choose a quadratic form $Q(\mathbf{x})=ax_1^2+2bx_1x_2+dx_2^2$ (presets are scaled so the right side is $1$). The curve $Q(\mathbf{x})=1$ is drawn in blue and its principal axes $\mathbf{u}_1,\mathbf{u}_2$ in purple. Press Play to rotate the eigenvectors while keeping the eigenvalues fixed: the curve turns rigidly but never changes shape — the shape is set by the eigenvalues, the tilt by the eigenvectors. Move $b$ to watch a cross term tilt the curve, and make $ad-b^2$ change sign to turn an ellipse into a hyperbola.
$Q(\mathbf{x})$—
eigenvalues—
in principal axes—
curve / form—
tilt of $\mathbf{u}_1$—
Classifying quadratic forms
Whether a quadratic form can take negative values decides, for example, whether a critical point of a function is a minimum, whether an equilibrium of a structure is stable, and whether a matrix can be a covariance matrix.
📘 Definition 8.12 — Definiteness
A quadratic form $Q(\mathbf{x})=\mathbf{x}^{T}A\mathbf{x}$ (and its symmetric matrix $A$) is
positive definite if $Q(\mathbf{x})\gt0$ for every $\mathbf{x}\ne\mathbf{0}$;
negative definite if $Q(\mathbf{x})\lt0$ for every $\mathbf{x}\ne\mathbf{0}$;
indefinite if $Q$ takes both positive and negative values;
positive semidefinite if $Q(\mathbf{x})\ge0$ for all $\mathbf{x}$, and negative semidefinite if $Q(\mathbf{x})\le0$ for all $\mathbf{x}$.
📐 Theorem 8.19 — Classification by eigenvalues
Let $A$ be a real symmetric $n\times n$ matrix. The quadratic form $\mathbf{x}^{T}A\mathbf{x}$ is
positive definite if and only if all eigenvalues of $A$ are positive;
negative definite if and only if all eigenvalues of $A$ are negative;
indefinite if and only if $A$ has at least one positive and at least one negative eigenvalue;
positive (negative) semidefinite if and only if all eigenvalues are $\ge0$ ($\le0$).
Moreover, the largest value of $\mathbf{x}^{T}A\mathbf{x}$ over unit vectors ($\|\mathbf{x}\|=1$) is the largest eigenvalue $\lambda_{\max}$ and the smallest is $\lambda_{\min}$; they are attained at the corresponding unit eigenvectors.
Why. In principal axes $Q=\lambda_1y_1^2+\cdots+\lambda_ny_n^2$, and $\mathbf{y}=P^{T}\mathbf{x}\ne\mathbf{0}$ exactly when $\mathbf{x}\ne\mathbf{0}$. A sum $\sum\lambda_iy_i^2$ is positive for every nonzero $\mathbf{y}$ exactly when every $\lambda_i\gt0$, and so on. If $\|\mathbf{x}\|=1$ then also $\|\mathbf{y}\|=1$ (orthogonal matrices preserve length), and $\lambda_{\min}\le\sum\lambda_iy_i^2\le\lambda_{\max}$ because $\sum y_i^2=1$.
For $2\times2$ matrices there is a quick test: $\lambda_1\lambda_2=\det A=ad-b^2$ and $\lambda_1+\lambda_2=a+d$. So $A=\begin{bmatrix}a&b\\b&d\end{bmatrix}$ is positive definite exactly when $\det A\gt0$ and $a\gt0$; indefinite exactly when $\det A\lt0$. For conics $\mathbf{x}^{T}A\mathbf{x}=1$: positive definite gives an ellipse, indefinite (with $\det A\ne0$) gives a hyperbola, and negative definite gives no points at all.
(a) Matrix $A=\begin{bmatrix}2&-1&0\\-1&2&-1\\0&-1&2\end{bmatrix}$ (each cross coefficient $-2$ split into $-1,-1$). This matrix appears whenever a string or a chain of springs is modelled by $3$ masses.
(a) Eigenvalues Expanding along the first row, $\det(A-\lambda I)=(2-\lambda)\left[(2-\lambda)^2-1\right]-(2-\lambda)=(2-\lambda)\left[(2-\lambda)^2-2\right]$. So $\lambda=2$ or $2-\lambda=\pm\sqrt2$: $\lambda=2-\sqrt2\approx0.586,\ 2,\ 2+\sqrt2\approx3.414$. All positive: $Q$ is positive definite.
(b) $A=\begin{bmatrix}1&3\\3&1\end{bmatrix}$, $\det A=1-9=-8\lt0$, so the eigenvalues ($4$ and $-2$) have opposite signs and $Q$ is indefinite: $Q(1,1)=8\gt0$ but $Q(1,-1)=-4\lt0$.
Check For (a), $Q(\mathbf{x})=x_1^2+(x_1-x_2)^2+(x_2-x_3)^2+x_3^2$, which is clearly positive unless $\mathbf{x}=\mathbf{0}$. ✓ (By Theorem 8.19 its minimum on the unit sphere is $2-\sqrt2$.)
⚠️ Common mistakes with symmetric matrices and quadratic forms
1. Forgetting to halve the cross term. $x_1^2+6x_1x_2+x_2^2$ has matrix $\begin{bmatrix}1&3\\3&1\end{bmatrix}$, not $\begin{bmatrix}1&6\\6&1\end{bmatrix}$ — the $3$ appears twice when you multiply out $\mathbf{x}^{T}A\mathbf{x}$.
2. Positive entries do not mean positive definite. $\begin{bmatrix}1&3\\3&1\end{bmatrix}$ has only positive entries but is indefinite. Look at the eigenvalues (or, for $2\times2$, at $\det A$ and $a$).
3. Normalise the eigenvectors. $P=\begin{bmatrix}2&-1\\1&2\end{bmatrix}$ has orthogonal but not orthonormal columns, so $P^{T}\ne P^{-1}$ and $PDP^{T}\ne A$.
4. Repeated eigenvalues need Gram–Schmidt. Two eigenvectors from the same eigenspace need not be orthogonal (Example 2); only eigenvectors from different eigenspaces are automatically orthogonal.
A Look Ahead: the Singular Value Decomposition
Most matrices met in practice are not symmetric — or even square. A digital greyscale photograph, for instance, is an $m\times n$ matrix of brightness values. The key trick is that for any matrix $A$, the matrix $A^{T}A$ is symmetric (since $(A^{T}A)^{T}=A^{T}A$) and its eigenvalues are never negative (Exercise I.8). So the Spectral Theorem applies to $A^{T}A$.
📘 The SVD in brief (no proofs)
Let $A$ be any $m\times n$ matrix. The singular values of $A$ are $\sigma_i=\sqrt{\lambda_i}$, where $\lambda_1\ge\lambda_2\ge\cdots\ge0$ are the eigenvalues of $A^{T}A$. If $A$ has rank $r$, it can be written as
$$A=U\Sigma V^{T}=\sigma_1\mathbf{u}_1\mathbf{v}_1^{T}+\sigma_2\mathbf{u}_2\mathbf{v}_2^{T}+\cdots+\sigma_r\mathbf{u}_r\mathbf{v}_r^{T},$$
where $U$ ($m\times m$) and $V$ ($n\times n$) are orthogonal, the $\mathbf{v}_i$ are orthonormal eigenvectors of $A^{T}A$, $\mathbf{u}_i=\tfrac{1}{\sigma_i}A\mathbf{v}_i$, and $\Sigma$ is $m\times n$ with $\sigma_1,\dots,\sigma_r$ on its diagonal and zeros elsewhere. Keeping only the first $k$ terms gives the best possible approximation of $A$ by a matrix of rank $k$.
For example, $A=\begin{bmatrix}3&0\\4&5\end{bmatrix}$ is not symmetric, but $A^{T}A=\begin{bmatrix}25&20\\20&25\end{bmatrix}$ has eigenvalues $45$ and $5$, so the singular values are $\sigma_1=3\sqrt5\approx6.708$ and $\sigma_2=\sqrt5\approx2.236$. Geometrically, $A$ maps the unit circle onto an ellipse with semi-axes $\sigma_1$ and $\sigma_2$ — just like Interactive 1, except the ellipse's axes are now the $\mathbf{u}_i$, not the input directions $\mathbf{v}_i$.
Image compression. A $1000\times800$ photograph needs $800\,000$ numbers. In most photographs the singular values fall off quickly, so keeping only the first $k=50$ terms $\sigma_i\mathbf{u}_i\mathbf{v}_i^{T}$ already gives a picture that looks almost the same — and it needs only $k(m+n+1)=50(1801)=90\,050$ numbers, about $11\%$ of the original. The same idea powers recommendation systems, noise reduction and principal component analysis in data science.
Exercises
I
Symmetric matrices and orthogonal diagonalisation.
1 Which of these matrices are symmetric? Which can be orthogonally diagonalised? $\ \text{(a)}\begin{bmatrix}1&2\\2&3\end{bmatrix}\ \ \text{(b)}\begin{bmatrix}0&1\\-1&0\end{bmatrix}\ \ \text{(c)}\begin{bmatrix}1&2&3\\2&4&5\\3&5&6\end{bmatrix}\ \ \text{(d)}\begin{bmatrix}1&0\\2&1\end{bmatrix}$
6 Orthogonally diagonalise $\begin{bmatrix}0&1&1\\1&0&1\\1&1&0\end{bmatrix}$. (A repeated eigenvalue — you will need Gram–Schmidt.)
7 For the matrix of part 3, verify directly that eigenvectors for the two eigenvalues are orthogonal, and that $P^{T}P=I$.
8Proof. Let $A$ be any $m\times n$ matrix. Show that $A^{T}A$ is symmetric and that all of its eigenvalues are $\ge0$. (Hint: if $A^{T}A\mathbf{v}=\lambda\mathbf{v}$, compute $\|A\mathbf{v}\|^2$.)
💡 Show solutions▶
1 (a) and (c) are symmetric, so by the Spectral Theorem they can be orthogonally diagonalised. (b) is not symmetric (its eigenvalues are $\pm i$, not even real); (d) is not symmetric (it has only one independent eigenvector, so it is not diagonalisable at all). By the converse in Theorem 8.16, neither (b) nor (d) is orthogonally diagonalisable.
5 The $x_2$-coordinate separates: $\lambda=3$ with $(0,1,0)$. The block $\begin{bmatrix}1&2\\2&1\end{bmatrix}$ in $x_1,x_3$ has eigenvalues $3$ ($(1,0,1)$) and $-1$ ($(1,0,-1)$). So $\lambda=3,3,-1$, and $(1,0,1)$, $(0,1,0)$ are already orthogonal. $$P=\begin{bmatrix}\tfrac1{\sqrt2}&0&\tfrac1{\sqrt2}\\0&1&0\\\tfrac1{\sqrt2}&0&-\tfrac1{\sqrt2}\end{bmatrix},\quad D=\begin{bmatrix}3&0&0\\0&3&0\\0&0&-1\end{bmatrix}.$$
6 $A=J-I$, so $\lambda=3-1=2$ (eigenvector $(1,1,1)$) and $\lambda=0-1=-1$ twice ($E_{-1}$: $x_1+x_2+x_3=0$). Gram–Schmidt on $(1,-1,0),(1,0,-1)$ gives $(1,-1,0),(1,1,-2)$ as in Example 2. $$P=\begin{bmatrix}\tfrac1{\sqrt3}&\tfrac1{\sqrt2}&\tfrac1{\sqrt6}\\\tfrac1{\sqrt3}&-\tfrac1{\sqrt2}&\tfrac1{\sqrt6}\\\tfrac1{\sqrt3}&0&-\tfrac2{\sqrt6}\end{bmatrix},\quad D=\begin{bmatrix}2&0&0\\0&-1&0\\0&0&-1\end{bmatrix}.$$
8 $(A^{T}A)^{T}=A^{T}(A^{T})^{T}=A^{T}A$, so it is symmetric. If $A^{T}A\mathbf{v}=\lambda\mathbf{v}$ with $\mathbf{v}\ne\mathbf{0}$, then $\|A\mathbf{v}\|^2=(A\mathbf{v})^{T}(A\mathbf{v})=\mathbf{v}^{T}A^{T}A\mathbf{v}=\lambda\,\mathbf{v}^{T}\mathbf{v}=\lambda\|\mathbf{v}\|^2$. Since $\|A\mathbf{v}\|^2\ge0$ and $\|\mathbf{v}\|^2\gt0$, $\lambda\ge0$.
II
Spectral decompositions.
1 Write the spectral decomposition of $\begin{bmatrix}3&1\\1&3\end{bmatrix}$ (Ex. I.2).
2 Write the spectral decomposition of $\begin{bmatrix}7&24\\24&-7\end{bmatrix}$ (Ex. I.4).
3 Find the symmetric $2\times2$ matrix with eigenvalues $4$ and $-2$ for which $(1,1)$ is an eigenvector for $\lambda=4$.
4 Use the spectral decomposition from part 1 to compute $\begin{bmatrix}3&1\\1&3\end{bmatrix}^5$.
5 For $A=\begin{bmatrix}0&1&1\\1&0&1\\1&1&0\end{bmatrix}$ (Ex. I.6) show that $A=2\,\mathbf{w}\mathbf{w}^{T}-(I-\mathbf{w}\mathbf{w}^{T})$ with $\mathbf{w}=\tfrac1{\sqrt3}(1,1,1)$, and explain what $I-\mathbf{w}\mathbf{w}^{T}$ is.
6Rank-one approximation. For $A=\begin{bmatrix}6&-2\\-2&9\end{bmatrix}$, keep only the term of the spectral decomposition with the larger eigenvalue. Write down this rank-one matrix $A_1$ and the error matrix $A-A_1$.
3 The other eigenvector must be orthogonal: $(1,-1)$. $A=4\cdot\tfrac12\begin{bmatrix}1&1\\1&1\end{bmatrix}-2\cdot\tfrac12\begin{bmatrix}1&-1\\-1&1\end{bmatrix}=\begin{bmatrix}1&3\\3&1\end{bmatrix}$. (Check: $A(1,1)=(4,4)$, $A(1,-1)=(-2,2)$.)
5 $\mathbf{w}\mathbf{w}^{T}=\tfrac13J$ ($J$ = all ones), so $2\cdot\tfrac13J-\left(I-\tfrac13J\right)=J-I=A$. ✓ $I-\mathbf{w}\mathbf{w}^{T}$ is the projection matrix onto the plane $E_{-1}=\{x_1+x_2+x_3=0\}$ (Exercise II.2 of Section 8.3); it equals $\mathbf{u}_2\mathbf{u}_2^{T}+\mathbf{u}_3\mathbf{u}_3^{T}$ for any orthonormal basis of that eigenspace.
6 $A_1=10\,\mathbf{u}_2\mathbf{u}_2^{T}=\begin{bmatrix}2&-4\\-4&8\end{bmatrix}$ (from Example 3) and $A-A_1=5\,\mathbf{u}_1\mathbf{u}_1^{T}=\begin{bmatrix}4&2\\2&1\end{bmatrix}$. The dropped term has the smaller eigenvalue, which is why this is the best rank-one approximation.
III
Quadratic forms.
1 Write the matrix of $Q(\mathbf{x})=4x_1^2-6x_1x_2+x_2^2$.
2 Write the matrix of $Q(\mathbf{x})=x_1^2+2x_2^2-3x_3^2+4x_1x_2-2x_2x_3$.
3 For $A=\begin{bmatrix}2&1\\1&3\end{bmatrix}$ evaluate $\mathbf{x}^{T}A\mathbf{x}$ at $\mathbf{x}=(1,-2)$.
4 Find an orthogonal change of variable that removes the cross term from $Q=3x_1^2+2x_1x_2+3x_2^2$, and write $Q$ in the new variables.
5 Do the same for $Q=7x_1^2+48x_1x_2-7x_2^2$.
6 Classify $Q=2x_1^2-2x_1x_2+2x_2^2$.
7 Classify $Q=x_1^2+6x_1x_2+x_2^2$ and find a vector where $Q\lt0$.
8 Classify $Q=x_1^2-4x_1x_2+4x_2^2$, and find a nonzero $\mathbf{x}$ with $Q(\mathbf{x})=0$.
9 Classify $Q=-2x_1^2-2x_2^2-x_3^2+2x_1x_2$.
10 For which values of $k$ is $\begin{bmatrix}2&k\\k&8\end{bmatrix}$ positive definite?
Check: $Q$ on the unit circle — its maximum and minimum are the eigenvalues
$Q(\cos\theta,\sin\theta)$—
max / min on the circle—
classification—
💡 Show solutions▶
1 $\begin{bmatrix}4&-3\\-3&1\end{bmatrix}$ (the cross coefficient $-6$ is split into $-3,-3$).
2 $\begin{bmatrix}1&2&0\\2&2&-1\\0&-1&-3\end{bmatrix}$ ($4x_1x_2\to a_{12}=a_{21}=2$; $-2x_2x_3\to a_{23}=a_{32}=-1$; no $x_1x_3$ term).
3 $A\mathbf{x}=(0,-5)$, so $\mathbf{x}^{T}A\mathbf{x}=1\cdot0+(-2)(-5)=10$. (Or: $2(1)^2+2(1)(-2)+3(-2)^2=2-4+12=10$.)
7 $A=\begin{bmatrix}1&3\\3&1\end{bmatrix}$, eigenvalues $4,-2$: indefinite. $Q(1,-1)=1-6+1=-4\lt0$ (an eigenvector for $-2$).
8 $A=\begin{bmatrix}1&-2\\-2&4\end{bmatrix}$, eigenvalues $0$ and $5$: positive semidefinite. Indeed $Q=(x_1-2x_2)^2$, which is $0$ at $\mathbf{x}=(2,1)$ (an eigenvector for $\lambda=0$).
9 $A=\begin{bmatrix}-2&1&0\\1&-2&0\\0&0&-1\end{bmatrix}$. The $2\times2$ block has eigenvalues $-1,-3$ and the last entry gives $-1$: eigenvalues $-1,-1,-3$, all negative: negative definite.
10 Need $a=2\gt0$ and $\det=16-k^2\gt0$: $-4\lt k\lt4$.
IV
Conics and applications.
1 Identify the curve $5x_1^2-4x_1x_2+8x_2^2=36$. Give its principal axes and semi-axis lengths.
2 Identify the curve $x_1^2+4x_1x_2+x_2^2=3$ and find its vertices.
3 Find the maximum and minimum of $Q=5x_1^2-4x_1x_2+8x_2^2$ subject to $x_1^2+x_2^2=1$, and where they occur.
4Engineering. At a point of a steel plate the stress matrix is $\begin{bmatrix}50&30\\30&-30\end{bmatrix}$ MPa. Find the principal stresses (eigenvalues) and principal directions. Which direction is pulled hardest?
5Data science. The covariance matrix of two measured quantities is $C=\begin{bmatrix}9&4\\4&3\end{bmatrix}$. Find the first principal component direction (the unit eigenvector for the largest eigenvalue) and the fraction of the total variance $\operatorname{tr}C$ that it explains.
6 Find the singular values of $A=\begin{bmatrix}3&0\\4&5\end{bmatrix}$ and check that $\sigma_1\sigma_2=|\det A|$.
7Image compression. A $1000\times800$ greyscale image is replaced by the sum of its first $50$ SVD terms $\sigma_i\mathbf{u}_i\mathbf{v}_i^{T}$. How many numbers must be stored, and what percentage of the original is that?
💡 Show solutions▶
1 $A=\begin{bmatrix}5&-2\\-2&8\end{bmatrix}$, $\lambda^2-13\lambda+36=(\lambda-4)(\lambda-9)$. $\lambda=4$: $\mathbf{u}_1=\tfrac1{\sqrt5}(2,1)$; $\lambda=9$: $\mathbf{u}_2=\tfrac1{\sqrt5}(-1,2)$. In principal axes $4y_1^2+9y_2^2=36$, i.e. $\dfrac{y_1^2}{9}+\dfrac{y_2^2}{4}=1$: an ellipse with semi-axis $3$ along $\mathbf{u}_1$ and $2$ along $\mathbf{u}_2$.
2 $A=\begin{bmatrix}1&2\\2&1\end{bmatrix}$, eigenvalues $3$ ($\mathbf{u}_1=\tfrac1{\sqrt2}(1,1)$) and $-1$ ($\mathbf{u}_2=\tfrac1{\sqrt2}(-1,1)$). $3y_1^2-y_2^2=3$, i.e. $y_1^2-\dfrac{y_2^2}{3}=1$: a hyperbola with vertices $y_1=\pm1$, $y_2=0$, i.e. $\mathbf{x}=\pm\tfrac1{\sqrt2}(1,1)$. (Check: $\tfrac12+2+\tfrac12=3$.)
3 By Theorem 8.19 with the eigenvalues of part 1: maximum $9$ at $\mathbf{x}=\pm\tfrac1{\sqrt5}(-1,2)$, minimum $4$ at $\mathbf{x}=\pm\tfrac1{\sqrt5}(2,1)$. (Check: $Q(-1,2)=5+8+32=45=9\cdot5$.)
4 $\lambda^2-20\lambda-2400=0$, $\lambda=10\pm50$: principal stresses $60$ MPa and $-40$ MPa. $\lambda=60$: $\begin{bmatrix}-10&30\\30&-90\end{bmatrix}$ gives $(3,1)$; $\lambda=-40$: $(-1,3)$. The plate is pulled hardest (tension $60$ MPa) along $\tfrac1{\sqrt{10}}(3,1)$, about $18.4^\circ$ above the $x$-axis, and compressed ($40$ MPa) in the perpendicular direction.
5 $\lambda^2-12\lambda+11=(\lambda-11)(\lambda-1)$. $\lambda=11$: $C-11I=\begin{bmatrix}-2&4\\4&-8\end{bmatrix}$ gives $\tfrac1{\sqrt5}(2,1)$. It explains $\tfrac{11}{12}\approx91.7\%$ of the total variance $\operatorname{tr}C=12$ — the data lie close to a line in that direction.
6 $A^{T}A=\begin{bmatrix}25&20\\20&25\end{bmatrix}$, $\lambda=45,5$, so $\sigma_1=3\sqrt5\approx6.708$, $\sigma_2=\sqrt5\approx2.236$. $\sigma_1\sigma_2=15=|\det A|=|15-0|$. ✓ (In general $\sigma_1\sigma_2=\sqrt{\det(A^{T}A)}=|\det A|$.)
7 Each term needs $\mathbf{u}_i$ ($1000$ numbers), $\mathbf{v}_i$ ($800$) and $\sigma_i$ ($1$): $50\times1801=90\,050$ numbers, versus $800\,000$ — about $11.3\%$.
🔬 Research–Teaching NexusResearch-tutored · optional, about 30 minutes
Research question. In which direction is a loaded machine part most likely to crack, and is it safe? How do engineers turn a stress measurement into a safety factor?
Why it matters. Strain gauges glued to aircraft wings, wind-turbine blades and bridge girders measure stresses in fixed $x$ and $y$ directions, but cracks grow across the direction of greatest tension, which is usually tilted. The stress matrix is always symmetric, so its eigenvalues (principal stresses) and perpendicular eigenvectors (principal directions) give the answer. Engineers traditionally find them with a drawing called Mohr's circle — which is the Spectral Theorem in disguise.
What to do.
Sample case: gauges on a steel bracket give the plane stress matrix $\begin{bmatrix}80&-30\\-30&0\end{bmatrix}$ MPa.
Find the principal stresses $\sigma_1\ge\sigma_2$ and the principal directions, and the angle of the first one to the $x$-axis.
Mohr's circle has centre $\tfrac{\sigma_x+\sigma_y}{2}$ and radius $\sqrt{\left(\tfrac{\sigma_x-\sigma_y}{2}\right)^2+\tau^2}$. Show that centre $\pm$ radius gives exactly your eigenvalues (compare the quadratic formula for $\det(A-\lambda I)=0$).
The Tresca criterion says the steel yields when the largest shear stress reaches half the yield strength. With the out-of-plane stress equal to $0$, the largest shear stress is half the largest difference between $\sigma_1$, $\sigma_2$ and $0$. For a steel with yield strength $250$ MPa, find the safety factor.
Tools: Calculator; optionally a spreadsheet or Python to draw Mohr's circle.
Hand in: A half-page note with the principal stresses and directions, a sketch of the bracket showing the direction in which a crack would open, and the safety factor with one sentence on whether you would approve the design.
Going further. In 3-D the stress matrix is $3\times3$ symmetric with three principal stresses. Look up the von Mises criterion, which is written in terms of the eigenvalues, and compare it with Tresca for this bracket.
Instructor note: $\lambda^2-80\lambda-900=0$ gives $\sigma_1=90$ MPa along $(3,-1)/\sqrt{10}$ (about $-18.4^\circ$) and $\sigma_2=-10$ MPa along $(1,3)/\sqrt{10}$. Mohr: centre $40$, radius $50$. Largest shear $\tfrac{90-(-10)}{2}=50$ MPa against an allowed $125$ MPa: safety factor $2.5$. (Von Mises: $\sqrt{90^2-90(-10)+10^2}\approx95.4$ MPa, factor $\approx2.6$.)
🧪 Self-check quiz
0/5 correct
Q1. The eigenvalues of $\begin{bmatrix}2&3\\3&2\end{bmatrix}$ are
Q2. For a symmetric matrix, eigenvectors belonging to different eigenvalues are always
Q3. In an orthogonal diagonalisation $A=PDP^{T}$, the matrix $P$ satisfies
Q4. The quadratic form $Q(\mathbf{x})=x_1^2+6x_1x_2+x_2^2$ is
Q5. The matrix of the quadratic form $2x_1^2-8x_1x_2+5x_2^2$ is