4.2 Dot Product, Length and Angle

In Section 4.1 we could add and scale vectors, but we could not yet ask the most basic geometric questions: How long is this vector? How far apart are two points? Are these two directions perpendicular? A surveyor checking that a wall is square, a physicist computing the work done by a force pulling at an angle, and a search engine deciding which web page best matches your query are all asking these questions — the last one in a space with tens of thousands of dimensions.

Remarkably, a single number answers all of them: the dot product. It is computed by multiplying matching components and adding. From it we get the length of a vector, the distance between points, the angle between directions, a test for perpendicularity, and the projection of one vector onto another — the shadow that one vector casts along the other.

The Dot Product and Length

The answer is a number, not a vector. For instance $(1,-2,3)\cdot(4,0,-1)=4+0-3=1$. The dot product obeys rules that look just like the rules for multiplying numbers — each follows by writing out the components.

Property (d) holds because $\mathbf{u}\cdot\mathbf{u}=u_1^2+\dots+u_n^2$ is a sum of squares. In $\mathbb{R}^2$ and $\mathbb{R}^3$, Pythagoras says that this sum is the square of the length of the arrow — so we use it to define length in every $\mathbb{R}^n$.

Angles, Orthogonality and Two Inequalities

Draw $\mathbf{u}$ and $\mathbf{v}$ from the same point in $\mathbb{R}^2$ or $\mathbb{R}^3$ and let $\theta$ ($0\le\theta\le\pi$) be the angle between them. The third side of the triangle is $\mathbf{u}-\mathbf{v}$, and the law of cosines says $$\|\mathbf{u}-\mathbf{v}\|^2=\|\mathbf{u}\|^2+\|\mathbf{v}\|^2-2\|\mathbf{u}\|\|\mathbf{v}\|\cos\theta .$$ Expanding the left side with Theorem 4.3 instead gives $$\|\mathbf{u}-\mathbf{v}\|^2=(\mathbf{u}-\mathbf{v})\cdot(\mathbf{u}-\mathbf{v})=\|\mathbf{u}\|^2-2\,\mathbf{u}\cdot\mathbf{v}+\|\mathbf{v}\|^2 .$$ Comparing the two expressions:

In $\mathbb{R}^n$ with $n\ge4$ there is no picture, so we turn the formula around and define $\cos\theta=\dfrac{\mathbf{u}\cdot\mathbf{v}}{\|\mathbf{u}\|\|\mathbf{v}\|}$. For this to make sense the fraction must lie between $-1$ and $1$. That is guaranteed by one of the most useful inequalities in mathematics.

The triangle inequality says that a detour is never shorter: going along $\mathbf{u}$ and then $\mathbf{v}$ is at least as long as going straight along $\mathbf{u}+\mathbf{v}$.

Projections

Very often we want to split a vector into a part along a given direction and a part perpendicular to it: the part of the wind that pushes a sailing boat forward, the part of gravity that pulls a box down a ramp, the part of a data point explained by a trend line. Suppose $\mathbf{u}\ne\mathbf{0}$ and we want to write $\mathbf{v}=c\mathbf{u}+\mathbf{w}$ with $\mathbf{w}$ orthogonal to $\mathbf{u}$. Dotting both sides with $\mathbf{u}$ gives $\mathbf{v}\cdot\mathbf{u}=c\,(\mathbf{u}\cdot\mathbf{u})+0$, so the only possible choice is $c=\dfrac{\mathbf{v}\cdot\mathbf{u}}{\mathbf{u}\cdot\mathbf{u}}$.

The component is a signed length: $\operatorname{proj}_{\mathbf{u}}\mathbf{v}=\left(\operatorname{comp}_{\mathbf{u}}\mathbf{v}\right)\dfrac{\mathbf{u}}{\|\mathbf{u}\|}$, so it is positive when the shadow points the same way as $\mathbf{u}$ and negative when it points backwards. Note the order: $\operatorname{proj}_{\mathbf{u}}\mathbf{v}$ lies along $\mathbf{u}$, and in general $\operatorname{proj}_{\mathbf{u}}\mathbf{v}\ne\operatorname{proj}_{\mathbf{v}}\mathbf{u}$ — compare the two settings in Interactive 1.

Applications: Work and Cosine Similarity

Work. When a constant force $\mathbf{F}$ moves an object through a straight displacement $\mathbf{d}$, only the part of the force along the motion does work. That part is $\operatorname{comp}_{\mathbf{d}}\mathbf{F}=\|\mathbf{F}\|\cos\theta$, so the work done is $$W=\left(\|\mathbf{F}\|\cos\theta\right)\|\mathbf{d}\|=\mathbf{F}\cdot\mathbf{d}.$$ With $\mathbf{F}$ in newtons and $\mathbf{d}$ in metres, $W$ is in joules (J).

Cosine similarity. In data science a document is often turned into a vector by counting how many times each word of a fixed vocabulary appears. Two documents about the same topic use words in similar proportions, even if one is much longer. So we compare directions, not lengths: the cosine similarity of two nonzero data vectors is $$\cos\theta=\frac{\mathbf{u}\cdot\mathbf{v}}{\|\mathbf{u}\|\,\|\mathbf{v}\|}.$$ For count data all entries are $\ge0$, so the value lies between $0$ (no words in common) and $1$ (exactly the same proportions). Recommendation systems, plagiarism checkers and search engines all use it.

Exercises