Back to the on-screen lesson ·
Splitting a vector into a piece along a subspace and a piece perpendicular to it, why that piece is the closest point, orthonormal bases whose coordinates are dot products, orthogonal matrices and the orthogonal complement.
Paper packet. Every task here also exists on screen, where it is checked automatically; answers written on paper are not assessed by Nydus. When you are back at a device, enter your answers there.
By the end of this lesson you will be able to project a vector onto a line and write the projection matrix $aa^{T}/a^{T}a$ that does it, explain why the projection is the closest point of the subspace and why that is the same as the leftover being perpendicular, prove that an orthogonal set of non-zero vectors is independent, read coordinates in an orthonormal basis off as dot products, recognise an orthogonal matrix and use $Q^{T}Q = I$ to show it preserves lengths, and compute an orthogonal complement as the null space of a transpose.
Finding the coordinates of a vector in a basis has meant solving a linear system: write the vector as an unknown combination of the basis vectors and reduce. That is a whole elimination for every vector whose coordinates you want. This lesson shows that one choice of basis — an orthonormal one — reduces the cost to a dot product per coordinate, and that the geometry which makes that work is the same geometry that finds closest points.
A set of vectors is orthogonal when every two distinct members have dot product zero, and orthonormal when it is orthogonal and every member has length one. The projection of $b$ onto a subspace $W$ is the vector of $W$ closest to $b$. The orthogonal complement $W^{\perp}$ is the set of vectors orthogonal to every vector of $W$; it is itself a subspace. A square matrix whose columns are orthonormal is an orthogonal matrix — an unhappy name, since orthogonal columns alone are not enough.
Given a line through a non-zero $a$, every vector $b$ splits as
$$b = \underbrace{\frac{a \cdot b}{a \cdot a}\,a}_{\text{along } a} \;+\; \underbrace{\left(b - \frac{a \cdot b}{a \cdot a}\,a\right)}_{\text{perpendicular to } a},$$
and the split is unique. The first piece is the projection $\operatorname{proj}_a b$; the second is what is left. Check the second really is perpendicular: dot it with $a$ and the two terms cancel exactly, by the choice of the coefficient. That coefficient is chosen for no other reason.
The projection is the closest point. For any other point $ta$ of the line, $|b - ta|^2 = |b - \operatorname{proj}_ab|^2 + |\operatorname{proj}_ab - ta|^2$ by Pythagoras, since the two pieces are perpendicular. The second term is a square and is smallest — zero — when $ta$ is the projection. So "closest point" and "the leftover is perpendicular" are the same condition, and it is usually the perpendicularity that is easier to impose.
Written with $a \cdot b = a^{T}b$, the formula becomes a matrix: $\operatorname{proj}_ab = \dfrac{aa^{T}}{a^{T}a}b$. The projection matrix $P = \dfrac{aa^{T}}{a^{T}a}$ is symmetric and satisfies $P^2 = P$ — projecting a second time does nothing, because the first projection already landed on the line.
For a subspace $W$ with an orthonormal basis $q_1, \dots, q_k$, the projection is just the sum of the one-dimensional ones:
$$\operatorname{proj}_W b = (b \cdot q_1)q_1 + \dots + (b \cdot q_k)q_k.$$
The denominators have vanished because each $q_i \cdot q_i = 1$, and the terms do not interfere because the $q_i$ are mutually orthogonal. Without orthogonality the formula is false, and a linear system is needed instead.
Another way: picture
Stand a vector $b$ up out of a plane $W$ and drop a perpendicular to the floor. The foot of that perpendicular is $\operatorname{proj}_W b$, the shadow of $b$; the dropped segment is $b - \operatorname{proj}_W b$, and it points straight up, in $W^{\perp}$. Every other point of the floor is further from the tip of $b$ than the foot is — which is the whole of the least-squares idea, seen before any algebra.
Another way: steps
Let $q_1, \dots, q_k$ be non-zero and mutually orthogonal. Two facts follow immediately, and both are free of computation.
They are independent. Suppose $c_1q_1 + \dots + c_kq_k = 0$. Dot the whole equation with $q_i$: every term except the $i$-th contains $q_j \cdot q_i = 0$, so what survives is $c_i(q_i \cdot q_i) = 0$, and $q_i \cdot q_i \ne 0$. Hence $c_i = 0$, for each $i$ in turn. No reduction, no determinant — the orthogonality did all of it.
Their coordinates are dot products. If $b = c_1q_1 + \dots + c_kq_k$ lies in their span, the same trick gives $c_i = \dfrac{b \cdot q_i}{q_i \cdot q_i}$, and if the set is orthonormal, $c_i = b \cdot q_i$. Compare the general case: finding coordinates in an arbitrary basis means solving a $k \times k$ system, and here it means $k$ dot products that do not interact.
That second fact is why orthogonal matrices are so convenient. If $Q$ has orthonormal columns then $Q^{T}Q = I$, since the $(i,j)$ entry of $Q^{T}Q$ is $q_i \cdot q_j$. For a square $Q$ this says $Q^{-1} = Q^{T}$: the hardest computation in the subject becomes a transposition. And $|Qx|^2 = x^{T}Q^{T}Qx = |x|^2$, so lengths and, with them, angles survive multiplication untouched.
$W^{\perp}$ is the set of vectors orthogonal to every vector in $W$. It is a subspace: if $u$ and $v$ are orthogonal to all of $W$ then so is any combination of them, because the dot product is linear in its first slot.
To compute it, note that being orthogonal to a spanning set is enough — orthogonality to a combination follows. So if $W$ is the column space of $A$, then $W^{\perp}$ is the set of $x$ with $A^{T}x = 0$: the null space of $A^{T}$. The orthogonal complement of the column space is the null space of the transpose. That is one of the four fundamental subspaces meeting another, and it is the identity least squares is built on.
Three facts worth holding:
The last of those is the projection theorem, and the rest of this unit is three applications of it.
The formula for projecting onto a subspace needs an orthogonal basis. $(b \cdot v_1)v_1 + (b \cdot v_2)v_2$ is not the projection onto the span of $v_1$ and $v_2$ unless those two are orthonormal. Using it on a basis that merely spans is the commonest error in this unit, and the next lesson exists to supply the basis it needs.
An orthogonal matrix is not a matrix with orthogonal columns. The columns must also have length one. $\begin{pmatrix} 2 & 0 \\ 0 & 3\end{pmatrix}$ has orthogonal columns and is not an orthogonal matrix; it stretches. The name is bad and it is not going to change.
$\operatorname{proj}_a b$ and $\operatorname{proj}_b a$ are different vectors. One lies along $a$, the other along $b$. The scalar coefficients differ too, unless the two vectors happen to have the same length.
The projection does not depend on the length of $a$. Replacing $a$ by $5a$ multiplies the numerator by $5$ and the denominator by $25$, then the final factor of $a$ by $5$ again — everything cancels. It is the direction of $a$ that matters, which is the point of the denominator.
$W^{\perp}$ is not the complement of $W$ as a set. It is a subspace, it contains the zero vector, and almost every vector outside $W$ is in neither.
Project $b = (4, 3)$ onto $a = (2, 0)$. $a \cdot b = 8$, $a \cdot a = 4$, so the coefficient is $2$.
One dot product each way.
$\operatorname{proj}_ab = 2(2, 0) = (4, 0)$, and the leftover is $(4,3) - (4,0) = (0,3)$.
Subtract to get the perpendicular part.
Check: $(0,3) \cdot (2,0) = 0$. And $|b|^2 = 25 = 16 + 9$ splits as the two pieces' squared lengths, as Pythagoras requires.
Two checks, both cheap.
$q_1 = \left(\tfrac{1}{\sqrt2}, \tfrac{1}{\sqrt2}\right)$, $q_2 = \left(\tfrac{1}{\sqrt2}, -\tfrac{1}{\sqrt2}\right)$, and $b = (5, 1)$.
Confirm first: each has length one and they dot to zero.
$c_1 = b \cdot q_1 = \tfrac{6}{\sqrt2}$ and $c_2 = b \cdot q_2 = \tfrac{4}{\sqrt2}$.
Two dot products, no system.
Check: $c_1^2 + c_2^2 = 18 + 8 = 26 = 25 + 1 = |b|^2$. The coordinates carry the length with them.
Only orthonormality makes that work.
$a \cdot b = 1 + 2 + 3 = 6$ and $a \cdot a = 3$, so the coefficient is $2$.
Numerator over denominator.
The projection is $2(1,1,1) = (2,2,2)$, and the leftover is $(-1, 0, 1)$.
Multiply the coefficient by $a$.
Check: $(-1,0,1) \cdot (1,1,1) = 0$. And notice the coefficient was the mean of the components of $b$ — an average is a projection.
Write the matrix that projects $\mathbb{R}^2$ onto the line through $a = (5, 1)$.
This task has no paper form; do it on a device.
$b = (2, 4)$ and $a = (1, 1)$. What is the second component of $b - \operatorname{proj}_{a}b$?
Answer:
$u$ and $v$ are non-zero vectors in $\mathbb{R}^{3}$ with $u \cdot v = 0$. Which of these must be true?
Match each set of two vectors to the description that fits it.
| Orthogonal but not orthonormal | Orthonormal | Not even orthogonal | |
|---|---|---|---|
| $(2, 0)$ and $(0, 3)$ | |||
| $\left(\tfrac{3}{5}, \tfrac{4}{5}\right)$ and $\left(-\tfrac{4}{5}, \tfrac{3}{5}\right)$ | |||
| $(2, 3)$ and $(3, 2)$ |
$W$ is a subspace and $Q$ is a square matrix. Select every statement that is true.
This task has no paper form; do it on a device.
$q_1 = \left(\tfrac{3}{5}, \tfrac{4}{5}\right)$ and $q_2 = \left(-\tfrac{4}{5}, \tfrac{3}{5}\right)$ are an orthonormal basis of $\mathbb{R}^2$, and $b = (20, 15)$. Fill in the table.
| Value | |
|---|---|
| $b \cdot q_1$ | |
| $b \cdot q_2$ | |
| $|b|^2$ |
Lesson test: one question per skill, one attempt each, no hints. Your answers are checked when you submit.
Write the matrix that projects $\mathbb{R}^2$ onto the line through $a = (3, 4)$.
This task has no paper form; do it on a device.
You can project onto a line or onto a subspace with an orthonormal basis, and say what the orthogonal complement of a column space is. Say in your own words why the projection formula fails on a basis that is merely independent. Next: how to manufacture the orthonormal basis the formula needs.
10. Your turn: project $b = (1, 2, 3)$ onto the line through $a = (1, 1, 1)$, step 3
Worth remembering for the last lesson of this unit.