Back to the on-screen lesson ·
Every matrix of every shape is a rotation, a scaling along perpendicular axes and another rotation; the singular values are the scalings, and they carry the rank, the four subspaces, the conditioning and the best low-rank approximation.
Paper packet. Every task here also exists on screen, where it is checked automatically; answers written on paper are not assessed by Nydus. When you are back at a device, enter your answers there.
By the end of this lesson you will be able to form $A^{T}A$ for a matrix of any shape and explain why the spectral theorem applies to it, find singular values as the non-negative square roots of its eigenvalues, put the steps of a singular value decomposition in the order their dependencies force, read the rank and the condition number off the list of singular values, describe what $U$, $\Sigma$ and $V^{T}$ each do to the unit circle and identify the ellipse they produce, and say what distinguishes a singular value from an eigenvalue.
The spectral theorem says a real symmetric matrix has real eigenvalues and an orthonormal basis of eigenvectors. Least squares needed $A^{T}A$ and noticed it was symmetric. Those two facts are the whole engine of this lesson: $A^{T}A$ is symmetric for every $A$ of every shape, so the spectral theorem applies to it even when $A$ is a $1000 \times 3$ table of measurements and has no eigenvalues of its own at all.
$A = U\Sigma V^{T}$ with $A$ of size $m \times n$: $U$ is $m \times m$ orthogonal, $V$ is $n \times n$ orthogonal, and $\Sigma$ is $m \times n$ with the singular values $\sigma_1 \ge \sigma_2 \ge \dots \ge 0$ on its main diagonal and zeros everywhere else. The columns of $V$ are the right singular vectors, the columns of $U$ the left singular vectors. The condition number is $\sigma_{\max} / \sigma_{\min}$. A rank-$k$ approximation keeps the first $k$ singular values and discards the rest.
Diagonalisation needed the matrix to be square and to have enough eigenvectors. The spectral theorem needed it to be symmetric. The singular value decomposition needs nothing: every real matrix, of every shape, can be written
$$A = U\Sigma V^{T}$$
with $U$ and $V$ orthogonal and $\Sigma$ diagonal and non-negative. A $7 \times 3$ matrix has one. A matrix of rank zero has one. A rotation has one.
Read the three factors right to left and the statement becomes a sentence about what linear maps do. $V^{T}$ is orthogonal, so it rotates or reflects and changes no lengths. $\Sigma$ stretches each coordinate axis by its own non-negative factor, possibly squashing some to nothing. $U$ rotates or reflects again. Every linear map is a rotation, then a scaling along perpendicular axes, then another rotation. There is nothing else a matrix can do.
That is why the unit sphere always goes to an ellipsoid: rotating a sphere leaves it a sphere, stretching along axes makes it an ellipsoid, and rotating an ellipsoid leaves it one. The semi-axes are the singular values and their directions are the columns of $U$.
The singular values come from the eigenvalues of $A^{T}A$, which is where the last lesson gets used: $A^{T}A$ is symmetric, so it has real eigenvalues and an orthonormal basis of eigenvectors; and $v^{T}A^{T}Av = \|Av\|^2 \ge 0$, so none of those eigenvalues is negative. Non-negative numbers have real square roots, and those roots are the singular values.
Another way: picture
Draw the unit circle and watch it move under $A$. It stays a circle under $V^{T}$, becomes an ellipse with axes along the coordinate directions under $\Sigma$, and is turned to its final position by $U$. The longest radius of the finished ellipse is $\sigma_1$ and points along the first column of $U$; the shortest is $\sigma_{\min}$. So $\sigma_1$ is the most the matrix can stretch any unit vector and $\sigma_{\min}$ is the least — which is the whole content of the condition number, and why a flat ellipse means trouble.
Another way: steps
The rank. The number of non-zero singular values is the rank of $A$, and this is the definition numerical work actually uses. Counting pivots is an all-or-nothing question about exact zeros; a singular value of $10^{-14}$ says something a pivot count cannot, namely that the matrix is a hair's breadth from rank deficient.
All four fundamental subspaces at once. With $r$ non-zero singular values:
| Subspace | Basis | Dimension |
|---|---|---|
| column space of $A$ | $u_1, \dots, u_r$ | $r$ |
| null space of $A^{T}$ | $u_{r+1}, \dots, u_m$ | $m - r$ |
| row space of $A$ | $v_1, \dots, v_r$ | $r$ |
| null space of $A$ | $v_{r+1}, \dots, v_n$ | $n - r$ |
Every one of those bases is orthonormal, which elimination never gives you, and rank-nullity is visible in the table rather than proved.
The best low-rank approximation. Writing $A = \sigma_1u_1v_1^{T} + \dots + \sigma_ru_rv_r^{T}$ expresses $A$ as a sum of rank-one pieces in decreasing order of size. Keeping the first $k$ gives the closest rank-$k$ matrix to $A$ there is, in the least-squares sense — the Eckart-Young theorem. That is the basis of principal component analysis, of low-rank compression of a data table, and of the recommendation systems that factor a ratings matrix; it is one theorem doing all three jobs.
The condition number. $\sigma_{\max}/\sigma_{\min}$ bounds how far an error in $b$ can be amplified when $Ax = b$ is solved. Near $1$ the matrix treats every direction alike; large and the solve is fragile in the squashed direction, whatever algorithm is used.
It is easy to file the singular values next to the eigenvalues and lose the distinctions, so here they are.
Singular values are not eigenvalues. $\begin{pmatrix} 1 & 5 \\ 0 & 1 \end{pmatrix}$ has both eigenvalues equal to $1$ and singular values of about $5.19$ and $0.19$. The eigenvalues say the shear leaves one direction fixed; the singular values say it stretches the plane by a factor of five. Both are true and they are answers to different questions.
For a symmetric positive definite matrix they coincide. Then $A = QDQ^{T}$ already has the right form, the singular values are the eigenvalues, and $U = V = Q$. If a symmetric matrix has a negative eigenvalue $\lambda$, the matching singular value is $|\lambda|$ and the sign moves into a column of $U$.
$A^{T}A$ and $AA^{T}$ give the same non-zero eigenvalues. One is $n \times n$ and the other $m \times m$, so they cannot have the same eigenvalues in total, but the non-zero ones match — which is why there is one list of singular values and not two. In practice you compute with whichever is smaller.
Uniqueness. The singular values are unique. The vectors are not: a repeated singular value leaves a whole subspace to choose an orthonormal basis from, and even a simple singular value leaves the sign of the pair $(u_i, v_i)$ free, since flipping both changes nothing.
Taking square roots in the wrong place. The eigenvalues of $A^{T}A$ are the squares of the singular values. Reporting an eigenvalue of $25$ as a singular value of $25$ rather than $5$ is the single most common error in this computation, and the table in the practice above exists to force the two apart.
Expecting a negative singular value. They are square roots of non-negative numbers and are taken non-negative by definition. A matrix that reverses a direction records that in a column of $U$, not in a minus sign on $\sigma$.
Thinking the matrix must be square. Nothing here needs it, and the shapes are the part worth checking: $U$ is $m \times m$, $\Sigma$ is $m \times n$ like $A$ itself, and $V^{T}$ is $n \times n$. If those do not multiply out to $m \times n$, something has been transposed.
Forgetting the ordering is a convention. $\sigma_1 \ge \sigma_2 \ge \dots$ is agreed, not forced, and every statement about "the first $k$" being the best approximation depends on it. Ordering the singular values means reordering the matching columns of $U$ and $V$ with them.
Reading $u_i = Av_i/\sigma_i$ where $\sigma_i = 0$. That formula is only for the non-zero singular values. The remaining columns of $U$ are completed to an orthonormal basis by any means available, and they are a basis of the null space of $A^{T}$.
$A = \begin{pmatrix} 3 & 0 \\ 4 & 5 \end{pmatrix}$. Then $A^{T}A = \begin{pmatrix} 25 & 20 \\ 20 & 25 \end{pmatrix}$, symmetric as promised.
Always start here.
Its eigenvalues are $45$ and $5$, for the eigenvectors $(1, 1)$ and $(1, -1)$ — perpendicular, by the spectral theorem. So $\sigma_1 = \sqrt{45}$ and $\sigma_2 = \sqrt{5}$, and $V$ has those normalised vectors as columns.
Eigenvalues of the product; square roots for the singular values.
$u_1 = Av_1/\sigma_1$ and $u_2 = Av_2/\sigma_2$ finish it. Check: $\sigma_1\sigma_2 = \sqrt{225} = 15 = |\det A|$, which is no accident — the determinant is the volume scaling and the singular values are the stretch factors.
A cheap check that catches most slips.
A $200 \times 50$ data matrix is reported to have singular values $412$, $88$, $31$, $0.004$, and then forty-six more below $10^{-9}$.
No entries needed.
Its numerical rank is $4$: four singular values are of a size that means something and the rest are noise. So the fifty columns really live in a four-dimensional subspace.
Rank, read from the list.
The condition number is $412/0.004$, about $10^5$, so solving a least-squares problem with this matrix will amplify measurement error in one direction by that factor. Keeping only the first three pieces gives the best rank-3 approximation and throws that direction away deliberately.
Conditioning and compression, from the same list.
$A$ is $3 \times 2$, so $A^{T}A$ is $2 \times 2$ and there are two singular values.
The shapes first.
$A^{T}A = \begin{pmatrix} 4 & 0 \\ 0 & 9 \end{pmatrix}$, already diagonal, with eigenvalues $9$ and $4$.
Order them decreasing.
So $\sigma_1 = 3$ and $\sigma_2 = 2$. Both are non-zero, so the rank is $2$ — the matrix is one-to-one, though nowhere near onto, and $U$ needs a third column spanning the null space of $A^{T}$.
Let $A = \begin{pmatrix} 8 & 3 \\ 0 & 2 \end{pmatrix}$. Write $A^{T}A$, the matrix every singular value decomposition begins from.
This task has no paper form; do it on a device.
$A$ is an $5 \times 2$ matrix. Put the steps of computing its singular value decomposition into the order they have to be carried out.
Number the steps in order (write the number in the box):
Let $A = \begin{pmatrix} 6 & -4 \\ 8 & 3 \end{pmatrix}$, whose two columns happen to be perpendicular. Fill in the eigenvalues of $A^{T}A$ and the singular values of $A$, largest first.
| Eigenvalue | Singular value | |
|---|---|---|
| Larger | ||
| Smaller |
A matrix has singular values $5$, $4$ and $20$, listed in no particular order. What is its condition number?
Answer:
$A$ is an $5 \times 4$ matrix of rank $1$. Which statement about its singular values is true?
The matrix $A = \begin{pmatrix} 0 & -5 \\ 2 & 0 \end{pmatrix}$ sends the unit circle to an ellipse. Plot the two points of that ellipse that are furthest from the origin.
The unit circle, centred at the origin with radius $1$.
Plot your answer on the grid:
Lesson test: one question per skill, one attempt each, no hints. Your answers are checked when you submit.
Let $A = \begin{pmatrix} 3 & 5 \\ 0 & 4 \end{pmatrix}$. Write $A^{T}A$, the matrix every singular value decomposition begins from.
This task has no paper form; do it on a device.
You can compute $A^{T}A$, get singular values from its eigenvalues, and say what the list of them tells you about rank and conditioning. Say in your own words why every matrix has a singular value decomposition when only some have an eigenvalue one. That is the end of the course: elimination told you what a system does, and this tells you how well it does it.
10. Your turn: what are the singular values of $\begin{pmatrix} 2 & 0 \\ 0 & 0 \\ 0 & 3 \end{pmatrix}$?, step 3
Say what the numbers mean, not just what they are.