Singular Values and Vectors of a Linear Map

Andrew Thangaraj

Aug-Nov 2020

Recap

  • Vector space \(V\) over a scalar field \(F= \mathbb{R}\) or \(\mathbb{C}\)
  • \(m\times n\) matrix A represents a linear map \(T:F^n\to F^m\)
    • dim null \(T+\) dim range \(T=\) dim \(V\)
    • Solution to \(Ax=b\) (if it exists): \(u+\) null\((A)\)
  • Four fundamental subspaces of a matrix
    • Column space, row space, null space, left null space
  • Eigenvalue \(\lambda\) and Eigenvector \(v\): \(Tv=\lambda v\)
    • There is a basis w.r.t. which a linear map is upper-triangular
    • If there is a basis of eigenvectors, linear map is diagonal w.r.t. it
  • Inner products, norms, orthogonality and orthonormal basis
    • There is an orthonormal basis w.r.t. which a linear map is upper-triangular
    • Orthogonal projection: distance from a subspace
  • Adjoint of a linear map: \(\langle Tv,w\rangle=\langle v,T^*w\rangle\)
    • null \(T=\) \((\)range \(T^*)^{\perp}\)
  • Self-adjoint: \(T=T^*\), Normal: \(TT^*=T^*T\)
  • Complex/real spectral theorem: \(T\) is normal/self-adjoint \(\leftrightarrow\) orthonormal basis of eigenvectors
  • Positive operators: self-adjoint with non-negative eigenvalues
  • Isometries: normal with absolute value 1 eigenvalues

Singular values of a linear map

\(T:V\to W\), linear map

Singular values of \(T\) are the eigenvalues of \(\sqrt{T^*T}\).

  1. \(T^*T:V\to V\): (self-adjoint) positive operator.

  2. Spectral theorem: \[T^*T\leftrightarrow \lambda_1e_1\overline{e^T_1}+\cdots+\lambda_ne_n\overline{e^T_n}\] Eigenvalues: \(\lambda_1\ge\cdots\ge\lambda_n\)
    \(\{e_1,\ldots,e_n\}\): orthonormal basis, \(n=\) dim \(V\)

  3. \(T^*T\) has a unique positive square root \(\sqrt{T^*T}\) \[\sqrt{T^*T}\leftrightarrow \sqrt{\lambda_1}e_1\overline{e^T_1}+\cdots+\sqrt{\lambda_n}e_n\overline{e^T_n}\]

  4. Singular values of \(T\): \(\sqrt{\lambda_1}\ge\cdots\ge\sqrt{\lambda_n}\)

Singular vectors of a linear map

\(T:V\to W\), linear map

\(T^*T:V\to V\), self-adjoint

Right-singular vectors of \(T\) are an orthonormal eigenvector basis vectors of \(T^*T\)

Note: Right-singular vectors are vectors in \(V\)

\(TT^*:W\to W\), self-adjoint

Left-singular vectors of \(T\) are an orthonormal eigenvector basis vectors of \(TT^*\)

Note: Left-singular vectors are vectors in \(W\)

Example: \(2\times 2\)

\(A=\begin{bmatrix}1&2\\3&4\end{bmatrix}\) (standard basis)

Eigenvalues: \(5.372\), \(-0.372\); Eigenvectors: \((0.415,0.909)\), \((0.824,-0.566)\)

\(\begin{bmatrix}1&2\\3&4\end{bmatrix}\begin{bmatrix}0.415\\0.909\end{bmatrix}=5.372\begin{bmatrix}0.415\\0.909\end{bmatrix},\quad\begin{bmatrix}1&2\\3&4\end{bmatrix}\begin{bmatrix}0.824\\-0.566\end{bmatrix}=-0.372\begin{bmatrix}0.824\\-0.566\end{bmatrix}\)

\(A^TA=\begin{bmatrix}10&14\\14&20\end{bmatrix}\), \(AA^T=\begin{bmatrix}5&11\\11&25\end{bmatrix}\)

Eigenvalues of \(A^TA\) and \(AA^T\): \(29.866\), \(0.134\)

Singular values of \(A\): \(\sigma_1=5.465\), \(\sigma_2=0.366\)

Right-singular vectors of \(A\): \(e_1=(0.576,0.817)\), \(e_2=(0.817,-0.576)\)

Left-singular vectors of \(A\): \(f_1=(0.404,0.914)\), \(f_2=(-0.914,0.404)\)

\(\begin{bmatrix}1&2\\3&4\end{bmatrix}\begin{bmatrix}0.576\\0.817\end{bmatrix}=5.465\begin{bmatrix}0.404\\0.914\end{bmatrix},\quad\begin{bmatrix}1&2\\3&4\end{bmatrix}\begin{bmatrix}0.817\\-0.576\end{bmatrix}=0.366\begin{bmatrix}-0.914\\0.404\end{bmatrix}\)

Observation: \(Ae_1=\sigma_1f_1\) and \(Ae_2=\sigma_2f_2\) (This is not an accident)

Example: \(2\times 3\)

\(A=\begin{bmatrix}1&2&3\\4&5&6\end{bmatrix}\) (eigenvalues undefined)

\(A^TA=\begin{bmatrix}17&22&27\\22&29&36\\27&36&45\end{bmatrix}\), \(AA^T=\begin{bmatrix}14&32\\32&77\end{bmatrix}\)

Eigenvalues of \(A^TA\): \(90.403\), \(0.597\), \(0\)

Eigenvalues of \(AA^T\): \(90.403\), \(0.597\)

Singular values of \(A\): \(\sigma_1=9.508\), \(\sigma_2=0.773\), \(\sigma_3=0\)

Right-singular vectors of \(A\): \(e_1=(0.429,0.566,0.704)\), \(e_2=(0.805,0.112,-0.581)\), \(e_3=(0.408,-0.816,0.408)\)

Left-singular vectors of \(A\): \(f_1=(0.386,0.922)\), \(f_2=(-0.922,0.386)\)

\(\begin{bmatrix}1&2&3\\4&5&6\end{bmatrix}\begin{bmatrix}0.429\\0.566\\0.704\end{bmatrix}=9.508\begin{bmatrix}0.386\\0.922\end{bmatrix},\quad\begin{bmatrix}1&2&3\\4&5&6\end{bmatrix}\begin{bmatrix}0.805\\0.112\\-0.581\end{bmatrix}=0.773\begin{bmatrix}-0.922\\0.386\end{bmatrix}\)

\(Ae_1=\sigma_1f_1\), \(Ae_2=\sigma_2f_2\) and \(Ae_3=0\)

Null/range of \(T\), \(T^*\), \(T^*T\), \(\sqrt{T^*T}\) and \(TT^*\)

\(T:V\to W\), \(T^*T:V\to V\), \(\sqrt{T^*T}:V\to V\), \(TT^*:W\to W\)

  1. null \(T=\) \((\)range \(T^*)^{\perp}\)
    null \(T^*=\) \((\)range \(T)^{\perp}\)
    dim range \(T=\) dim range \(T^*\)

  2. null \(T^*T=\) null \(T\)
    range \(T^*T=\) \((\)null \(T^*T)^{\perp}=\) \((\)null \(T)^{\perp}=\) range \(T^*\)
    dim range \(T^*T=\) dim range \(T^*=\) dim range \(T\)

  3. null \(\sqrt{T^*T}=\) null \(T^*T=\) null \(T\) 
    range \(\sqrt{T^*T}=\) \((\)null \(\sqrt{T^*T})^{\perp}=\) \((\)null \(T)^{\perp}\)
    dim range \(\sqrt{T^*T}=\) dim range \(T\)

  4. null \(TT^*=\) null \(T^*=\) \((\)range \(TT^*)^{\perp}\)
    range \(TT^*=\) \((\)null \(T^*)^{\perp}=\) range \(T\)