Projection and distance from a subspace

Andrew Thangaraj

Aug-Nov 2020

Recap

  • Vector space \(V\) over a scalar field \(F\)
    • \(F\): real field \(\mathbb{R}\) or complex field \(\mathbb{C}\) in this course
  • \(m\times n\) matrix A represents a linear map \(T:F^n\to F^m\)
    • dim null \(T\) + dim range \(T\) = dim \(V\)
    • Solution to \(Ax=b\) (if it exists): \(u+\) null\((A)\)
  • Four fundamental subspaces of a matrix
    • Column space, row space, null space, left null space
  • Eigenvalue \(\lambda\) and Eigenvector \(v\): \(Tv=\lambda v\)
    • Some linear maps are diagonalizable
  • Inner products, norms, orthogonality and orthonormal basis
    • Upper triangular matrix for a linear map over an orthonormal basis
    • \(V=U\oplus U^{\perp}\) for any subspace \(U\)
    • Orthogonal projection

Distances and norms

\(v_1,v_2\in V\)

Distance between \(v_1\) and \(v_2\), \(d(v_1,v_2)\), is defined as \[d(v_1,v_2)=\lVert v_1-v_2\rVert\]

Properties

  • \(d(v_1,v_2)=0\) iff \(v_1=v_2\)

  • \(d(v_1,v_2)^2=\langle v_1-v_2,v_1-v_2\rangle\)

How to define distance between a vector and a subspace?

Distance of a vector from a subspace

\(v\in V\) and \(U\): subspace

Distance between \(v\) and \(U\), \(d(v,U)\), is defined as \[d(v,U)=\min_{u\in U}\lVert v-u\rVert\]

Properties

  • \(d(v,U)=0\) iff \(v\in U\)

  • \(Tx=v\) has a solution iff \(d(v,\text{range } T)=0\)

Examples

  1. \(\mathbb{R}^2\): \(d((x,y),x\text{-axis})=?\)

  2. \(\mathbb{R}^2\): \(d((2,5),\{(x,y):x=y\})=?\)

  3. \(\mathbb{R}^2\): \(d((2,5),\{(x,y):ax+by=0\})=?\)

Closest point in subspace

\(v\in V\) and \(U\): subspace

Point closest to \(v\) in \(U\) is defined as \[\arg\min_{u\in U}\lVert v-u\rVert\]

  • How is “closest” justified? Is it unique?

  • Distance of \(v\) from \(U\) is the distance between \(v\)

    and the point closest to \(v\) in \(U\)

Examples

  1. \(\mathbb{R}^3\): \((1,2,3)\) and \(x{-}y\) plane?

  2. \(\mathbb{R}^3\): \((1,2,3)\) and \(\{(x,y,z):x+y+z=x+2y+3z=0\}\)?

  3. \(\mathbb{R}^{100}\): \((1,2,\ldots,100)\) and a 50-dimensional subspace?

Orthogonal projection and minimization

\(v\in V\) and \(U\): subspace

\(P_U\): orthogonal projection onto \(U\)

Closest point to \(v\) in \(U\) is \(P_Uv\) (unique).

Distance of \(v\) from \(U\) is \(\lVert v-P_Uv\rVert\).

Proof

Let \(u\in U\) be some vector in \(U\).

\(\begin{align} \lVert v-u\rVert^2&=\lVert (v-P_Uv)+(P_Uv-u)\rVert^2\\ &=\lVert v-P_Uv\rVert^2+\lVert P_Uv-u\rVert^2\\ &\ge \lVert v-P_Uv\rVert^2 \end{align}\)

Examples: \(\mathbb{R}^2\)

  1. \((x,y)\) and \(x\)-axis

    \(U=\) span\(\{(1,0)\}\)

    \(P_U(x,y)=\langle (x,y),(1,0)\rangle (1,0)=(x,0)\)

  2. (2,5) and \(\{(x,y):x=y\}\)

    \(U=\) span\(\{(1/\sqrt{2},1/\sqrt{2})\}\)

    \(P_U(x,y)=((x+y)/\sqrt{2},(x+y)/\sqrt{2})\)

  3. (2,5) and \(\{(x,y):ax+by=0\}\)

    \(U=\) span\(\{(b,-a)\}=\) span\(\{(b/\sqrt{a^2+b^2},-a/\sqrt{a^2+b^2})\}\)

    \(P_U(x,y)=((bx-ay)b/\sqrt{a^2+b^2},-(bx-ay)a/\sqrt{a^2+b^2})\)

Examples

  1. \((1,2,3)\) and \(x{-}y\) plane

    \(U=\) span\(\{(1,0,0),(0,1,0)\}\)

    \(P_U(x,y,z)=x(1,0,0)+y(0,1,0)=(x,y,0)\)

  2. \((1,2,3)\) and \(\{(x,y,z):x+y+z=x+2y+3z=0\}\)

    \(U=\) span\(\{(1,-2,1)/\sqrt{6}\}\)

    \(P_U(x,y,z)=(x-2y+z)(1,-2,1)/6\)

  3. \((1,2,\ldots,100)\) and a 50-dimensional subspace?