Orthogonal Projection

Andrew Thangaraj

Aug-Nov 2020

Recap

  • Vector space \(V\) over a scalar field \(F\)
    • \(F\): real field \(\mathbb{R}\) or complex field \(\mathbb{C}\) in this course
  • \(m\times n\) matrix A represents a linear map \(T:F^n\to F^m\)
    • dim null \(T\) + dim range \(T\) = dim \(V\)
    • Solution to \(Ax=b\) (if it exists): \(u+\) null\((A)\)
  • Four fundamental subspaces of a matrix
    • Column space, row space, null space, left null space
  • Eigenvalue \(\lambda\) and Eigenvector \(v\): \(Tv=\lambda v\)
    • Some linear maps are diagonalizable
  • Inner products, norms, orthogonality and orthonormal basis
    • Upper triangular matrix for a linear map over an orthonormal basis
    • \(V=U\oplus U^{\perp}\) for any subspace \(U\)

Linear maps onto a subspace

\(U\): subspace of \(V\)

Is there a linear operator \(T\) s.t. range\((T)=U\)?

Yes… there are many!

Basis for \(V\): \(\{v_1,\ldots,v_n\}\)

Define \(T\) as mapping each \(v_i\) to some \(u_i\in U\)

Othogonal projection: linear operator taking \(v\in V\) into \(U\) in a specific way

Recap: orthogonal complements

\(U\): subspace of \(V\)

\(U^{\perp}\): orthogonal complement of \(U\)

\[U^{\perp}=\{v\in V: \langle v,u\rangle=0\text{ for all }u\in U\}\]

\[V=U\oplus U^{\perp}\]

How to find \(U^{\perp}\)?

Orthonormal basis for \(U\): \(\{e_1,\ldots,e_m\}\)

Extend to orthonormal basis for \(V\): \(\{e_1,\ldots,e_m,e_{m+1},\ldots,e_n\}\)

Orthonormal basis for \(U^{\perp}\): \(\{e_{m+1},\ldots,e_n\}\)

\(v=\langle v,e_1\rangle e_1+\cdots+\langle v,e_m\rangle e_m+\langle v,e_{m+1}\rangle e_{m+1}+\cdots+\langle v,e_n\rangle e_n\)

\(u=\langle v,e_1\rangle e_1+\cdots+\langle v,e_m\rangle e_m\in U\)

\(u^{\perp}=\langle v,e_{m+1}\rangle e_{m+1}+\cdots+\langle v,e_n\rangle e_n\)

\(v=u+u^{\perp}\)

Examples

\(U=\) span\(\{(1,0,0,0),(0,1,0,0)\}\)

Orthonormal basis extension

\(\{(1,0,0,0),(0,1,0,0),(0,0,1,0),(0,0,0,1)\}\)

\(U^{\perp}=\) span\(\{(0,0,1,0),(0,0,0,1)\}\)

\(v=(1,0,1,1)=u+u^{\perp}\), where \(u=(1,0,0,0)\), \(u^{\perp}=(0,0,1,1)\)

\(U=\) span\(\{(1/\sqrt{2},1/\sqrt{2},0,0),(1/\sqrt{2},-1/\sqrt{2},0,0)\}\)

Orthonormal basis extension

\(\{(1/\sqrt{2},1/\sqrt{2},0,0),(1/\sqrt{2},-1/\sqrt{2},0,0),(0,0,1/\sqrt{2},1/\sqrt{2}),(0,0,1/\sqrt{2},-1/\sqrt{2})\}^1\)

\(U^{\perp}=\{(0,0,1/\sqrt{2},1/\sqrt{2}),(0,0,1/\sqrt{2},-1/\sqrt{2})\}\)

\(v=(1,0,1,1)^1=u+u^{\perp}\), where \(u=(1,0,0,0)^1\), \(u^{\perp}=(0,0,1,1)^1\)

Orthogonal projection

\(U\): finite-dimensional subspace of \(V\)

Orthogonal projection operator \(P_U\) maps \(v=u+u^{\perp}\), where \(u\in U\), \(u^{\perp}\in U^{\perp}\), to \(u\).

  • \(P_U\): a linear operator, well-defined by uniqueness of \(u\)

Example

  1. \(x\in V\), \(x\ne0\), \(U=\) span\((x)\)

    Orthonormal basis for \(U\): \(\dfrac{x}{\lVert x\rVert}\)

    \[P_Uv=\frac{\langle v,x\rangle}{\lVert x\rVert^2}x\]

  1. \(x,y\in V\), linearly independent, and \(U=\) span\((x,y)\)

    Orthonormal basis for \(U\): \(e_1=\dfrac{x}{\lVert x\rVert}\), \(e_2=\dfrac{y-\langle y,e_1\rangle e_1}{\lVert y-\langle y,e_1\rangle e_1\rVert}\)

    \[P_Uv=\langle v,e_1\rangle e_1+\langle v,e_2\rangle e_2\]

Matrices for the projection operator

\(V=\mathbb{R}^n\), dot product, standard basis

\(U=\) span\(\{(x_1,\ldots,x_n)\}\)

\[P_Uv=\frac{\langle v,x\rangle}{\lVert x\rVert^2}x = \frac{1}{\lVert x\rVert^2}x(x^Tv)\]

\(P_U\leftrightarrow \dfrac{1}{\lVert x\rVert^2}xx^T\)

\(U=\) span\(\{e_1=(e_{11},\ldots,e_{1n}),\ldots,e_m=(e_{m1},\ldots,e_{mn})\}\)

\(e_1,\ldots,e_m\): orthonormal

\(\begin{align} P_Uv&=\langle v,e_1\rangle e_1+\cdots+\langle v,e_m\rangle e_m\\ &=(e_1e^T_1+\ldots+e_me^T_m)v \end{align}\)

\(P_U\leftrightarrow e_1e^T_1+\ldots+e_me^T_m\)

Properties

\(U\): finite-dimensional subspace of \(V\), \(v\in V\)

  1. \(P_Uu=u\) for \(u\in U\)

  2. \(P_Uw=0\) for \(w\in U^{\perp}\)

  3. range \(P_U=U\)

  4. null \(P_U=U^{\perp}\)

  5. \(v-P_Uv\in U^{\perp}\)

  6. \(P^2_U=P_U\)

  7. \(\lVert P_Uv\rVert \le \lVert v\rVert\)