Andrew Thangaraj
Aug-Nov 2020
\(\mathbb{R}^2\): Dot product of \(x=(x_1,x_2)\) and \(y=(y_1,y_2)\)
\(\langle x,y \rangle=x_1y_1+x_2y_2\in\mathbb{R}\)
Fix \(y\): linear in \(x\)
Related to angle between \(x\) and \(y\)
\(\mathbb{R}^2\): norm of \(x=(x_1,x_2)\)
\(\lVert x\rVert=\sqrt{\langle x,x\rangle}=\sqrt{x^2_1+x^2_2}\in\mathbb{R}^+\)
Nonlinear, but closely connected to dot product
Length or magnitude of \(x\) or distance from origin
\(\mathbb{C}^n\): Dot product of \(x=(x_1,\ldots,x_n)\) and \(y=(y_1,\ldots,y_n)\)
\(\langle x,y \rangle=x_1\overline{y_1}+\cdots+x_n\overline{y_n}\in\mathbb{C}\)
Fix \(y\): linear in \(x\)
Conjugation in the second argument
Generalize notion of angle to \(n\) dimensions
\(\mathbb{C}^n\): norm of \(x=(x_1,\ldots,x_n)\)
\(\lVert x\rVert=\sqrt{\langle x,x\rangle}=\sqrt{\lvert x_1\rvert^2+\cdots+\lvert x_n\rvert^2}\in\mathbb{R}^+\)
Nonlinear, but closely connected to dot product
Conjugation in the second argument is important
Length/magnitude in \(n\) dimensions
\(V\): vector space over \(F=\mathbb{R}\) or \(\mathbb{C}\)
Inner product on \(V\): function mapping two vectors \(u,v\in V\) (in that order) to a scalar \(\langle u,v\rangle\in F\) satisfying the following properties.
positivity: \(\langle v,v\rangle\ge0\)
definiteness: \(\langle v,v\rangle=0\) if and only if \(v=0\)
additivity in first argument: \(\langle u_1+u_2,v\rangle=\langle u_1,v\rangle+\langle u_2,v\rangle\)
homogeneity in first argument: \(\langle \lambda u,v\rangle=\lambda\langle u,v\rangle\), \(\lambda\in F\)
conjugate symmetry: \(\langle u,v\rangle=\overline{\langle v,u\rangle}\)
Dot product in \(\mathbb{R}^n\) and \(\mathbb{C}^n\)
Yes
\(\langle x,y\rangle=x_1y_1+2x_2y_2\) in \(\mathbb{R}^2\)
Yes
\(\langle x,y\rangle=x_1y_1-3x_1y_2-3x_2y_1+20x_2y_2\) in \(\mathbb{R}^2\)
Yes
\(\langle x,y\rangle=x_1y_1-3x_1y_2-3x_2y_1+9x_2y_2\) in \(\mathbb{R}^2\)
No
\[\langle f,g\rangle=\int_{0}^1 f(x)g(x)dx\]
Yes
\[\langle f,g\rangle=\int_{0}^1 f(x)g(x)e^{-x}dx\]
Yes
\[\langle f,g\rangle=\int_{0}^1 f(x)g(x-1/2)dx\]
No
\(V\): vector space over \(F\) is called an inner product space if there is a valid inner product defined on \(V\).
Properties
Fix \(u\in V\). Then, \(Tv=\langle v,u\rangle\) defines a linear map.
\(\langle u,0\rangle=\langle 0,u\rangle=0\)
\(\langle u,v_1+v_2\rangle=\langle u,v_1\rangle+\langle u,v_2\rangle\)
\(\langle u,\lambda v\rangle=\overline{\lambda}\langle u,v\rangle\)
\(V\): inner product space
\(u,v\) are orthogonal if \(\langle u,v\rangle=0\)
Orthogonal decomposition
For \(u,v\in V\) with \(v\ne0\), there exists \(c\) such that \(\langle u-cv,v\rangle=0\).
\(c=\langle u,v\rangle/\langle v,v\rangle\)
\(u = \dfrac{\langle u,v\rangle}{\langle v,v\rangle}v + w\)
\(cv\): parallel to \(v\)
\(w=u-cv\): orthogonal to \(v\)
\(V\): inner product space and \(u,v\in V\). Then \(|\langle u,v\rangle|^2\le \langle u,u\rangle\langle v,v\rangle\)
Proof
If \(v=0\), done
If \(v\ne0\), orthogonal decomposition
\(u = \dfrac{\langle u,v\rangle}{\langle v,v\rangle}v + w\), and \(\langle v,w\rangle=0\)
\(\begin{align} \langle u,u\rangle&=\dfrac{|\langle u,v\rangle|^2}{\langle v,v\rangle^2}\langle v,v\rangle+\langle w,w\rangle\\ &\ge \dfrac{|\langle u,v\rangle|^2}{\langle v,v\rangle} \end{align}\)
\(V\): vector space over \(F=\mathbb{R}\) or \(\mathbb{C}\)
Norm on \(V\): function mapping vector \(v\in V\) to a scalar \(\lVert v\rVert\in \mathbb{R}\) satisfying the following properties.
positivity: \(\lVert v\rVert\ge0\)
definiteness: \(\lVert v\rVert=0\) if and only if \(v=0\)
absolute scalability: \(\lVert \lambda v\rVert=\lvert \lambda\rvert \lVert v \rVert\)
triangle inequality: \(\lVert u+v \rVert\le \lVert u \rVert + \lVert v \rVert\)
Euclidean or 2-norm or square norm
\[\lVert x\rVert=\sqrt{\lvert x_1\rvert^2+\cdots+\lvert x_n\rvert^2}\]
1-norm or Manhattan norm or taxicab norm
\[\lVert x\rVert=\lvert x_1\rvert+\cdots+\lvert x_n\rvert\]
\(\infty\)-norm or max norm
\[\lVert x\rVert=\max_{k=1,\ldots,n} \lvert x_k \rvert\]
\(p\)-norm, \(p\ge 1\)
\[\lVert x\rVert=\left(\lvert x_1\rvert^p+\cdots+\lvert x_n\rvert^p\right)^{1/p}\]
\(V\): inner product space. \(\lVert v\rVert=\sqrt{\langle v,v\rangle}\) is a norm.
Cauchy-Shwarz: \(\lvert\langle u,v\rangle\rvert\le \lVert u\rVert\lVert v\rVert\)
Proof for triangle inequality
\(\begin{align} \lVert u+v\lVert^2 &= \langle u+v,u+v\rangle\\ &= \langle u,u\rangle + \langle u,v\rangle + \langle v,u\rangle + \langle v,v\rangle\\ &= \langle u,u\rangle + 2 \text{Re}\langle u,v\rangle + \langle v,v\rangle\\ &\le \langle u,u\rangle + 2 \lvert\langle u,v\rangle\rvert + \langle v,v\rangle\\ &\le \lVert u\rVert^2+2\lVert u\rVert\lVert v\rVert+\lVert v\rVert^2\\ &=(\lVert u\rVert+\lVert v\rVert)^2 \end{align}\)
2-norm: from dot product
\(p\)-norm for \(p\ne2\): not from any inner product
\(V\): inner product space and norm defined from inner product
Pythogorean law: if \(u,v\in V\) are orthogonal, \[\lVert u+v\rVert^2=\lVert u\rVert^2+\lVert v\rVert^2\]
Parallelogram law: if \(u,v\in V\), \[\lVert u+v\rVert^2+\lVert u-v\rVert^2=2(\lVert u\rVert^2+\lVert v\rVert^2)\]
Exercise: A norm is from an inner product if and only if it satisfies the parallelogram law.