What is SVD and Pseudo Inverse
For matrix \(P \in R^{M\times N}\), there exists a decomposition such that
\[P = U \Sigma V^T,\]
where \(U \in R^{M \times M}\) and \(V \in R^{N \times N}\) are two rotation matrixes satisfying
\(UU^T = U^TU = I^{M\times M}\) and \(VV^T = V^TV = I^{N \times N}\),
and \(\Sigma\) is a diagonal matrix of size \(M \times N\) with non-negative diagonal elements.
Following is an example of SVD
\[\begin{split} \begin{bmatrix}
1 & 2 & 3\\
4 & 5 & 6
\end{bmatrix}
=
\begin{bmatrix}
-0.3863 & -0.9224\\
-0.9224 & 0.3863
\end{bmatrix}
\begin{bmatrix}
9.5080 & 0 & 0\\
0 & 0.7729 & 0
\end{bmatrix}
\begin{bmatrix}
-0.4287 & -0.5663 & -0.7039\\
0.8060 & 0.1124 & -0.5812\\
0.4082 & -0.8165 & 0.4082
\end{bmatrix}\end{split}\]
For matrix \(P\), does it have a inverse matrix or something similar? Since \(P\) is not necessarily a square matrix, \(M \neq N\), it doesn’t have inverse. But we can find another \(N \times M\) matrix \(P'\) such that
\[P P' = I\]
Pseudo-inverse
For matrix \(P \in R^{M\times N}\) with SVD \(P = U \Sigma V^T\), it’s pseudo-inverse \(P^\dagger\) is
\[P^\dagger = V \Sigma^\dagger U^T,\]
where \(\Sigma^\dagger\) is the transpose of \(\Sigma\) and the positive diagonal elements are the reciprocal of the original.
For example,
\[\begin{split}\begin{bmatrix}
1 & 2 & 3\\
4 & 5 & 6
\end{bmatrix}
=
\begin{bmatrix}
-0.3863 & -0.9224\\
-0.9224 & 0.3863
\end{bmatrix}
\begin{bmatrix}
9.5080 & 0 & 0\\
0 & 0.7729 & 0
\end{bmatrix}
\begin{bmatrix}
-0.4287 & -0.5663 & -0.7039\\
0.8060 & 0.1124 & -0.5812\\
0.4082 & -0.8165 & 0.4082
\end{bmatrix}\end{split}\]
,and
\[\begin{split}\begin{bmatrix}
1 & 2 & 3\\
4 & 5 & 6
\end{bmatrix}^\dagger
=
\begin{bmatrix}
-0.4287 & 0.8060 & 0.4082\\
-0.5663 & 0.1124 & -0.8165\\
-0.7039 & -0.5812 & 0.4082
\end{bmatrix}
\begin{bmatrix}
0.1052 & 0\\
0 & 1.2939\\
0 & 0
\end{bmatrix}
\begin{bmatrix}
-0.3863 & -0.9224\\
-0.9224 & 0.3863
\end{bmatrix}\end{split}\]