Introduction to High-Dimensional Statistics, 2nd Edition

Appendix C

Linear Algebra

C.1 Singular Value Decomposition

The Singular Value Decomposition (SVD) is a matrix decomposition that is very useful in many fields of applied mathematics. In the following, we will use that, for any n × p matrix A, the matrices A^TA and AA^T are symmetric positive semi-definite.

Theorem C.1 Singular value decomposition

Any n × ρ matrix A of rank r can be decomposed as

$A={\displaystyle \sum _{j=1}^r{\sigma }_j{u}_j{v}_j^T,}$ (C.1)

where

• r = rank (A),
• σ₁ ≥ ... ≥ σ_r > 0,
• {σ²₁,…,σ²_r} are the nonzero eigenvalues of A^TA (which are also the nonzero eigenvalues of AA^T), and
• {u₁,..., u_r} and {v₁,..., v_r} are two orthonormal families of ℝⁿ and ℝ^p, such that

$A{A}^T{u}_j={\sigma }_j^2{u}_{j}and{A}^{T}A{v}_j={\sigma }_j^2{v}_j.$

The values ...