Analysis and Linear Algebra: The Singular Value Decomposition and Applications

Book excerpt: This book provides an elementary analytically inclined journey to a fundamental result of linear algebra: the Singular Value Decomposition (SVD). SVD is a workhorse in many applications of linear algebra to data science. Four important applications relevant to data science are considered throughout the book: determining the subspace that “best” approximates a given set (dimension reduction of a data set); finding the “best” lower rank approximation of a given matrix (compression and general approximation problems); the Moore-Penrose pseudo-inverse (relevant to solving least squares problems); and the orthogonal Procrustes problem (finding the orthogonal transformation that most closely transforms a given collection to a given configuration), as well as its orientation-preserving version. The point of view throughout is analytic. Readers are assumed to have had a rigorous introduction to sequences and continuity. These are generalized and applied to linear algebraic ideas. Along the way to the SVD, several important results relevant to a wide variety of fields (including random matrices and spectral graph theory) are explored: the Spectral Theorem; minimax characterizations of eigenvalues; and eigenvalue inequalities. By combining analytic and linear algebraic ideas, readers see seemingly disparate areas interacting in beautiful and applicable ways.