Optimization for Learning and Control

2Linear Algebra

Linear algebra is the study of vector spaces and linear maps on such spaces. It constitutes a fundamental building block in optimization and is used extensively for theoretical analysis and derivations as well as numerical computations.

A procedure for solving systems of simultaneous linear equations appeared already in an ancient Chinese mathematical text. Systems of linear equations were introduced in Europe in the seventeenth century by René Descartes in order to represent lines and planes by linear equations and to compute their intersections. Gauss developed the method of elimination. Further important developments were done by Gottfried Wilhelm von Leibniz, Gabriel Cramer, Hermann Grassmann, and James Joseph Sylvester, the latter introducing the term “matrix.”

The purpose of this chapter is to review key concepts from linear algebra and calculus in finite‐dimensional vector spaces as well as a number of useful identities that will be used throughout the book. We also discuss some computational aspects, including a number of matrix factorizations and their application to solving systems of linear equations.

2.1 Vectors and Matrices

We start by introducing vectors and matrices. A vector of length is an ordered collection of numbers,

where is the th element ...