A First Course in Applied Mathematics

CHAPTER 4

LEAST SQUARES

Beginning with the problem of a linear system Ax = b that does not have a solution, in this short chapter, we investigate the possibility of finding a vector x* that even though it does not solve the given system, it minimizes the error . The matrix A under consideration is an arbitrary rectangular m × n matrix, but since the minimizing solution is related to the matrix A^T A, we start the exposition by studying quadratic functions where the matrix involved is symmetric positive definite. The minimization problem of least squares is directly related to orthogonal projection matrices; thus, we study such projections onto the column space of A and then we find explicit expressions for these orthogonal projections onto an arbitrary subspace S. The least squares problem can also be studied, especially in the computational framework, through matrix factorizations introduced in Chapter 3, in particular, QR and Choleski factorization. Finally, we also consider minimization problems under certain constraints by using Lagrange multipliers. We show that this can boil down to solving a linear system of equations.

4.1 PROJECTIONS AND NORMAL EQUATIONS

Recall that according to Theorem 1.49, the system Ax = b has a solution if and only if b is in the column space of A, col(A). However, in several systems of this type coming from some applications, b is not necessarily in ...