CHAPTER 4

LEAST SQUARES

Beginning with the problem of a linear system Ax = b that does not have a solution, in this short chapter, we investigate the possibility of finding a vector x* that even though it does not solve the given system, it minimizes the error images. The matrix A under consideration is an arbitrary rectangular m × n matrix, but since the minimizing solution is related to the matrix AT A, we start the exposition by studying quadratic functions where the matrix involved is symmetric positive definite. The minimization problem of least squares is directly related to orthogonal projection matrices; thus, we study such projections onto the column space of A and then we find explicit expressions for these orthogonal projections onto an arbitrary subspace S. The least squares problem can also be studied, especially in the computational framework, through matrix factorizations introduced in Chapter 3, in particular, QR and Choleski factorization. Finally, we also consider minimization problems under certain constraints by using Lagrange multipliers. We show that this can boil down to solving a linear system of equations.

4.1 PROJECTIONS AND NORMAL EQUATIONS

Recall that according to Theorem 1.49, the system Ax = b has a solution if and only if b is in the column space of A, col(A). However, in several systems of this type coming from some applications, b is not necessarily in ...

Get A First Course in Applied Mathematics now with the O’Reilly learning platform.

O’Reilly members experience live online training, plus books, videos, and digital content from nearly 200 publishers.