Digital Signal and Image Processing using MATLAB, Volume 1: Fundamentals, 2nd Edition

Chapter 9

The Least Squares Method

In this chapter, we are going to present a series of techniques based on minimizing mean square criteria to solve linear problems. But first we are going to state a fundamental theorem, called the projection theorem. It was mentioned more or less explicitly in the affine trend suppression problem, or when we estimated the amplitudes of a harmonic signal’s components. We will see that it has major applications both in a deterministic or random context.

9.1 The projection theorem

The projection theorem is presented in mathematical form. However, readers that are not used to this formalism should not be worried, since the result expressed by relation 9.1 is quite intuitive, as it is shown in Figure 9.1.

Definition 9.1 (Hilbert space) Let be a vector space with a dot product (x, y) for any two of its elements:

– the norm of an element x of is the positive number defined by ||x|| = ;

– x and y are said to be orthogonal, which is denoted by x ⊥ y, if (x, y) = 0;

– the distance between two elements x and y of is the positive number defined by d(x, y) = || ...