The Least Squares Method
In this chapter, we are going to present a series of techniques based on minimizing mean square criteria to solve linear problems. But first we are going to state a fundamental theorem, called the projection theorem. It was mentioned more or less explicitly in the affine trend suppression problem, or when we estimated the amplitudes of a harmonic signal’s components. We will see that it has major applications both in a deterministic or random context.
9.1 The projection theorem
The projection theorem is presented in mathematical form. However, readers that are not used to this formalism should not be worried, since the result expressed by relation 9.1 is quite intuitive, as it is shown in Figure 9.1.
Definition 9.1 (Hilbert space) Let be a vector space with a dot product (x, y) for any two of its elements: