Adaptive Filters

CHAPTER 10 LMS Algorithm

In Chapters 10–14 we start to develop the theory of adaptive algorithms by studying stochastic-gradient methods. These methods are obtained from steepest-descent implementations by replacing the required gradient vectors and Hessian matrices by some suitable approximations. Different approximations lead to different algorithms with varied degrees of complexity and performance properties. The resulting methods will be genetically called stochastic-gradient algorithms since, by employing estimates for the gradient vector, the update directions become subject to random fluctuations that are often referred to as gradient noise.

Stochastic-gradient algorithms serve at least two purposes. First, they avoid the need to know the exact signal statistics (e.g., covariances and cross-covariances), which are necessary for a successful steepest-descent implementation but are nevertheless rarely available in practice. Stochastic-gradient methods achieve this feature by means of a learning mechanism that enables them to estimate the required signal statistics. Second, these methods possess a tracking mechanism that enables them to track variations in the signal statistics. The two combined capabilities of learning and tracking are the main reasons behind the widespread use of stochastic-gradient methods (and the corresponding adaptive filters). It is because of these abilities that we tend to describe adaptive filters as “smart systems”; smart in the sense that they ...