Adaptive Filters

CHAPTER 3 NORMAL EQUATIONS

In Sees. 1.2 and 2.1 we studied the problem of determining the optimal function h(·) that minimizes the mean-square error of estimating a random variable x from another random variable y. Specifically, we solved

(3.1)

over all functions h(·) of y. The optimal solution was found to be the conditional expectation of x given y, i.e.,

Such conditional expectations are generally hard to evaluate in closed-form, except in some special cases. We encountered three such cases in Part I (Optimal Estimation), namely the case of a BPSK signal embedded in additive Gaussian noise (Ex. 1.2), the case of jointly Gaussian random variables (studied in Sees. 1.4 and 2.2), and the case of random variables with exponential distributions (studied in Prob. 1.16). Due to the difficulty in evaluating E (x|y) in general, it is common practice to restrict the choice of h(·) to the subclass of affine functions of y, i.e., to functions of the form

(3.2)

for some matrix K and for some vector b to be determined. For general vector-valued variables {x, y}, K is a matrix and b is a vector. When {x, y} are scalars, then {K, b} will also be scalars. If x is a scalar and y is a vector, ...