Probability, Random Variables, and Random Processes: Theory and Signal Processing Applications

9.13 ORTHOGONALITY CONDITION

Next, we demonstrate an important property of the conditional-mean estimator. Orthogonality was also discussed in Chapter 5 where various properties of conditional expectation were described.

Theorem 9.8. The conditional mean estimator hMSE(x) has the following orthogonality property for all functions g(X):

(9.194)

Proof. Conditioning on X in nested expectations gives

(9.195)

In the second line, the conditioning allows us to bring g(X) outside the inner expectation, and in the last line we have used the fact that because the estimator is a function of X. Since , zero is obtained in the last expression, which completes the proof.

The orthogonality condition can be used to derive the optimal estimator, and thus it is necessary and sufficient. Assuming that (9.194) holds, then

(9.196)

for all g(X). Conditioning the right-hand side on X in a nested expectation gives

(9.197)

Since equality must hold for all functions g(X), this can happen only if . This approach ...