
i
i
i
i
i
i
i
i
1.2 Source Coding 13
I(S;
ˆ
S) = h(S) −h(S |
ˆ
S), (1.33)
=
1
2
log(2πe)σ
2
s
−h(S −
ˆ
S |
ˆ
S), (1.34)
≥
1
2
log(2πe)σ
2
s
−h(S −
ˆ
S), (1.35)
≥
1
2
log(2πe)σ
2
s
−h(N(0,E(S −
ˆ
S)
2
)), (1.36)
=
1
2
log(2πe)σ
2
s
−
1
2
log(2πe)E(S −
ˆ
S)
2
, (1.37)
≥
1
2
log(2πe)σ
2
s
−
1
2
log(2πe)D, (1.38)
=
1
2
log
σ
2
s
D
, (1.39)
where Eq. (1.35) follows from the fact that conditioning reduces entropy and Eq.
(1.36) follows from the fact that normal distribution maximizes entropy for a given
second moment. Hence
R(D) ≥
1
2
log
σ
2
s
D
. (1.40)
To find the conditional density f ( ˆs | s) that achieves this lower bound Eq. (1.40),
it is usually more convenient to look at the conditional density f (s | ˆs), which is
sometimes called ...