Introduction to Cryptography with Coding Theory, 3rd Edition

20.5 The Entropy of English

In an English text, how much information is obtained per letter? If we had a random sequence of letters, each appearing with probability 1/26, then the entropy would be ${log}_2(26)=4.70$; so each letter would contain 4.7 bits of information. If we include spaces, we get ${log}_2(27)=4.75$. But the letters are not equally likely: $a$ has frequency .082, $b$ has frequency .015, etc. (see Section 2.3). Therefore, we consider

$$-(.082{log}_2.082+.015{log}_2.015+\cdots )=\mathrm{4.18.}$$

However, this doesn’t tell the whole story. Suppose we have the sequence of letters we are studyin. There is very little uncertainty as to what the last letter is; it is easy to guess that it is $g$. Similarly, if we see the letter $q$, it is extremely likely that the next letter ...