Advanced Deep Learning with Python

Information theory

Information theory attempts to determine the amount of information an event has. The amount of information is guided by the following principles:

• The higher the probability of an event, the less informative the event is considered. Conversely, if the probability is lower, the event carries more informational content. For example, the outcome of a coin flip (with a probability of 1/2) provides less information than the outcome of a dice throw (with a probability of 1/6).
• The information that's carried by independent events is the sum of their individual information contents. For example, two dice rows that come up on the same side of the dice (let's say, 4) are twice as informative as the individual rows.

We'll define ...