Digital Communications 1

2Measurement of Information of a Discrete Source and Channel Capacity

2.1. Introduction and definitions

The purpose of a message is to provide information to a recipient. Information, like energy, for that matter, is a fundamental notion, which is part of our daily life. It can be transported (transmitted), stored (memorized) and transformed (processed).

The value of information lies in the surprise effect it causes; it is all the more interesting because it is less predictable. As a result, information is assimilated to an event of a random nature, considered, in the following, as a random variable.

The measure of the information is then given by measuring the uncertainty of an event system, that is to say, the random choice of an event in a set of possible and distinct events.

We will use the terms message or sequence to indicate any finite sequence of symbols taken in an alphabet A = { } of a discrete source: a finite set of predetermined symbols, for example:

We will use the term discrete source of information S_n to indicate the system selecting, from the set of symbols from A_s = S, the symbols to transmit:

The choice of s_m,n is carried out according to a given probability law, which can be steady temporally or variable over time.

2.2. Examples of discrete sources