Digital Communication Systems

CHAPTER

5

Information Theory

5.1   Introduction

As mentioned in Chapter 1 and reiterated along the way, the purpose of a communication system is to facilitate the transmission of signals generated by a source of information over a communication channel. But, in basic terms, what do we mean by the term information? To address this important issue, we need to understand the fundamentals of information theory.¹

The rationale for studying the fundamentals of information theory at this early stage in the book is threefold:

1.  Information theory makes extensive use of probability theory, which we studied in Chapter 3; it is, therefore, a logical follow-up to that chapter.

2.  It adds meaning to the term “information” used in previous chapters of the book.

3. Most importantly, information theory paves the way for many important concepts and topics discussed in subsequent chapters.

In the context of communications, information theory deals with mathematical modeling and analysis of a communication system rather than with physical sources and physical channels. In particular, it provides answers to two fundamental questions (among others):

1.  What is the irreducible complexity, below which a signal cannot be compressed?

2.  What is the ultimate transmission rate for reliable communication over a noisy channel?

The answers to these two questions lie in the entropy of a source and the capacity of a channel, respectively:

1.  Entropy is defined in terms of the probabilistic behavior of ...