15 Math Concepts Every Data Scientist Should Know

13

Information Theory

Information theory and information-theoretic concepts are very useful, but it is unlikely that you will have to formally make use of them as a data scientist. By this, we mean that you are unlikely to have to use detailed information-theoretic mathematical proofs or techniques in your work. But – and it’s an important but – the ideas and ways of thinking that information theory introduces are worth understanding. And that is what this chapter aims to achieve. To do that, we will cover the following topics:

• What is information and why is it useful?: Here, we’ll define precisely what we mean by information and how we quantify it mathematically
• Entropy as expected information: Here, we’ll introduce the concept of the average ...