Optimization for Learning and Control

3Probability Theory

In this chapter, we will discuss the basics of probability theory. It is a branch of mathematics where uncertain events are given a number between zero and one to describe how likely they are. Loosely speaking this number should be close to the relative frequency with which the event occurs when it is repeated many times. As an example we may consider throwing a fair dices one hundred times and recording how many times a one occurs. If it occurs 18 times, the relative frequency is . This is close to the theoretical value of the probability which is 1/6. The reason we know it should be 1/6 is that all the possible six outcomes of the experiment should have the same probability if the dice is fair. In case a probability is one, we are almost sure the event will occur, and if it is zero, we are almost sure it will not occur.

The roots of probability theory go back to the Arab mathematician Al‐Khalil who studied cryptography. Initially, probability theory only considered combinatorial problems. The theory is much easier for this case as compared to the case when the number of events is not countable. Mathematicians struggled for many years to provide a solid foundation, and it was not until in 1933 when Andrey Nikolaevich Kolmogorov made an axiomatic definition of probabilities that the problem was resolved, and modern probability theory was born. We are however ...