Autonomous Learning Systems: From Data Streams to Knowledge in Real-time

Appendix A

Mathematical Foundations

A.1 Probability Distributions

• Gaussian (normal) The Gaussian, also known as normal (when the mean is zero) distribution is the most widely used one in practice because of its properties. For example, the so called central limit theorem states that the averages of random variables and signals tend to Gaussian; a sum of two Gaussians is a Gaussian again; the distribution that maximises the entropy for a given variance is the Gaussian; it is robust to linear transformations, and so on. For a single variable, x it is defined by two parameters, the mean, μ and the variance, σ2 or equivalently by its square root, the standard deviation, σ:

  (A.1)

  The entropy is given by:

  (A.2)

  The inverse of the variance, σ−2 is called the precision. In a vector form, the Gaussian distribution is defined by the n-dimensional vector, μ and an (n × n)-dimensional covariance matrix, Σ as described in Chapter 3. The covariance matrix is by definition symmetric and positive (since it is formed by squares of distances) quantities/elements.

  (A.3)

  The entropy is then given by:

  (A.4)

  The precision is again defined as an inverse, but in this case of the covariance matrix, ...