Machine Learning with Python Cookbook, 2nd Edition

Chapter 18. Naive Bayes

18.0 Introduction

Bayes’ theorem is the premier method for understanding the probability of some event, $P(A\mid B)$, given some new information, $P(B\mid A)$, and a prior belief in the probability of the event, $P\left(A\right)$:

$$P(A\mid B)=\frac{P(B\mid A)P\left(A\right)}{P\left(B\right)}$$

The Bayesian method’s popularity has skyrocketed in the last decade, increasingly rivaling traditional frequentist applications in academia, government, and business. In machine learning, one application of Bayes’ theorem to classification comes in the form of the naive Bayes classifier. Naive Bayes classifiers combine a number of desirable qualities in practical machine learning into a single classifier. These include:

• An intuitive approach

• The ability to work with small data

• Low computation costs for training and prediction

• Often solid results in a variety of settings

Specifically, a naive Bayes classifier is based on:

$$P(y\mid x_1,\dots ,x_j)=\frac{P(x_1,\dots ,x_j\mid y)P\left(y\right)}{P(x_1,\dots ,x_j)}$$

where:

• $P(y\mid x_1,\dots ,x_j)$ is called the posterior and is the probability that an observation is class $y$ given the observation’s values for the $j$ features, $x_1,\dots ,x_j$.

• $P(x_1,\dots ,x_j\mid y)$ is called likelihood and is the likelihood of an observation’s values for features $x_1,\dots ,x_j$ given their class, $y$.

• $P\left(y\right)$ is called the prior and is our belief for the probability of class $y$ before looking at the data.

• $P(x_1,\dots ,x_j)$ is called the marginal probability.

In naive Bayes, ...