Hands-On Artificial Intelligence for Cybersecurity

Brier score example

In the following example, we evaluate the quality of estimated probabilities using the brier_score_loss() method of scikit-learn:

 import numpy as np from sklearn.metrics import brier_score_loss y_true = np.array([0, 1, 1, 1]) y_cats = np.array(["fraud", "legit", "legit", "legit"]) y_prob = np.array([0.2, 0.7, 0.9, 0.3]) y_pred = np.array([1, 1, 1, 0]) brier_score_loss(y_true, y_prob) brier_score_loss(y_cats, y_prob, pos_label="legit") brier_score_loss(y_true, y_prob > 0.5)

Now, let's continue the evaluation of model performances by introducing the effects deriving from the splitting of the sample dataset into training and testing subsets.