Appendix C

In this appendix we provide a brief description of the PAV algorithm used in Chapter 6. Assume that there exist a validation set of posterior probabilities (Appendix A) that H1 is true, namely Pi, and the respective ground-truth labels of the proposition that is actually true in each case, namely yi {0, 1}, where i is an index representing the case in the validation set for which a posterior probability Pi is assigned. This index i {1, …, N}, where N = N1 + N2, N1 is the number of cases in the validation set for which H1 is true, and N2 is the number of cases in the validation set for which H2 is true. Thus, N is the total number of cases in the validation set. With this notation, when yi = 1 then H1 is true for case i, and when yi = 0 then H2 is true for case i. Therefore, Pi = P(yi = 1). Moreover, without loss of generality, it is assumed that for two cases j and k in the validation set, PjPk if j < k, or in other words, that the probabilities Pi are sorted in ascending order by case index i. Each yi has a value of yi = 1 with probability Pi if H1 is true and yi = 0 with probability Pi if H2 is true. Therefore, the yi values are in fact the oracle probabilities that an oracle forecaster would elicit, as defined in Section 6.5.1. ...