Prediction Revisited

6Reliability

In Chapter 5, we characterized prediction as a voting process in which each observation casts a ballot. We argued that our confidence in a prediction depends on how closely the individual votes tend to agree. And we showed that some predictions warrant more confidence than others by virtue of their fit. We would now like to broaden our perspective to consider how reliable a particular voting procedure is overall, based on the underlying data.

Reliability Conceptually

Let us continue with the voting analogy from earlier. The votes we are talking about are not discrete choices like votes for candidates in an election. Rather, each prediction vote is the product of a previously observed outcome and its relevance. The average of all votes gives the final prediction. For this discussion, it is important to distinguish between information that is available at the time of a prediction, which is used for voting, and the actual outcome of the event we are trying to predict, which is unknown. The measure of fit we introduced in the previous chapter comes from the observed information. It captures the information inside a single prediction, and it does not consider, in any way, the unknown outcome to be revealed later.

As a brief review, the fit of a prediction tells us whether the most relevant voters tend to agree. In other words, do pairs of relevant observations have similar outcomes?¹ If the answer is usually yes, the fit is tight. Now, we must keep in mind that the ...