Prediction Revisited

5Fit

In Chapter 4, we described how to predict an unknown outcome by taking a relevance-weighted average of its past values. This process will always yield a prediction, whether the data is useful for predicting the unknown outcome or not. Therefore, it is important to evaluate the relationships underlying a prediction to determine its quality, using a measure we call fit. In general, higher fit indicates greater consistency, and should inspire more confidence in a prediction.

Fit Conceptually

Your predictions will most likely be wrong. If you are good, they will not be wrong by much. Professional forecasters and pundits have an incentive to hide their doubts. But for the rest of us, it is usually better to confront our uncertainty head-on.

Recall that a single prediction is a weighted average of many observations. We can think of each observation as a vote for the unknown outcome, reflecting both the relevance of the attributes, X, and the value of the observed outcome, Y. If relevance and outcomes are aligned, the final tally probably reflects an underlying truth. But if relevance and outcomes are inconsistent, their aggregate tally could be meaningless. In general, we should be more confident in a prediction if its observations display consistency.

Imagine planning a trip to Paris and asking your friends how much they think it will cost. You are likely to get a range of estimates, which reflects uncertainty. But if some of your friends have traveled to Paris before—which ...