Bayesian Statistics: An Introduction, 4th Edition

3.11 Reference posterior distributions

3.11.1 The information provided by an experiment

Bernardo (1979) suggested another way of arriving at a reference standard for Bayesian theory. The starting point for this is that the log-likelihood ratio can be regarded as the information provided by the observation x for discrimination in favour of against (cf. Good, 1950, Section 6.1). This led Kullback and Leibler (1951) to define the mean information in such data to be

(cf. Kullback, 1968, and Barnett, 1999, Section 8.6).1 Note that although there is a relationship between information as defined here and Fisher’s information I as defined in Section 3.3 earlier (see Kullback, 1968, Chapter 2, Section 6), you are best advised to think of this as a quite separate notion. It has in common with Fisher’s information the property that it depends on the distribution of the data rather than on any particular value of it.

Following this, Lindley (1956) defined the expected amount of information that ...