Mutual Information and Bayes Methods for Learning a Distribution
Each parameter w in an abstract parameter space W is associated with a diier-ent probability distribution on a set Y. A parameter w is chosen at random from W according to some a priori distribution on W , a n d n conditionally independent random variables Y n = Y 1 : : : Y n are observed with common distribution determined by w. Viewing W as a random variable, we obtain bounds on the mutual information between the random variable W , giving the choice of parameter , and the random variable Y n , giving the sequence of observations. This quantity is the cumulative risk in predicting Y 1 : : : Y n under the log loss, minus the risk if the true parameter w is known. The upper bounds are stated in terms of the Laplace transform of the rate of growth of the volume of relative e n tropy neighborhoods in the parameter space W , and the lower bounds are given in terms of the corresponding quantity using Hellinger neighborhoods. We show how these bounds can be interpreted in terms of an average local dimension of the parameter space W under suitable conditions.