A note on the evidence and Bayesian Occam's razor
In his thesis, MacKay (1991) introduced figure 1, explaining how Bayes rule provides an automatic “Occam’s razor” effect, penalizing unnecessarily complex models. This figure has been adopted by several authors in the same schematic form. Here, after briefly reviewing necessary material, we compute a realization of the plot for a toy data modeling problem. We discuss interesting aspects of this plot and their implications for understanding model complexity.