Bayes-optimal performance in a discrete space
Statistical mechanics techniques have been used with success to study and understand key properties of inferential learning [1, 2]. This approach provides explicit and detailed results that are in many ways complementary to the more general, but also more blend results obtained by statistics. The case of non-smooth problems, in which the parameters that have to be estimated take discrete values, is of particular interest. On the one hand, many of the results from statistics can no longer be applied, while on the other hand, the estimation of these parameters is often a computationally hard problem. In this paper, we present a detailed analysis of a simple model of unsupervised learning [3, 4, 5, 6, 7, 8], involving a single symmetry breaking vector with binary components ±1 and highlight the differences with the case of smooth components. In particular we compare the results from Gibbs learning and Bayes learning with the ones for the best binary vector and a vector which minimizes a variationally optimal potential. The problem is as follows: p N -dimensional patterns {ξ, μ = 1, ..., p}, are sampled independently from a distribution P (ξ|B) ∼ δ(ξ ·ξ−N) exp [