Generative binary
codes
AbstractA generative or latent variable model corresponds to a
Bayesian network where arcs point from (presumed) hidden sources
to observed variables. In this paper we introduce a particular
generative model with binary
valued hidden sources (i. e. each source can be either ‘on’ or
‘off’) but continuous observable variables. The purpose of this
model is to learn a binary,
distributed code for continuous data, for example to
learn a bit code for gray value image data. For inference we
rely on a mean field approximation. A novel and surprisingly
simple derivation of general mean field equations is given. The
structure of our model is chosen such that it is optimally
suited for the structure of mean field inference. Hence, the
mean field equations can be solved efficiently even with a few
hundred hidden nodes, thus allowing one to learn highly
distributed codes. For learning the parameters in the generative
model from data, an appropriate EM-procedure is derived. In the
second part of the paper we employ our approach to learning a
sparse representation of natural images which is applied to code
and to denoise images. The image compression rate is comparable
to JPEG-coding and image denoising clearly outperforms standard
methods such as Wiener filtering. In the outlook we present
potential further directions of research in particular with
respect to a more complex hidden topography.