The EM-EP algorithm for Gaussian process classification
Gaussian process classifiers (GPCs) are fully statistical kernel classification models derived from Gaussian processes for regression. In GPCs, the probability of belonging to a certain class at an input location is monotonically related to the value of some latent function at that location. Starting from a prior over this latent function, the data are used to infer both the posterior over the latent function and the values of hyperparameters determining various aspects of the function. GPCs can also be viewed as graphical models with latent variables. Based on the work of [1, 2], we present an approximate EM algorithm, the EM-EP algorithm for learning both the latent function and the hyperparameters of a GPC. The algorithm alternates the following steps until convergence. In the E-step, given the hyperparameters, a density for the latent variables defining the latent function is computed via the Expectation-Propagation (EP) algorithm [1, 2]. In the M-step, given the density for the latent values, the hyperparameters are selected to maximize a variational lower bound on the marginal likelihood (i.e. the model evidence). This algorithm is found to converge in practice and provides an efficient Bayesian framework for learning hyperparameters of the kernel. We examine the role of various different hyperparameters which model labeling errors, the lengthscales (i.e. relevances) of different features, and sharpness of the decision boundary. The added flexibility these provide results in significantly improved performance. Experimental results on synthetic and real data sets show that the EM-EP algorithm works well, with GPCs giving equal or better performance than support vector machines (SVMs) on all data sets tested.