Expectation-conjugate gradient: An alternative to EM
We show a close relationship between bound optimization (BO ) algorithms such as Expectation-Maximization and direct optimi zation (DO) algorithms such as gradient-based methods for parameter le arning. We identify analytic conditions under which BO algorithms exh ibit QuasiNewton convergence behavior, and conditions under which th ese algorithms possess poor, first-order convergence. In particula r, for the EM algorithm we show that if a certain measure of the proportion of missing information is small, then EM exhibits Quasi-Newton behavi or; when it is large, EM converges slowly. Based on this analysis, we pre sent a new Expectation-Conjugate-Gradient (ECG) algorithm for maxi mum likelihood estimation, and report empirical results, showing tha t, as predicted by the theory, ECG outperforms EM in certain cases.