Relationship between gradient and EM steps in latent variable models
We present a close relationship between Expectation Maximization algorithm and direct optimization approaches such as gradient-based methods for parameter learning. We show that the step EM takes in the parameter space and true gradient are related by the symmetric positive definite P matrix, and provide an explicit form of this matrix for several widely used latent variable models. We then go on deriving a general form of the P matrix for the regular exponential family in terms of its natural parameters.