Likelihood Ratio of Unidenti Þ able Models and Multilayer Neural Networks
This paper discusses the behavior of the maximum likelihood estimator, when the true parameter cannot be identiÞed uniquely. Among many statistical models with unidentiÞability, neural network models are the main concern of this paper. The set of unidentiÞable true parameters is formulated as a conic singularity of the model, which is embedded in an inÞnite dimensional space of probability density functions. It has been known in some models with unidentiÞability the asymptotics of the likelihood ratio of MLE has an unusually larger order. Following Hartigans idea, the likelihood ratio of MLE is described by the supremum of an empirical process over a set of functions, and a useful sufficient condition of such larger orders is derived. This result is applied to neural network models, and a larger order is observed if the true function is realized by a network with a smaller number of hidden units than the model. A stronger lower bound of the order of likelihood ratio is also derived on condition that there are at least two redundant hidden units to realize the true function.