4 From Model Selection to Adaptive Estimation
Many different model selection information criteria can be found in the literature in various contexts including regression and density estimation. There is a huge amount of literature concerning this subject and we shall, in this paper, content ourselves to cite only a few typical references in order to illustrate our presentation. Let us just mention AIC, Cp or CL, BIC and MDL criteria proposed by Akaike (1973), Mallows (1973), Schwarz (1978), and Rissanen (1978) respectively. These methods propose to select among a given collection of parametric models that model which minimizes an empirical loss (typically squared error or minus log-likelihood) plus some penalty term which is proportional to the dimension of the model. From one criterion to another the penalty functions differ by factors of logn, where n represents the number of observations. The reasons for choosing one penalty rather than another come either from information theory or Bayesian asymptotic computations or approximate evaluations of the risk on specific families of models. Many efforts were made to understand in what circumstances these criteria allow to identify the right model asymptotically (see Li (1987) for instance). Much less is known about the performances of the estimators provided by these methods from a nonparametric point of view. Let us consider the particular context of density estimation in L for instance. By a nonparametric point of view, we mean that the unknown density does not necessarily belong to any of the given models and that the best model should approximately realize the best trade-off between the risk of estimation within the model and the distance of the unknown density to the model. When the models have good approximation properties (following Grenander (1981) these models will be called sieves), an adequate choice of the penalty can produce adaptive estimators in the sense that they estimate a density of unknown