Feature Bagging: Preventing Weight Undertraining in Structured Discriminative Learning
Discriminatively-trained probabilistic models are very popular in NLP because of the latitude they afford in designing features. But training involves complex trade-offs among weights, whichcanbedangerous: afew highlyindicative features can swamp the contribution of many individually weaker features, causing their weights to be undertrained. Such a model is less robust, for the highly-indicative features may be noisy or missing in the test data. To ameliorate this weight undertraining, we propose a new training method, called feature bagging, in which separate models are trained on subsets of the original features, and combined using a mixture model or a product of experts. We evaluate feature bagging on linear-chain conditional random fields for two natural-language tasks. On both tasks, the feature-bagged CRF performs better than simply training a single CRF on all the features.