Some relationships between minimum Bayes error and information theoretical feature extraction
Feature extraction and selection are important problems in statistical learning. We study the relationships between two previously proposed principles for their optimal solution: the minimization of Bayes error and the maximization of mutual information between features and class labels. It is shown that a quantity which provides insight on this relationship is the set of non-increasing probability mass functions (NIPMFs). We derive some basic properties of the members of this set, show that any classification problem defines an ensemble of NIPMFs, and that the probability distribution of this ensemble uniquely determines the associated Bayes error and mutual information. These results are then used to show that, when the classification problem is binary and some generic constraints hold, the optimal feature space is the same under the two formulations.