Statistical Learning and Kernel Methods in Bioinformatics
We briefly describe the main ideas of statistical learning theory, support vector machines, and kernel feature spaces. In addition, we present an overview of applications of kernel methods in bioinformatics.1 1 An Introductory Example In this Section, we formalize the problem of pattern recognition as that of classifying objects called “pattern” into one of two classes. We introduce a simple pattern recognition algorithm that illustrates the mechanism of kernel methods. Suppose we are given empirical data "!$#"% (1) Here, the domain is some nonempty set that the patterns '& are taken from; the "& are called labels or targets. Unless stated otherwise, indices ( and ) will always be understood to run over the training set, i.e., ( )+* #, . . Note that we have not made any assumptions on the domain other than it being a set. In order to study the problem of learning, we need additional structure. In learning, we want to be able to generalize to unseen data points. In the case of pattern recognition, given some new pattern / 0 , we want to predict the corresponding 1 2 "!$#"% . By this we mean, loosely speaking, that we choose such that 3 4 is in some sense similar to the training examples. To this end, we need similarity measures in and in "!$#"% . The latter is easier, as two target values can only be identical or different. For the former, we require a similarity measure 576 8 9 :; 3 <= ?>9 5 3 <= (2) i.e., a function that, given two examples and < , returns a real number characterizing their similarity. For reasons that will become clear later, the function 5 is called a kernel [28, 1, 9]. 1The present article is partly based on Microsoft TR-2000-23, Redmond, WA. 2 B. Schölkopf, I. Guyon, and J. Weston A type of similarity measure that is of particular mathematical appeal are dot products. For instance, given two vectors < : , the canonical dot product is defined as <= 6 * < (3) Here, denotes the -th entry of . The geometrical interpretation of this dot product is that it computes the cosine of the angle between the vectors and < , provided they are normalized to length # . Moreover, it allows computation of the length of a vector as , and of the distance between two vectors as the length of the difference vector. Therefore, being able to compute dot products amounts to being able to carry out all geometrical constructions that can be formulated in terms of angles, lengths and distances. Note, however, that we have not made the assumption that the patterns live in a dot product space. In order to be able to use a dot product as a similarity measure, we therefore first need to embed them into some dot product space , which need not be identical to : . To this end, we use a map 6 9 >9 6 * ' (4) The space is called a feature space. To summarize, embedding the data into has three benefits. 1. It lets us define a similarity measure from the dot product in , 5 3 <= 6 * < * ' <= (5) 2. It allows us to deal with the patterns geometrically, and thus lets us study learning algorithms using linear algebra and analytic geometry. 3. The freedom to choose the mapping will enable us to design a large variety of learning algorithms. For instance, consider a situation where the inputs already live in a dot product space. In that case, we could directly use the dot product as a similarity measure. However, we might still choose to first apply another nonlinear map to change the representation into one that is more suitable for a given problem and learning algorithm. We are now in the position to describe a simple pattern recognition algorithm. The idea is to compute the means of the two classes in feature space,