Confidence Bands for Roc Curves
We address the problem of comparing the performance of classifiers. In this paper we study techniques for generating and evaluating bands on ROC curves. Historically this has been done using one-dimensional confidence intervals by freezing one variable - false-positiverate, or threshold on the classification scoring function. Weadapt two prior methods and introduce a new radial sweepmethod to generate confidence bands. We show, throughempirical studies, that the bands are too tight and introducea general optimization methodology for creatingbands that better fit the data, as well as methods for evaluatingconfidence bands. We show empirically that theoptimized confidence bands fit much better and that, usingour new evaluation method, it is possible to gauge therelative fit of different confidence bands.