ROC Confidence Bands : An Empirical Study 0
This paper is about constructing confidence bands around an R OC curve such that (1 − δ)% of the ROC curves traced by data sets of sizer will fall completely within the bands. We introduce to the machine learning community three methods from the medic al field that are applicable to generate such bands. We then eval uate these methods on the simple case of “binormal” distribution s— the scores for positive and the score for negative instances ar drawn from two normal distributions. We show that none of the methods generate appropriate bands and investigate two typ es of variances problems. We show that widening the bands does not produce the proper bandwidths but that fitting a normal distr ibution to the observed drawn samples and drawing samples from this distribution (parametric bootstrap) does generate ba nds that are much closer to the desired coverage although still not pe rfect. We tested the original methods as well as parametric bootstr ap on the covertype data set from the UCI ML-repority. The origina l methods perform the same as in the synthetic case, whereas th parametric bootstrap technique did not yield the expected r esults. This is primarily due to not being able to generate a good fit fo r the score distributions. Whether it is possible to fit well-b ehaving parametric distribution to learned models is an open questi on we leave to the machine learning community to answer.