A Method for Evaluating Elicitation Schemes for Probabilities
We present an objective approach for evaluating probabilit,, clicitati~m methods in probabilistic models. Our method draw~, tm ideas from research on learning Bayesian etworks: iI ~c assume that the expert’s knowledge is manifested es~,cntially a.,, a database of records that have been collected in the course of the expert’s experience, and if this database of records i available It:, us, then the ~tructure and parameters of the expert’s beliefs could be reliabl)’ constructed using techniques for Bayesian learning from data. This learned model could, m turn. be compared to elicited models to judge the effectiveness of the clicitation process. We describe a general pn~ccdurc by which it is possible to capture the data corre,,ptmding to the expert’s beliefs, and we present a simple experiment in which we utilize this technique to compare three methods t~r eliciting discrete prt~babilities: (I) direct numerteal assc,,smcnt. (2) the probability wheel, and (3) the scaled iwohahihty bar. We show that t’or our domain, the scaled prub:lhlht} bar i~, the retest eflcctive tool for probability clici-