Adaptive combination of evidence for information retrieval
The ubiquity of the World Wide Web and the growth in both size and availability of collections of free text in general argues for smarter and more effective techniques for finding relevant documents. We explore one class of IR system algorithms (or “search engines”) based on combining the lists of documents returned by other IR systems, attempting to test the adage “two heads are better than one.” These algorithms also have the critical feature that the they are adaptive, taking feedback from the user and using it to change the way they combine the component search engines. We explore two specific neural network based models: the Linear Combination (LC), which scores a document as a weighted sum of the scores the document received from the component systems, and the Adaptive Combination of Evidence (ACE) model, which modifies how much it listens to any one of the search engines based on the semantic content of the document. Via extensive experiments combining a wide range of search engines from the standardized TREC database, we discover not only which types of IR tasks each combination model is best suited for, but also what types of IR systems it makes sense to combine. We also characterize the amount of gain achievable by adding progressively more search engines to the combination, and describe methods which will ensure moderate to extensive levels of improvement.