Proceedings of the 31th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2008),pages 19-26, July 2008, Singapore.
In the context of document retrieval in the biomedical domain, this paper explores the
complex relationship between the quality of initial query results and the overall utility
of an interactive system. We demonstrate that a content-similarity browsing tool can
compensate for poor retrieval results, and that the relationship between retrieval perfor-
mance and overall utility is non-linear. Arguments are advanced with user simulations,
which characterize the relevance of documents that a user might encounter with dierent
browsing strategies. With broader implications to IR, this work provides a case study of
how user simulations can be exploited as a formative tool for automatic utility evalua-
tion. Simulation-based studies provide researchers with an additional evaluation tool to
complement interactive and Craneld-style experiments.