HCIL-2010-08

Yeh, T., White, B., Davis, L., Katz, B.
May 2010
HCIL-2010-08
Many online articles contain useful know-how knowledge about GUI applications. Even though these articles tend to be richly illustrated by screenshots, no system has been designed to take advantage of these screenshots to visually search know-how articles e ectively. In this paper, we present a novel system to index and search software knowhow articles that leverages the visual correspondences between screenshots. To retrieve articles about an application, users can take a screenshot of the application to query the system and retrieve a list of articles containing a matching screenshot. Useful snippets such as captions, references, and nearby text are automatically extracted from the retrieved articles and shown alongside with the thumbnails of the matching screenshots as excerpts for relevancy judgement. Retrieved articles are ranked by a comprehensive set of visual, textual, and site features, whose weights are learned by RankSVM. Our prototype system currently contains 150k articles that are classi ed into walkthrough, book, gallery, and general categories. We demonstrated the system's ability to retrieve matching screenshots for a wide variety of programs, across language boundaries, and provide subjectively more useful results than keyword-based web and image search engines.
Return to Main TRs Page