A research agenda for the evaluation of semantic search interfaces

For research libraries to move successfully from experimentation to implementation with library linked data and semantic search interfaces, we need to better understand how these systems can best support users' information discovery needs. This presentation outlines a research agenda and methodologies we will use in the University of Pennsylvania Libraries to evaluate, implement, and extend library discovery systems using enriched and linked metadata, including systems used in the LD4 community. Our intended audience is librarians and developers of linked data systems with an interest in discovery. Our research agenda begins with user tasks described in the IFLA Library Reference Model (LRM), but also considers extensions to those basic tasks that linked data-enabled systems support. These extensions include enhanced topical browsing; the discovery of works, people, and topics across multiple information collections; and the selection and delivery of the most appropriate copy of a sought-after work in a multi-library context that takes into account both content and obtainability. Our evaluation methodologies include both quantitative and qualitative analyses. We will begin with analyses of current discovery search logs, and continue with user studies to determine user tasks that are well-supported by our semantic search systems, and to identify gaps in data, services, and interface designs for meeting user needs. We may also prototype extensions to our platforms and metadata and test their effectiveness. Platforms we will use for evaluation include the SHARE-VDE linked data catalog being developed with various library partners, the Blacklight-based catalog developed at Penn for our local library collection and for an experimental Ivy+ shared discovery service, and Penn's Online Books Page catalog and accompanying Forward to Libraries service. We hope in our research to articulate the best uses of semantic searching, and the most effective investments in systems, interfaces, and metadata to support discovery in a linked-data environment.