Rethinking the Recommender Research Ecosystem: Reproducibility, Openness, and LensKit
Publication Type  Conference Paper
Year of Publication  2011
Authors  Ekstrand, M.D.; Ludwig, M.; Konstan, J.A.; Riedl, J.T.
Conference Name  The Fifth ACM Conference on Recommender Systems
Conference Location  Chicago, IL
Series Title  Proceedings of the Fifth ACM Conference on Recommender Systems
Pagination  133-140
Conference Start Date  10/23/2011
Publisher  Association of Computing Machinery
Key Words  Recommender systems; implementation; evaluation
Abstract  

Recommender systems research is being slowed by the di- culty of replicating and comparing research results. Published research uses various experimental methodologies and metrics that are dicult to compare. It also often fails to suciently document the details of proposed algorithms or the evaluations employed. Researchers waste time reimplementing well-known algorithms, and the new implementations may miss key details from the original algorithm or its subsequent re nements. When proposing new algorithms, researchers should compare them against nely-tuned implementations of the leading prior algorithms using state-of-the-art evaluation methodologies. With few exceptions, published algorithmic improvements in our eld should be accompanied by working code in a standard framework, including test harnesses to reproduce the described results. To that end, we present the design and freely distributable source code of LensKit, a exible platform for reproducible recommender systems research. LensKit provides carefully tuned implementations of the leading collaborative ltering algorithms, APIs for common recommender system use cases, and an evaluation framework for performing reproducible oine evaluations of algorithms. We demonstrate the utility of LensKit by replicating and extending a set of prior comparative studies of recommender algorithms | showing limitations in some of the original results |and by investigating a question recently raised by a leader in the recommender systems community on problems with error-based prediction evaluation.

URL  http://doi.acm.org/10.1145/2043932.2043958
DOI  10.1145/2043932.2043958
Export  Tagged XML BibTex
AttachmentSize
p133-ekstrand.pdf638.92 KB