Article
Computer Science, Artificial Intelligence
Roberto Henriques, Luis Pinto
Summary: Recommender systems were initially introduced to address information overload in enterprises and have since been widely applied in major websites such as e-commerce, music and video streaming, travel and movie sites, social media, and mobile app stores. However, there is limited research on evaluating recommender systems. This study proposes a novel evaluation metric that incorporates user behavior in online platforms, aiming to improve customer usage beyond the baseline level. An empirical application in a real-world mobile app store demonstrates the correlation between this metric and existing ones, as well as its ability to integrate cost information.
EXPERT SYSTEMS WITH APPLICATIONS
(2023)
Article
Computer Science, Artificial Intelligence
Jacopo Tagliabue, Federico Bianchi, Tobias Schnabel, Giuseppe Attanasio, Ciro Greco, Gabriel de Souza Moreira, Patrick John Chia
Summary: The organizers of the EvalRS recommender systems competition argue for the consideration of robustness and fairness in addition to accuracy.
NATURE MACHINE INTELLIGENCE
(2023)
Article
Computer Science, Information Systems
Elisa Mena-Maldonado, Rocio Canamares, Pablo Castells, Yongli Ren, Mark Sanderson
Summary: The study reveals that false-positive metrics tend to penalize popular items, opposite to true-positive metrics, resulting in an inconsistent trend between the two types of metrics under popularity biases. Through theoretical analysis, the reasons for the disagreement between metrics are identified, as well as rare situations where they might agree. Empirical study results confirm the analytical findings, guiding researchers on when to use true-positive or false-positive metrics in offline evaluation of recommender systems.
ACM TRANSACTIONS ON INFORMATION SYSTEMS
(2021)
Article
Computer Science, Artificial Intelligence
Joao Vinagre, Alipio Mario Jorge, Conceicao Rocha, Joao Gama
Summary: The article proposes a statistical validation method for online recommendation algorithms evaluation, and experiments show that this method is effective in streaming data environment.
IEEE TRANSACTIONS ON KNOWLEDGE AND DATA ENGINEERING
(2021)
Review
Health Care Sciences & Services
Yue Sun, Jia Zhou, Mengmeng Ji, Lusi Pei, Zhiwen Wang
Summary: This study aimed to identify and evaluate the development of Health Recommender Systems (HRSs) and create an evidence map. A total of 51 studies were included for data extraction. The findings showed that only 19.6% of the systems considered the personal preferences of end users in the design stage. The evaluation methods varied, with 62.7% of the studies using offline evaluations and 33.3% including end users in the evaluation process. More user-centered evaluation studies are needed in the future.
JOURNAL OF MEDICAL INTERNET RESEARCH
(2023)
Article
Computer Science, Information Systems
Junha Park, Yeon-Chang Lee, Sang-Wook Kim
Summary: This paper addresses the issue of negative sampling strategy in metric learning-based recommendation methods, introducing the cage-based NS strategy (CNS) and its improved version CNS+. Through experiments, it is demonstrated that both strategies significantly improve accuracy and reduce computation overhead, with linear scalability as the number of ratings increases. Applying CNS strategy to existing ML recommendation methods consistently enhances their accuracy, while CNS+ significantly reduces execution times without sacrificing accuracy.
INFORMATION SCIENCES
(2022)
Article
Computer Science, Artificial Intelligence
Tome Eftimov, Bibek Paudel, Gorjan Popovski, Dragi Kocev
Summary: Personalized ranking systems, also known as recommender systems, employ various big data methods, but existing performance measures do not effectively assist end-users in selecting suitable algorithms. To address this issue, we introduce a novel benchmarking framework that combines different evaluation measures to rank recommender systems on individual benchmark datasets.
Article
Computer Science, Artificial Intelligence
Edgar Ceh-Varela, Huiping Cao, Hady W. Lauw
Summary: Recommender systems play an important role in decision-making processes, and there is a need for recommendations for groups in various real-world activities. However, evaluating the performance of Group Recommender Systems (GRecSys) proves challenging due to different types of groups and items. This study experimentally compares eight representative GRecSys and finds that those using Singular Value Decomposition or Neural Collaborative Filtering methods perform better, with the Average aggregation function yielding better results.
ACM TRANSACTIONS ON INTELLIGENT SYSTEMS AND TECHNOLOGY
(2022)
Article
Computer Science, Artificial Intelligence
Yuanbo Xu, Yongjian Yang, En Wang, Fuzhen Zhuang, Hui Xiong
Summary: This paper addresses the issue of professional malicious users in e-commerce and proposes an unsupervised multi-modal learning model to detect these users. The model considers both ratings and reviews, and utilizes metric learning and attention mechanism to improve performance. Experimental results demonstrate its effectiveness, and it also enhances the performance of recommender models.
IEEE TRANSACTIONS ON KNOWLEDGE AND DATA ENGINEERING
(2022)
Review
Computer Science, Information Systems
Fethi Fkih
Summary: This paper provides an in-depth review of similarity measures used in collaborative filtering-based recommender systems. Through experimental studies, the performance of different measures is compared, and important conclusions are drawn. Evaluation results show that different similarity measures have different suitability in user-based and item-based recommendations.
JOURNAL OF KING SAUD UNIVERSITY-COMPUTER AND INFORMATION SCIENCES
(2022)
Article
Computer Science, Artificial Intelligence
Yuanbo Xu, En Wang, Yongjian Yang, Yi Chang
Summary: The study introduces a novel supervised collaborative representation learning model called Magnetic Metric Learning (MML), which leverages dual triplets to handle both observed and underlying relationships between users and items, addressing some of the drawbacks in traditional recommender system approaches.
IEEE TRANSACTIONS ON KNOWLEDGE AND DATA ENGINEERING
(2022)
Review
Computer Science, Hardware & Architecture
Reza Jafari Ziarani, Reza Ravanmehr
Summary: A recommender system is used to accurately recommend items to attract users, but focusing too much on accuracy can lead to boring and predictable recommendations. Novelty and diversity are helpful, but serendipity, unexpectedness, and relevance are also important criteria for appealing and useful recommendations. Recent studies have shown progress in the quality and quantity of articles on serendipity-oriented recommender systems.
JOURNAL OF COMPUTER SCIENCE AND TECHNOLOGY
(2021)
Article
Information Science & Library Science
Pan Li, Alexander Tuzhilin
Summary: This paper proposes a variety-seeking framework to measure the level of variety-seeking behavior of customers in recommendations based on their consumption records. The effectiveness of the framework is validated through user questionnaire studies conducted at Alibaba. Furthermore, a recommendation framework that combines the identified variety-seeking levels with unexpected recommender systems is presented. Experimental results demonstrate the effectiveness and economic impact of the recommendation framework, providing important managerial implications.
INFORMATION SYSTEMS RESEARCH
(2023)
Article
Computer Science, Theory & Methods
Bushra Alhijawi, Arafat Awajan, Salam Fraihat
Summary: This article provides a comprehensive review of recent research efforts on recommender systems. It focuses on objectives beyond accuracy and relevance, such as diversity, novelty, coverage, and serendipity. The article also explores the definitions and measures associated with these objectives, as well as the evaluation methodology and new applications in the field.
ACM COMPUTING SURVEYS
(2023)
Article
Mathematics
Zeshan Aslam Khan, Naveed Ishtiaq Chaudhary, Waqar Ali Abbasi, Sai Ho Ling, Muhammad Asif Zahoor Raja
Summary: A recommender system aims to gain users' confidence and reduce their time and effort. In this study, an improved, confidence-integrated denoising auto-encoder (DAE) is proposed to enhance the performance of recommender systems. The proposed model achieves improved scores in various evaluation metrics and proves to be efficient and accurate in generating recommendations.
Article
Computer Science, Information Systems
Alistair Moffat, Matthias Petri
ACM TRANSACTIONS ON INFORMATION SYSTEMS
(2020)
Editorial Material
Computer Science, Information Systems
Joemon M. Jose, Emine Yilmaz, Joao Magalhaes, Pablo Castells
INFORMATION RETRIEVAL JOURNAL
(2021)
Article
Computer Science, Information Systems
Elisa Mena-Maldonado, Rocio Canamares, Pablo Castells, Yongli Ren, Mark Sanderson
Summary: The study reveals that false-positive metrics tend to penalize popular items, opposite to true-positive metrics, resulting in an inconsistent trend between the two types of metrics under popularity biases. Through theoretical analysis, the reasons for the disagreement between metrics are identified, as well as rare situations where they might agree. Empirical study results confirm the analytical findings, guiding researchers on when to use true-positive or false-positive metrics in offline evaluation of recommender systems.
ACM TRANSACTIONS ON INFORMATION SYSTEMS
(2021)
Article
Computer Science, Information Systems
Alfan Farizki Wicaksono, Alistair Moffat
Summary: This paper introduces a session-based offline evaluation framework for measuring the overall usefulness of search sessions. By modeling data from two commercial search engines, the user conditional continuation probability and user conditional reformulation probability are proposed to develop new metrics that show greater correlation with observed user behavior during search sessions.
INFORMATION PROCESSING & MANAGEMENT
(2021)
Article
Computer Science, Information Systems
Joel Mackenzie, Matthias Petri, Alistair Moffat
Summary: Inverted indexes are crucial for efficient querying of large document collections in text search engines. Document-ordered indexes are common and allow for various query types, but they may scatter high-scoring documents. Impact-ordered indexes, on the other hand, support anytime query processing and improve search quality gradually.
ACM TRANSACTIONS ON INFORMATION SYSTEMS
(2022)
Article
Computer Science, Information Systems
Hanan Alkhammash, Artem Polyvyanyy, Alistair Moffat, Luciano Garcia-Banuelos
Summary: Access to large volumes of data is crucial for developing precise models in various fields of computing. This article introduces the entropic relevance measure for conformance checking of stochastic process models, which provides information about the likelihood and patterns of process events.
INFORMATION SYSTEMS
(2022)
Article
Computer Science, Artificial Intelligence
Pablo Castells, Alistair Moffat
Summary: This article reviews and reflects on the development and current status of recommender system evaluation, with a focus on offline evaluation. It discusses the challenges posed by adaptation and specific needs, as well as important choices in experiment configuration and broader perspectives of evaluation.
Article
Computer Science, Information Systems
Alistair Moffat, Joel Mackenzie
Summary: This paper introduces an index structure and processing regime that allows immediate access and efficient querying in a dynamic retrieval system. A new compression operation and extensible lists approach are described to achieve incremental document-level indexing and fast document insertion. The mechanism supports various types of queries and facilitates conversion to a compressed inverted index structure.
INFORMATION PROCESSING & MANAGEMENT
(2023)
Proceedings Paper
Computer Science, Information Systems
Artem Polyvyanyy, Alistair Moffat, Luciano Garcia-Banuelos
Summary: This paper proposes a bootstrap-based estimator for process mining generalization, which reduces errors when the quality of the log improves and supports industry-scale data-driven systems engineering.
ADVANCED INFORMATION SYSTEMS ENGINEERING (CAISE 2022)
(2022)
Proceedings Paper
Computer Science, Information Systems
Chenglong Ma, Yongli Ren, Pablo Castells, Mark Sanderson
Summary: This article discusses the problem of concept drift in stream data, particularly the temporal dynamics of user behavior observed in recommender systems. The authors find that irrational behavior by users can impair the knowledge learned by recommender algorithms and exacerbate preference biases. However, existing research often focuses on individual concept drift and overlooks the synergistic effect among users in the same social group. The authors conduct a study on user behavior to detect collaborative concept drift among users and empirically show that increasing individual experience can weaken herd effects.
PROCEEDINGS OF THE 45TH INTERNATIONAL ACM SIGIR CONFERENCE ON RESEARCH AND DEVELOPMENT IN INFORMATION RETRIEVAL (SIGIR '22)
(2022)
Proceedings Paper
Computer Science, Information Systems
Javier Sanz-Cruzado, Pablo Castells
Summary: Link recommendation is a significant problem at the crossroad of recommender systems and online social networks. We introduce RELISON, an extensible framework for conducting link recommendation experiments, which includes various algorithms and evaluation tools.
PROCEEDINGS OF THE 45TH INTERNATIONAL ACM SIGIR CONFERENCE ON RESEARCH AND DEVELOPMENT IN INFORMATION RETRIEVAL (SIGIR '22)
(2022)
Proceedings Paper
Computer Science, Information Systems
Xinyi Yan, Chengxi Luo, Charles L. A. Clarke, Nick Craswell, Ellen M. Voorhees, Pablo Castells
Summary: The dramatic improvements in core information retrieval tasks engendered by neural rankers create a need for novel evaluation methods. Several recent papers explore pairwise preference judgments as an alternative to traditional graded relevance assessments. This method allows for finegrained distinctions by comparing items side-by-side.
PROCEEDINGS OF THE 45TH INTERNATIONAL ACM SIGIR CONFERENCE ON RESEARCH AND DEVELOPMENT IN INFORMATION RETRIEVAL (SIGIR '22)
(2022)
Proceedings Paper
Computer Science, Artificial Intelligence
Michael D. Ekstrand, Allison Chaney, Pablo Castells, Robin Burke, David Rohde, Manel Slokom
Summary: There is a growing interest in using synthetic data and simulation infrastructures for recommender systems research, but there are currently no clear best practices in this area. A workshop was proposed to discuss the state of the art in this research and address methodological questions, resulting in a report documenting currently-known best practices and outlining an agenda for further research.
15TH ACM CONFERENCE ON RECOMMENDER SYSTEMS (RECSYS 2021)
(2021)