Home
Generative models are changing the way we seek information online. Large language models (LLMs) such as Chat-GPT represent one successful application of generative models, leveraging vast amounts of texts encoded in their billion-scale parameters. Recommendation systems employing generative models go beyond LLMs, encompassing a broader range of models such as deep generative models (DGMs) trained directly on user-item interactions, multi-modal foundation models, and other non-LLM generative models. These models offer new opportunities in the field of recommender systems by enhancing how user preferences are learned, connecting us with vast amounts of information available on the Internet. They are capable of delivering more personalized and contextually relevant content, generating recommendations without reliance on narrowly defined datasets, and addressing the cold-start issue. Furthermore, these models significantly enhance the level of interactivity users have with recommender systems, boosting conversational capabilities.
However, there is no “free cake”; new advantages bring new challenges and risks that must be addressed when using LLMs and other categories of DGMs. Some of these challenges are new (e.g., hallucination, out-of-inventory recommendations) and some are newly intensified due to the expanded capabilities of these systems (privacy, fairness and biases, security and robustness, manipulation, opacity, accountability, over-reliance on automation). A critical aspect of utilizing these technologies is to develop robust evaluation systems that can effectively assess the performance, fairness, and security of these Gen-RecSys. Proper evaluation is essential to ensure these systems are reliable and trustworthy, especially when dealing with sensitive user data and making impactful recommendations.
This workshop will specifically focus on the Risks, Opportunities, and Evaluation in real-world Recommender System applications, aiming to cover a full spectrum of current challenges and advances. Additionally, the workshop invites discussions on the application of LLMs and Generative Models in specific tasks and areas such as Conversation, Explanation, Bundle Recommendation, among others. The discussions are encouraged as long as the goal pertains to some form of information seeking.
Topics
This workshop will focus on the opportunities and challenges of Recommender Systems in real-world applications, structured around three key pillars: opportunities, risks and challenges, and evaluation and mitigation strategies. Discussions will cover topics such as data preprocessing, model evaluation, fairness, debiasing, cold-start problems, model distillation, and user interaction design. We also welcome stories of Recommender System usage in specific applications, such as E-commerce, Streaming Services, News, Social Media, and Personalized Marketing. The goal is to bring together industry and academia researchers to foster knowledge sharing and create discussions that pave the way to societally beneficial exploitation of this technology.
Opportunities
- Integrating large language models and other (multimodal) pre-trained generative models to enhance recommender algorithms for user modeling.
- Generative recommendation using AI to help create personalized item content, such as advertisements, images, and micro-videos.
- Combining content generation and retrieval for personalized information seeking and presentation.
- Leveraging the advantages of generative models to improve traditional recommender tasks, including collaborative, sequential, cold-start, social, conversational, multimodal, and causal recommendation.
- Applications of generative model-enhanced recommender systems in various sectors, such as finance, streaming platforms, social networks, entertainment, music, e-commerce, education, fashion, and healthcare.
Risks and Challenges
- Identifying and addressing the potential risks and challenges associated with deploying generative models in recommender systems.
- Challenges of hallucination and misinformation risks, particularly with the advent of sophisticated image generation models.
- Examining bias and fairness in models with a special emphasis on mitigating biases related to race, gender, or brands.
- Privacy implications of utilizing extensive data for model training.
- Technical challenges in achieving transparency, explainability, security challenges, accountability and user control in recommendations.
- Ensuring compliance with emerging ethical and legal standards.
Evaluation and Mitigation
- Developing new benchmarks, evaluation metrics, and protocols to assess the efficacy, fairness, and security of generative models in recommender systems.
- Novel strategies for bias detection, measurement, and understanding.
- Red-teaming and ensuring recommendations are transparent, explainable, and safeguard user privacy.
- Evaluating the robustness, trustworthiness, and real-time performance of generative models across different domains and modalities.
- Designing evaluation methodologies to examine the usage of generative models in recommender systems, including human evaluation paradigms and interfaces.
- Novel approaches for (generative model) Alignment and RLHF for recommendation tasks.
Submission Guidelines for the Workshop
We welcome submissions in the form of full papers, short papers, and extended abstracts that address any of the listed topics and related areas. Submissions should clearly articulate the contribution to the field, methodology, results, and implications for the design, implementation, or understanding of LLMs and Generative models in recommender systems. Submissions should follow the CEUR-WS 2 column (https://ceur-ws.org/Vol-XXX/CEURART.zip).
- Full Papers: Full Papers (up to 8 pages, excluding references): Detailed studies, theoretical analyses, or extensive reviews of specific aspects of LLMs and Generative models in recommender systems.
- Short Papers: (up to 4 pages, excluding references): Preliminary findings, innovative concepts, or case studies on the application of LLMs and Generative models in recommender systems.
- Extended Abstracts: (2-3 pages, excluding references): Proposals for discussions, work-in-progress, or initial insights into the application of LLMs and Generative models in recommender systems.
Submissions must be anonymized and adhere to the specified formatting guidelines, which will be provided on the workshop website.
The submission site is EasyChair RecSys 2024 Workshops.
Make sure to select the “The 1st Workshop on Risks, Opportunities, and Evaluation of Generative Models in Recommender Systems (ROEGEN@RECSYS'24)” track when creating a submission.
All papers will undergo a rigorous double-blind peer review process, focusing on relevance, originality, technical quality, relation to the workshop scope, and overall contribution to the field. Acceptance will be based on these criteria. Accepted papers will be presented at the workshop and included in the proceedings, and will be published in CEUR Workshop Proceedings. All papers will undergo the same review process and review period. High-quality submissions will be recommended for a special issue in ACM TORS on using generative models for recommendation.
Important Dates
- Submission Deadline:
September 5, 2024September 7, 2024 Final (AOE) - Notification of Acceptance:
September 13, 2024September 20, 2024 - Camera-Ready Submission:
September 20, 2024September 27, 2024 - Workshop Date: October 18, 2024
Program Schedule
| Time | Event |
|---|---|
| 09:00-09:45 | Invited Talk – Michael Ekstrand (Drexel University): Responsible Recommendation in the Age of Generative AI |
| 09:45-10:30 |
Contributed Paper Talks:
|
| 10:30-11:00 | Coffee Break |
| 11:00-11:45 | Invited Talk – Craig Boutilier (Google Research): Can Generative Models Improve Recommender Systems? Three Useful Notions of Alignment to Help with the Task |
| 11:45-12:30 | Invited Talk – Jianling Wang (Google DeepMind): When LLMs Meet Recommendations: A Scalable Hybrid Approach |
| 12:30-14:15 | Lunch Break |
| 14:15-15:00 | Invited Talk – Aixin Sun (Nanyang Technological University): Understanding and Evaluating Recommender Systems from a User Perspective |
| 15:00-15:45 |
Contributed Paper Talks:
|
| 15:45-16:15 | Coffee Break |
| 16:15-17:00 | Invited Talk – Jiaqi Zhai (Meta): Actions Speak Louder than Words: Building the Next-Generation Recommendation Systems |
| 17:00-17:45 |
Contributed Paper Talks:
|
Featured Speakers
Craig Boutilier
Principal Scientist at Google, USA
Can Generative Models Improve Recommender Systems? Three Useful Notions of Alignment to Help with the Task
Alignment is a broad notion that is often interpreted in various ways in the context of generative models (GenAI). This is due, in large part, to the staggering range of uses to which GenAI can be put. The application of Gen to recommender systems is no different: the variety of ways GenAI can be employed requires diverse perspectives on alignment. In this talk, I outline three uses of GenAI in recommenders that require distinct formulations of alignment: (a) GenAI in multi-modal, interactive recommenders; (b) GenAI for user simulation in recommenders; and (c) GenAI for filling in “content gaps,” i.e., generating novel items to better match the preferences of a user population. I briefly describe some work that tackles these three use cases (to varying degrees), and outline several significant research challenges that must be addressed to better exploit GenAI in the development of recommenders.
Michael Ekstrand
Assistant Professor at Drexel University, USA
Responsible Recommendation in the Age of Generative AI
Generative AI is bringing significant upheaval to how we think about and build recommender systems, both in the underlying recommendation techniques and in the design of surrounding applications and user experiences, with potentially profound impact on the social, ethical, and humanitarian impacts of recommendation. Some concepts developed for responsible recommendation, retrieval, and AI can be applied to generative recommendation, while others are substantially complicated or hindered by the significant changes in the relationships between users, creators, items, and platforms that generative AI brings. In this talk, I will argue that thinking clearly and explicitly about the underlying social, legal, and moral principles that responsible recommendation efforts seek to advance will provide guidance for thinking about how best to apply and revise impact concerns in the face of current and future changes to recommendation paradigms and applications.
Aixin Sun
Associate Professor at Nanyang Technological University, Singapore
Understanding and Evaluating Recommender Systems from a User Perspective
Recommender Systems (RecSys) have garnered significant attention from both industry and academia for decades. This interest has led to the development of various solutions, ranging from classic models to deep learning and, more recently, generative models. Typically, the evaluation of these models relies on predefined RecSys research problems and available offline datasets. However, the user perspective of RecSys is often underemphasized. In this talk, I will share my understanding as a user of several RecSys systems, discussing the RecSys research problem and the expected evaluations. I believe that considering the user perspective can significantly influence our model design and evaluation, especially in the era of RecSys powered by generative models.
Jianling Wang
Senior Research Scientist at Google DeepMind, USA
When LLMs Meet Recommendations: A Scalable Hybrid Approach
While LLMs’ reasoning and generalization capabilities can aid higher level user understanding and longer term planning for recommendations, directly applying them to industrial recommendation systems have been shown challenging. The talk will cover our recent work on a hybrid approach to combine LLMs and classic recommendation models, and study its effectiveness for a challenging recommendation task on user exploration.
Jiaqi Zhai
Distinguished Engineer at Meta, USA
Actions Speak Louder than Words: Building the Next-Generation Recommendation Systems
Recommendation systems enable billions of people to make informed decisions on a daily basis in every single online content and e-commerce platforms. The scale of such systems has increased by close to 10,000x in the last few years. Despite these being the largest software systems on the planet, most Deep Learning Recommendation Models (DLRMs) scale with data but fail to scale with compute.
We discuss alternative approaches vs classical DLRMs in this talk. We show that by reformulating the classical IR tasks, ranking and retrieval, as a generative modeling problem (“Generative Recommenders”), we can overcome expressiveness constraints in traditional sequential formulations. Together with new algorithms tailored for recommendation settings, we can improve offline and online metrics by double digit percentages vs SotA baselines while using significantly fewer features. To enable efficient scaling, we next present new algorithms that speed up training and inference by 10x-1000x, by leveraging inherent data redundancy in recommendations and algorithmically increasing sparsity. The GR formulation and algorithms have improved offline and online metrics by double digit percentages vs SotA baselines on a large internet platform. More importantly, GRs overcome compute scaling bottlenecks in traditional DLRMs, and we observe scaling law in deployed billion-user scale recommendation systems for the first time, up to GPT-3/LLaMa-2 compute scale.
Workshop Organizers
- Yashar Deldjoo, Tenure-Track Assistant Professor, Polytechnic University of Bari, Italy
- Julian McAuley, Professor, UC San Diego, USA
- Scott Sanner, Associate Professor, University of Toronto, Canada
- Pablo Castells, Professor, Autonomous University of Madrid, Spain
- Shuai Zhang, Applied Scientist, Amazon Web Services AI, USA
- Enrico Palumbo, Senior Research Scientist, Spotify
Program Committee
- Aleksandr Petrov, [email protected], University of Glasgow
- Branislav Kveton, [email protected], Amazon
- Chao Zhang, [email protected], University of Science and Technology of China
- Chengkai Huang, [email protected], The University of New South Wales
- Chen Ma, [email protected], City University of Hong Kong
- Claudia Hauff, [email protected], Spotify
- Dietmar Jannach, [email protected], University of Klagenfurt
- Gustavo Penha, [email protected], Spotify
- Hugues Bouchard, [email protected], Spotify
- Martin Mladenov, [email protected], Google
- Michael Ekstrand, [email protected], Drexel University
- Mohammad Aliannejadi, [email protected], University of Amsterdam
- Narges Tabari, [email protected], Amazon
- Paolo Garza, [email protected], Politecnico di Torino
- Reza Shirvany, [email protected], Zalando
- Thong Nguyen, [email protected], Microsoft
- Zhankui He, [email protected], University of California San Diego