Github
Huggingface
论文awesome-generative-information-retrieval 
Conversational models started to be able to access the web or backup their claims with sources (a.k.a. attribution). These chatbots are thus arguably information retrieval machines, competing against or even substituing traditional search engines. We would like to dedicate a space to these models but also to the more general field of generative information retrieval. We tentatively devide the field in two main topics: Grounded Answer Generation and Generative Document Retrieval. We also include generative recommendation, generative grounded summarization etc.
Pull-requests welcome!
Table of Contents
- Blog Posts
- Datasets
- Tools
- Evaluation
- Workshops and Tutorials
- Epistemology Papers
- Grounded Answer Generation
- Retrieval Augmented Generation (RAG) (external grounding/retrieval at inference time)
- LLM Memory Manipulation (grounded in internal model weights at inference time)
- Re-Ranking
- Fact Uncertainty Estimates
- Constrained Generation
- Data Centric
- Utility Maximization
- Multimodal
- Prompting
- Generate Code
- Query Generation
- Summarization and Document Rewriting
- Table QA
- Generative Document Retrieval
- Generative Recommendation
- Generative Knowledge Graphs
- Live Generative Retrieval
Blog Posts
Deterministic Quoting: Making LLMs Safer for Healthcare
Matt Yeung
Personal Blog – Apr 2024 [link]
Retrieval Augmented Generation Research: 2017-2024
Moritz Mallawitsch
Scaling Knowledge – Feb 2024 [link]
Mastering RAG: How To Architect An Enterprise RAG System
Pratik Bhavsar
Galileo Labs – Jan 2024 [link]
Running Mixtral 8x7 locally with LlamaIndex
LlamaIndex
LlamaIndex Blog – Dec 2023 [link]
Advanced RAG Techniques: an Illustrated Overview
Ivan Ilin
Towards AI – Dec 2023 [link]
Multimodal RAG pipeline with LlamaIndex and Neo4j
Tomaz Bratanic
LlamaIndex Blog – Dec 2023 [link]
Benchmarking RAG on tables
LangChain
LangChain Blog – Dec 2023 [link]
Advanced RAG 01: Small-to-Big Retrieval
Sophia Yang
Towards Data Science – Nov 2023 [link]
Query Transformations
LangChain
LangChain Blog – Oct 2023 [link]
What Makes a Dialog Agent Useful?
Nazneen Rajani, Nathan Lambert, Victor Sanh, Thomas Wolf
Hugging Face Blog – Jan 2023 [link]
Forecasting potential misuses of language models for disinformation campaigns and how to reduce risk
Josh A. Goldstein, Girish Sastry, Micah Musser, Renée DiResta, Matthew Gentzel, Katerina Sedova
OpenAI Blog – Jan 2023 [link]
Datasets
LitSearch: A Retrieval Benchmark for Scientific Literature Search
Anirudh Ajith, Mengzhou Xia, Alexis Chevalier, Tanya Goyal, Danqi Chen, Tianyu Gao
arXiv – Jul 2023 [paper] [data]
BRIGHT: A Realistic and Challenging Benchmark for Reasoning-Intensive Retrieval
Hongjin Su, Howard Yen, Mengzhou Xia, Weijia Shi, Niklas Muennighoff, Han-yu Wang, Haisu Liu, Quan Shi, Zachary S. Siegel, Michael Tang, Ruoxi Sun, Jinsung Yoon, Sercan O. Arik, Danqi Chen, Tao Yu
arXiv – Oct 2023 [paper] [data] [code]
FreshLLMs: Refreshing Large Language Models with Search Engine Augmentation
Tu Vu, Mohit Iyyer, Xuezhi Wang, Noah Constant, Jerry Wei, Jason Wei, Chris Tar, Yun-Hsuan Sung, Denny Zhou, Quoc Le, Thang Luong
arXiv – Oct 2023 [paper] [code]
LegalBench: A Collaboratively Built Benchmark for Measuring Legal Reasoning in Large Language Models
Neel Guha, Julian Nyarko, Daniel E. Ho, Christopher Ré, Adam Chilton, Aditya Narayana, Alex Chohlas-Wood, Austin Peters, Brandon Waldon, Daniel N. Rockmore, Diego Zambrano, Dmitry Talisman, Enam Hoque, Faiz Surani, Frank Fagan, Galit Sarfaty, Gregory M. Dickinson, Haggai Porat, Jason Hegland, Jessica Wu, Joe Nudell, Joel Niklaus, John Nay, Jonathan H. Choi, Kevin Tobia, Margaret Hagan, Megan Ma, Michael Livermore, Nikon Rasumov-Rahe, Nils Holzenberger, Noam Kolt, Peter Henderson, Sean Rehaag, Sharad Goel, Shang Gao, Spencer Williams, Sunny Gandhi, Tom Zur, Varun Iyer, Zehua Li
arXiv – Aug 2023 [paper] [dataset]
OpenAssistant Conversations - Democratizing Large Language Model Alignment
Andreas Köpf, Yannic Kilcher, Dimitri von Rütte, Sotiris Anagnostidis, Zhi-Rui Tam, Keith Stevens, Abdullah Barhoum, Nguyen Minh Duc, Oliver Stanley, Richárd Nagyfi, Shahul ES, Sameer Suri, David Glushkov, Arnav Dantuluri, Andrew Maguire, Christoph Schuhmann, Huu Nguyen, Alexander Mattick
arXiv – April 2023 [paper]
ChatGPT-RetrievalQA
Arian Askari, Mohammad Aliannejadi, Evangelos Kanoulas, Suzan Verberne
Github – Feb 2023 [code]
KAMEL : Knowledge Analysis with Multitoken Entities in Language Models
Jan-Christoph Kalo, Leandra Fichtel
AKBC 22 – [paper]
TruthfulQA: Measuring How Models Mimic Human Falsehoods
Stephanie Lin, Jacob Hilton, Owain Evans
arXiv – Sep 2021 [paper] [code]
Complex Answer Retrieval
Laura Dietz, Manisha Verma, Filip Radlinski, Nick Craswell, Ben Gamari, Jeff Dalton, John Foley
TREC – 2017-2019 [link]
Tools
GraphRAG
Jonathan Larson, Steven Truitt
Microsoft – Feb 2024 [code]
Narrowing the Knowledge Evaluation Gap: Open-Domain Question Answering with Multi-Granularity Answers
Gal Yona, Roee Aharoni, Mor Geva
arXiv – Jan 2024 [paper]
DHS LLM Workshop - Module 6
Sourab Mangrulkar
GitHub – Dec 2023 [code]
PrimeQA: The Prime Repository for State-of-the-Art Multilingual Question Answering Research and Development
Avirup Sil, Jaydeep Sen, Bhavani Iyer, Martin Franz, Kshitij Fadnis, Mihaela Bornea, Sara Rosenthal, Scott McCarley, Rong Zhang, Vishwajeet Kumar, Yulong Li, Md Arafat Sultan, Riyaz Bhat, Radu Florian, Salim Roukos
arXiv – Jan 2023 [paper] [code]
TRL: Transformer Reinforcement Learning
Leandro von Werra, Younes Belkada, Lewis Tunstall, Edward Beeching, Tristan Thrush, Nathan Lambert, Shengyi Huang
GitHub – 2020 [code]
Evaluation
FACTSCORE: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation
Sewon Min, Kalpesh Krishna, Xinxi Lyu, Mike Lewis, Wen-tau Yih, Pang Wei Koh, Mohit Iyyer, Luke Zettlemoyer, Hannaneh Hajishirzi
Pypi – May 2023 [paper] [code]
FACTKB: Generalizable Factuality Evaluation using Language Models Enhanced with Factual Knowledge
Shangbin Feng, Vidhisha Balachandran, Yuyang Bai, Yulia Tsvetkov
arXiv – May 2023 [paper] [code]
Evaluating Verifiability in Generative Search Engines
Nelson F. Liu, Tianyi Zhang, Percy Liang
arXiv – April 2023 [paper] [code]
Workshops and Tutorials
Workshop on Generative AI for Recommender Systems and Personalization
Narges Tabari, Aniket Deshmukh, Wang-Cheng Kang, Rashmi Gangadharaiah, Hamed Zamani, Julian McAuley, George Karypis
KDD 24 – Aug 2024 [link]
Second Workshop on Generative Information Retrieval
Gabriel Bénédict, Ruqing Zhang, Donald Metzler, Andrew Yates, Ziyan Jiang
SIGIR 24 – Jul 2024 [link]
Personalized Generative AI
Zheng Chen, Ziyan Jiang, Fan Yang, Zhankui He, Yupeng Hou, Eunah Cho, Julian McAuley, Aram Galstyan, Xiaohua Hu, Jie Yang
CIKM 23 – Oct 2023 [link]
First Workshop on Recommendation with Generative Models
Wenjie Wang, Yong Liu, Yang Zhang, Weiwen Liu, Fuli Feng, Xiangnan He, Aixin Sun
CIKM 23 – Oct 2023 [link]
First Workshop on Generative Information Retrieval
Gabriel Bénédict, Ruqing Zhang, Donald Metzler
SIGIR 23 – Jul 2023 [link]
Retrieval-based Language Models and Applications
Akari Asai, Sewon Min, Zexuan Zhong, Danqi Chen
ACL 23 – Jul 2023 [link]
Epistemology Papers
Recite, Reconstruct, Recollect: Memorization in LMs as a Multifaceted Phenomenon
USVSN Sai Prashanth, Alvin Deng, Kyle O'Brien, Jyothir S V, Mohammad Aflah Khan, Jaydeep Borkar, Christopher A. Choquette-Choo, Jacob Ray Fuehne, Stella Biderman, Tracy Ke, Katherine Lee, Naomi Saphra
arXiv – Jun 2024 [paper]
ChatGPT is bullshit
Michael Townsen Hicks, James Humphries, Joe Slater
Ethics Inf Technol – Jun 2024 [paper]
Hallucination of Multimodal Large Language Models: A Survey
Zechen Bai, Pichao Wang, Tianjun Xiao, Tong He, Zongbo Han, Zheng Zhang, Mike Zheng Shou
arXiv – Apr 2024 [paper]
From Matching to Generation: A Survey on Generative Information Retrieval
Xiaoxi Li, Jiajie Jin, Yujia Zhou, Yuyao Zhang, Peitian Zhang, Yutao Zhu, and Zhicheng Dou
arXiv – Apr 2024 [paper]
Knowledge Conflicts for LLMs: A Survey
Rongwu Xu, Zehan Qi, Cunxiang Wang, Hongru Wang, Yue Zhang, Wei Xu
arXiv – Mar 2024 [paper]
Report on the 1st Workshop on Generative Information Retrieval (Gen-IR 2023) at SIGIR 2023
Gabriel Bénédict, Ruqing Zhang, Donald Metzler, Andrew Yates, Romain Deffayet, Philipp Hager, Sami Jullien
SIGIR Forum – Dec 2023 [paper]
Report on the 1st Workshop on Task Focused IR in the Era of Generative AI
Chirag Shah, Ryen W. White
SIGIR Forum – Dec 2023 [paper]
Towards Generative Search and Recommendation: A keynote at RecSys 2023
Tat-Seng Chua
SIGIR Forum – Dec 2023 [paper]
Large Search Model: Redefining Search Stack in the Era of LLMs
Liang Wang, Nan Yang, Xiaolong Huang, Linjun Yang, Rangan Majumder, Furu Wei
SIGIR Forum – Dec 2023 [paper]
**Large Language Models
