Github
文档
论文Awesome LLMs Evaluation Papers :bookmark_tabs:
The papers are organized according to our survey:
Evaluating Large Language Models: A Comprehensive Survey
Zishan Guo*, Renren Jin*, Chuang Liu*, Yufei Huang, Dan Shi, Supryadi,
Linhao Yu, Yan Liu, Jiaxuan Li, Bojian Xiong, Deyi Xiong†
Tianjin University
(*: Co-first authors, †: Corresponding author)
If you find our survey useful, please kindly cite our paper:
@article{guo2023evaluating,
title={Evaluating Large Language Models: A Comprehensive Survey},
author={Guo, Zishan and Jin, Renren and Liu, Chuang and Huang, Yufei and Shi, Dan and Yu, Linhao and Liu, Yan and Li, Jiaxuan and Xiong, Bojian and Xiong, Deyi and others},
journal={arXiv preprint arXiv:2310.19736},
year={2023}
}
Contributing to this paper list
Feel free to open an issue/PR or e-mail guozishan@tju.edu.cn, rrjin@tju.edu.cn, liuc_09@tju.edu.cn and dyxiong@tju.edu.cn if you find any missing areas, papers, or datasets. We will keep updating this list and survey.
Updates
- [2023-10-30] Initial Paperlist for LLMs Evaluation from Zishan Guo, Renren Jin, Chuang Liu, Yufei Huang, Dan Shi, Supryadi, Linhao Yu, Jiaxuan Li, Bojian Xiong and Deyi Xiong.
Survey Introduction
Large language models (LLMs) have demonstrated remarkable capabilities across a broad spectrum of tasks. They have attracted significant attention and been deployed in numerous downstream applications. Nevertheless, akin to a double-edged sword, LLMs also present potential risks. They could suffer from private data leaks or yield inappropriate, harmful, or misleading content. Additionally, the rapid progress of LLMs raises concerns about the potential emergence of superintelligent systems without adequate safeguards. To effectively capitalize on LLM capacities as well as ensure their safe and beneficial development, it is critical to conduct a rigorous and comprehensive evaluation of LLMs.
This survey endeavors to offer a panoramic perspective on the evaluation of LLMs. We categorize the evaluation of LLMs into three major groups: knowledge and capability evaluation, alignment evaluation and safety evaluation. In addition to the comprehensive review on the evaluation methodologies and benchmarks on these three aspects, we collate a compendium of evaluations pertaining to LLMs' performance in specialized domains, and discuss the construction of comprehensive evaluation platforms that covers LLM evaluations on capabilities, alignment, safety, sand applicability.
We hope that this comprehensive overview will stimulate further research interests in the evaluation of LLMs, with the ultimate goal of making evaluation serve as a cornerstone in guiding the responsible development of LLMs. We envision that this will channel their evolution into a direction that maximizes societal benefit while minimizing potential risks.
Markups
The paper proposes a dataset that can be used for LLMs evaluation.
The paper proposes an evaluation method that can be used for LLMs.
The paper proposes a platform for LLMs evaluation.
The paper examines the performance of LLMs in a particular domain.
Table of Contents
Related Surveys for LLMs Evaluation
-
"Through the Lens of Core Competency: Survey on Evaluation of Large Language Models".
-
"A Survey on Evaluation of Large Language Models".
Yupeng Chang and Xu Wang et al. arXiv 2023. [Paper] [GitHub]
Papers
:books:Knowledge and Capability Evaluation
Question Answering
-
Squad: "Squad: 100, 000+ questions for machine comprehension of text".
-
NarrativeQA: "The narrativeqa reading comprehension challenge".
-
Hotpotqa: "Hotpotqa: A dataset for diverse, explainable multi-hop question answering".
-
CoQA: "Coqa: A conversational question answering challenge".
-
NQ: "Natural questions: a benchmark for question answering research".
-
DuReader: "Dureader_robust: A chinese dataset towards evaluating robustness and generalization of machine reading comprehension in real-world applications".
-
RAGAS: "RAGAS: Automated Evaluation of Retrieval Augmented Generation".
-
"Why Does ChatGPT Fall Short in Providing Truthful Answers?".
Shen Zheng and Jie Huang et al. arXiv 2023. [Paper]
Knowledge Completion
-
LAMA: "Language Models as Knowledge Bases?".
-
Kola: "Kola: Carefully Benchmarking World Knowledge of Large Language models".
-
WikiFact: "Assessing the Factual Accuracy of Generated Text".
Ben Goodrich et al. KDD 2019. [Paper]
Reasoning
Commonsense Reasoning
-
ARC: "Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge".
-
QASC: "QASC: A Dataset for Question Answering via Sentence Composition".
-
MCTACO: ""Going on a vacation" takes longer than "Going for a walk": A Study of Temporal Commonsense Understanding".
-
TRACIE: "Temporal Reasoning on Implicit Events from Distant Supervision".
-
TIMEDIAL: "TIMEDIAL: Temporal Commonsense Reasoning in Dialog".
-
HellaSWAG: "HellaSwag: Can a Machine Really Finish Your Sentence?".
-
PIQA: "PIQA: Reasoning about Physical Commonsense in Natural Language".
-
Pep-3k: "Modeling Semantic Plausibility by Injecting World Knowledge".
-
Social IQA: "Social IQa: Commonsense Reasoning about Social Interactions".
Maarten Sap and Hannah Rashkin et al. EMNLP 2019. [Paper] [Source]
-
CommonsenseQA: "CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledge".
Alon Talmor and Jonathan Herzig et al. NAACL 2019. [Paper] [GitHub]
-
OpenBookQA: "Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering".
-
"A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivity".
-
"ChatGPT is a Knowledgeable but Inexperienced Solver: An Investigation of Commonsense Problem in Large Language Models".
Ning Bian et al. arXiv 2023. [Paper]
Logical Reasoning
-
SNLI: "A large annotated corpus for learning natural language inference".
Samuel R. Bowman et al. EMNLP 2015. [Paper]
-
MultiNLI: "A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference".
-
LogicNLI: "Diagnosing the First-Order Logical Reasoning Ability Through LogicNLI".
Jidong Tian and Yitian Li et al. EMNLP 2021.
