Awesome-LLMs-Datasets

Summarize existing representative LLMs text datasets across five dimensions: Pre-training Corpora, Fine-tuning Instruction Datasets, Preference Datasets, Evaluation Datasets, and Traditional NLP Datasets. (Regular updates)
New dataset sections have been added: Multi-modal Large Language Models (MLLMs) Datasets, Retrieval Augmented Generation (RAG) Datasets. (Gradual updates)

Paper

The paper "Datasets for Large Language Models: A Comprehensive Survey" has been released.（2024/2）

Abstract:

This paper embarks on an exploration into the Large Language Model (LLM) datasets, which play a crucial role in the remarkable advancements of LLMs. The datasets serve as the foundational infrastructure analogous to a root system that sustains and nurtures the development of LLMs. Consequently, examination of these datasets emerges as a critical topic in research. In order to address the current lack of a comprehensive overview and thorough analysis of LLM datasets, and to gain insights into their current status and future trends, this survey consolidates and categorizes the fundamental aspects of LLM datasets from five perspectives: (1) Pre-training Corpora; (2) Instruction Fine-tuning Datasets; (3) Preference Datasets; (4) Evaluation Datasets; (5) Traditional Natural Language Processing (NLP) Datasets. The survey sheds light on the prevailing challenges and points out potential avenues for future investigation. Additionally, a comprehensive review of the existing available dataset resources is also provided, including statistics from 444 datasets, covering 8 language categories and spanning 32 domains. Information from 20 dimensions is incorporated into the dataset statistics. The total data size surveyed surpasses 774.5 TB for pre-training corpora and 700M instances for other datasets. We aim to present the entire landscape of LLM text datasets, serving as a comprehensive reference for researchers in this field and contributing to future studies.

Fig 1. The overall architecture of the survey. Zoom in for better view

Dataset Information Module

The following is a summary of the dataset information module.

Corpus/Dataset name
Publisher
Release Time
- “X” indicates unknown month.
Size
Public or Not
- “All” indicates full open source;
- “Partial” indicates partially open source;
- “Not” indicates not open source.
License
Language
- “EN” indicates English;
- “ZH” indicates Chinese;
- “AR” indicates Arabic;
- “ES” indicates Spanish;
- “RU” indicates Russian;
- “DE” indicates German;
- “PL” indicates Programming Language;
- “Multi” indicates Multilingual, and the number in parentheses indicates the number of languages included.
Construction Method
- “HG” indicates Human Generated Corpus/Dataset;
- “MC” indicates Model Constructed Corpus/Dataset;
- “CI” indicates Collection and Improvement of Existing Corpus/Dataset.
Category
Source
Domain
Instruction Category
Preference Evaluation Method
- “VO” indicates Vote;
- “SO” indicates Sort;
- “SC” indicates Score;
- “-H” indicates Conducted by Humans;
- “-M” indicates Conducted by Models.
Question Type
- “SQ” indicates Subjective Questions;
- “OQ” indicates Objective Questions;
- “Multi” indicates Multiple Question Types.
Evaluation Method
- “CE” indicates Code Evaluation;
- “HE” indicates Human Evaluation;
- “ME” indicates Model Evaluation.
Focus
Numbers of Evaluation Categories/Subcategories
Evaluation Category
Number of Entity Categories (NER Task)
Number of Relationship Categories (RE Task)

Changelog

（2024/01/17）Create the Awesome-LLMs-Datasets dataset repository.
（2024/02/02）Revise information for some datasets; add Dolma (Pre-training Corpora | General Pre-training Corpora | Multi-category).
（2024/02/15）Add Aya Collection (Instruction Fine-tuning Datasets | General Instruction Fine-tuning Datasets | HG & CI & MC); Aya Dataset (Instruction Fine-tuning Datasets | General Instruction Fine-tuning Datasets | HG).
（2024/02/22）Add OpenMathInstruct-1 (Instruction Fine-tuning Datasets | Domain-specific Instruction Fine-tuning Datasets | Math); FinBen (Evaluation Datasets | Financial).
（2024/04/05）
- Add new dataset sections: (1) Multi-modal Large Language Models (MLLMs) Datasets; (2) Retrieval Augmented Generation (RAG) Datasets.
- Add MMRS-1M (MLLMs Datasets | Instruction Fine-tuning Datasets); VideoChat2-IT (MLLMs Datasets | Instruction Fine-tuning Datasets); InstructDoc (MLLMs Datasets | Instruction Fine-tuning Datasets); ALLaVA-4V Data (MLLMs Datasets | Instruction Fine-tuning Datasets); MVBench (MLLMs Datasets | Evaluation Datasets); OlympiadBench (MLLMs Datasets | Evaluation Datasets); MMMU (MLLMs Datasets | Evaluation Datasets).
- Add CLUE Benchmark Series (Evaluation Datasets | Evaluation Platform); OpenLLM Leaderboard (Evaluation Datasets | Evaluation Platform); OpenCompass (Evaluation Datasets | Evaluation Platform); MTEB Leaderboard (Evaluation Datasets | Evaluation Platform); C-MTEB Leaderboard (Evaluation Datasets | Evaluation Platform).
- Add NAH (Needle-in-a-Haystack) (Evaluation Datasets | Long Text); ToolEyes (Evaluation Datasets | Tool); UHGEval (Evaluation Datasets | Factuality); CLongEval (Evaluation Datasets | Long Text).
- Add MathPile (Pre-training Corpora | Domain-specific Pre-training Corpora | Math); WanJuan-CC (Pre-training Corpora | General Pre-training Corpora | Webpages).
- Add IEPile (Instruction Fine-tuning Datasets | General Instruction Fine-tuning Datasets | CI); InstructIE (Instruction Fine-tuning Datasets | General Instruction Fine-tuning Datasets | HG).
- Add CRUD-RAG (RAG Datasets); WikiEval (RAG Datasets); RGB (RAG Datasets); RAG-Instruct-Benchmark-Tester (RAG Datasets); ARES (RAG Datasets).
（2024/04/06）
- Add GPQA (Evaluation Datasets | Subject); MGSM (Evaluation Datasets | Multilingual); HaluEval-Wild (Evaluation Datasets | Factuality); CMATH (Evaluation Datasets | Subject); FineMath (Evaluation Datasets | Subject); RealTime QA (Evaluation Datasets | Factuality); WYWEB (Evaluation Datasets | Subject); ChineseFactEval (Evaluation Datasets | Factuality); Counting-Stars (Evaluation Datasets | Long Text).
- Add SlimPajama (Pre-training Corpora | General Pre-training Corpora | Multi-category); MassiveText (Pre-training Corpora | General Pre-training Corpora | Multi-category); MADLAD-400 (Pre-training Corpora | General Pre-training Corpora | Webpages); Minerva (Pre-training Corpora | General Pre-training Corpora | Multi-category); CCAligned (Pre-training Corpora | General Pre-training Corpora | Parallel Corpus); WikiMatrix (Pre-training Corpora | General Pre-training Corpora | Parallel Corpus); OpenWebMath (Pre-training Corpora | Domain-specific Pre-training Corpora | Math).
- Add WebQuestions (Traditional NLP Datasets | Question Answering | Knowledge QA).
- Add ALCE (RAG Datasets).
- Add AlphaFin (Instruction Fine-tuning Datasets | Domain-specific Instruction Fine-tuning Datasets | Other); COIG-CQIA (Instruction Fine-tuning Datasets | General Instruction Fine-tuning Datasets | HG & CI).
（2024/06/15）
- Add CLUE (Evaluation Datasets | Medical); CHC-Bench (Evaluation Datasets | General); CIF-Bench (Evaluation Datasets | General); ACLUE (Evaluation Datasets | Subject); LeSC (Evaluation Datasets | NLU); AlignBench (Evaluation Datasets | Multitask); SciKnowEval (Evaluation Datasets | Subject).
- Add MAP-CC (Pre-training Corpora | General Pre-training Corpora | Multi-category); FineWeb (Pre-training Corpora | General Pre-training Corpora | Webpages); CCI 2.0 (Pre-training Corpora | General Pre-training Corpora | Webpages).
- Add WildChat (Instruction Fine-tuning Datasets | MC).
- Add OpenHermesPreferences (Preference Datasets | Sort); huozi_rlhf_data (Preference Datasets | Vote); HelpSteer (Preference Datasets | Score); HelpSteer2 (Preference Datasets | Score).
- Add MMT-Bench (MLLMs Datasets | Evaluation Datasets); mOSCAR (MLLMs Datasets | Pre-training Corpora); MM-NIAH (MLLMs Datasets | Evaluation Datasets).
- Add CRAG (RAG Datasets).
We will release the dataset information in CSV format.