Datasets for Large Language Models: A Comprehensive Survey (AI summary)

Key Points

In this paper, a survey and analysis of Large Language Model (LLMs) datasets is presented with a focus on the foundational infrastructure that sustains the development of LLMs. The paper consolidates and categorizes the fundamental aspects of LLM datasets from five perspectives: Pre-training Corpora, Instruction Fine-tuning Datasets, Preference Datasets, Evaluation Datasets, and Traditional NLP Datasets.

The paper provides a comprehensive review of the existing available dataset resources, covering statistics from 444 datasets across 8 language categories and 32 domains, incorporating information from 20 dimensions. The dataset statistics surpass 774.5 TB for pre-training corpora and over 700M instances for other datasets. The paper also focuses on the construction and analysis of LLM datasets, detailing the historical development of text datasets, challenges posed by the current explosion in LLM datasets, and data preprocessing techniques.

Additionally, the paper outlines different construction methods for datasets, including manual creation, model generation, collection and improvement of existing datasets, and a combination of these methods. It also covers detailed information on the general instruction fine-tuning datasets, which are categorized into four main types based on their construction methods: Human Generated Datasets, Model Constructed Datasets, Collection and Improvement of Existing Datasets, and Datasets Created with Multiple Methods. The paper presents different approaches to instruction construction and provides detailed summaries of various general instruction fine-tuning datasets.

The research paper delves into extensive details about various datasets used for the training and evaluation of large language models (LLMs) such as GPT-3.5 and GPT-4. It categorizes the datasets into three types: instruction fine-tuning datasets, preference datasets, and evaluation datasets. The paper also features key evaluation categories, such as general, exam, subject, natural language understanding, reasoning, knowledge, long text, tool, agent, code, OOD, law, medical, financial, social norms, factuality, evaluation, multitask, multilingual, and other. The datasets mentioned assess the performance of LLMs across various tasks, including general instructions, exams, specialized LLMs, and specific domains such as medical, legal, financial, and social norms.

These datasets serve as crucial tools for training and evaluating LLMs in different contexts, including multi-turn interactions, sentiment analysis, and programming tasks. The evaluation datasets are largely publicly available under different licenses and are primarily constructed using human-generated, model-constructed, or collection and improvement approaches. Overall, the datasets play a vital role in improving the performance and versatility of LLMs in understanding, decision-making, and cognitive reasoning.

The research paper discusses the evaluation datasets within the examination domain, crafted for formulating instructions derived from significant exam questions across diverse nations. The paper provides detailed information about various evaluation datasets curated for assessing the proficiency of LLMs (Language Model Models) in comprehending the nuances of question intent and their reservoir of knowledge pertaining to examinations. The datasets are designed for diverse fields including disciplines like mathematics, law, psychology, and more. The datasets are further categorized into specific tasks such as question answering, text simplification, code generation, sentiment analysis, and more. Additionally, the paper provides a distribution statistics of evaluation datasets based on release time, license, size, construction method, language, domain, question type, and evaluation method. The comprehensive summary provides insights into the myriad of evaluation datasets and their varied applications in the field of natural language processing.

The research paper provides a comprehensive analysis of datasets associated with Large Language Models (LLMs) across different dimensions such as pre-training corpora, fine-tuning instruction datasets, preference datasets, evaluation datasets, and traditional NLP datasets. The paper identifies various challenges and potential future directions for dataset development in the areas of pre-training, fine-tuning instruction, reinforcement learning, and model evaluation.

Key Points:

1. Pre-training Corpora: The paper discusses numerous pre-training corpora such as CC-Stories, CC100, CLUECorpus2020, Common Crawl, CulturaX, C4, mC4, OSCAR, RealNews, RedPajama-V2, RefinedWeb, WuDaoCorpora-Text, ANC, BNC, News-crawl, Anna’s Archive, BookCorpusOpen, PG-19, Project Gutenberg, Smashwords, Toronto Book Corpus, arXiv, and S2ORC, among others. These corpora are derived from sources such as web archives, books, academic papers, programming languages, social media platforms, and more.

2. Fine-tuning Instruction Datasets: The paper outlines various instruction fine-tuning datasets including Aya Dataset, databricks-dolly-15K, InstructionWild v2, LCCC, OASST1, OL-CC, Zhihu-KOL, Alpaca data, BELLE train datasets, ChatGPT corpus, Unnatural Instructions, MoodleDlg, and QuestDL, as well as datasets focused on specific domains such as finance, medicine, language, and artificial intelligence.

3. Preference Datasets: The paper delves into the limited availability of resources and the evaluation method settings in preference datasets. It addresses challenges related to scarce open-source preference datasets, language limitations, and the need for comprehensive evaluation methods.

4. Evaluation Datasets: The paper discusses the establishment of evaluation datasets, addressing evaluation gaps, addressing evaluation approaches, and a comprehensive evaluation framework to simplify model invocation, unify the selection of datasets, and efficiently evaluate models.

The research paper provides an extensive overview of datasets associated with Large Language Models across various dimensions, presenting challenges and opportunities for future advancements in these areas.
The research paper covers a wide range of datasets used for instruction fine-tuning and evaluation of language models. It includes datasets such as OpenOrca, which comprises 1M instructions generated by GPT-4 and 3.2M by GPT-3.5-Turbo, and datasets like ChatDoctor and ChatMed Consult, which address the limitations of existing language models in the medical field. The paper also mentions datasets like Code Alpaca 20K, which is designed for fine-tuning the Code Alpaca model, and DISC-Law-SFT, a Chinese legal instruction dataset. Additionally, the paper discusses datasets for evaluating LLMs' performance in various domains, including SuperGLUE for NLU, Chain-of-Thought Hub for intricate reasoning tasks, and InfiniteBench for assessing texts beyond 100K, among others. These datasets are utilized to evaluate the models' abilities in cross-lingual, multitask, and comprehensive contextual comprehension, among others.

The paper discusses the creation of several datasets for evaluating the capabilities of Language Model Models (LLMs) across various natural language processing (NLP) tasks. The datasets cover a wide range of tasks including Chinese language capabilities evaluation, Q&A, translation, text summarization, sentiment analysis, common sense reasoning, and more. The paper introduces multiple datasets such as CLiB104 for Chinese language capabilities evaluation, decaNLP for English task processing proficiency evaluation, FlagEval for nuanced evaluation framework, HELM for evaluating LLMs' capabilities, LLMEVAL-1 for evaluating LLMs' capabilities in Chinese, LMentry for assessing LLMs' performance on simple tasks, XNLI for multilingual sentence classification evaluation, XTREME for assessing LLMs through four NLP tasks in various languages, and many others.

These datasets provide a comprehensive resource for evaluating the performance of LLMs on diverse NLP tasks in various languages and domains. The paper outlines the details and objectives of each dataset, making them valuable resources for NLP research and evaluation.

1. The paper discusses various datasets that focus on named entity recognition (NER) and relation extraction (RE) in different domains, such as entertainment, dialogue, scientific literature, and medical annotations for COVID-19-related social media texts.

2. Each dataset offers a unique perspective on entity recognition and relation extraction, with entity categories ranging from geopolitical entities, geographical locations, institutional names, and personal names, to more specific categories like corporations, creative works, groups, locations, persons, products, medical entities (disease, drug, symptom, vaccine), and Q&A explanations.

3. The datasets challenge models to detect and categorize emerging named entities amidst noisy textual data, engage with multiple sentences within a document for entity recognition and relationship inference at the document level, integrate few-shot learning with relation extraction, and facilitate tasks in NER, sentiment analysis, and Q&A scenarios.

Summary

The paper explores Large Language Model (LLM) datasets and their crucial role in advancing LLMs. It addresses the lack of comprehensive overview and thorough analysis of LLM datasets by categorizing them from five perspectives: Pre-training Corpora, Instruction Fine-tuning Datasets, Preference Datasets, Evaluation Datasets, and Traditional Natural Language Processing Datasets. The paper provides a comprehensive review of existing dataset resources, covering 444 datasets, 8 language categories, and 32 domains, with data sizes exceeding 774.5 TB for pre-training corpora and over 700M instances for other datasets. The paper categorizes the fundamental aspects of LLM datasets and discusses the challenges and future research directions. It also presents insights into the growth and development of LLM datasets and provides specific examples of data types and construction methods in the instruction fine-tuning datasets. The paper concludes by emphasizing the importance of data preprocessing and review for data quality improvement.
The paper provides an extensive overview and categorization of large language model (LLM) datasets, focusing on their critical role in the development of LLMs. The paper specifically categorizes the fundamental aspects of LLM datasets from the perspectives of pre-training corpora and instruction fine-tuning datasets. The main points of the paper include the lack of comprehensive oversight and thorough analysis of LLM datasets, as well as the importance of evaluating LLM performance on evaluation datasets.

The paper categorizes 112 datasets into 20 evaluation domains, covering various evaluation aspects such as general capabilities, subject-specific knowledge, natural language understanding, reasoning, knowledge, and safety evaluation, among others. Evaluation datasets assess LLM performance on a wide range of tasks, including understanding, generating responses, reasoning, and knowledge utilization. The paper also covers human-centric standardized exams, language proficiency, multimodal interaction, code understanding and generation, reasoning problems, and language capabilities.

In addition to the assessment of LLM capabilities, the paper highlights the increasing importance of preference datasets, which reflect the relative preferences of humans or models for different responses within a given task or context. The paper provides detailed insights into the different preference evaluation methods utilized in these datasets, including human voting, sorting, scoring, and other alternative methods. These preference datasets are instrumental in aligning LLMs with human expectations across various tasks and comprehensive safety considerations.

Overall, the paper emphasizes the significance of evaluation datasets and preference datasets in assessing LLM performance and guiding the development and optimization of LLMs. It provides a comprehensive overview of the diverse evaluation domains and the role of preference datasets in aligning LLMs with human preferences.
The research paper explores Large Language Models (LLMs) and their critical role in natural language understanding. It addresses the lack of comprehensive overview and thorough analysis of LLM datasets. The paper categorizes the fundamental aspects of LLM datasets from the perspectives of pre-training corpora and instruction fine-tuning datasets.

The paper extensively covers the categorization and evaluation of LLM datasets for various natural language tasks, including medical language understanding, problem-solving abilities, complex tasks, world knowledge utilization, legal and mathematical reasoning, multidisciplinary abilities, and open-ended question answering. It also delves into the evaluation datasets within the examination domain, derived from significant exam questions across diverse nations. The paper discusses the creation of benchmarks centered on human-centric tests, featuring a selection of official, public, and stringent entrance and qualification examinations.

The paper provides a thorough examination of LLM datasets in the academic domain, including datasets for subjects like mathematics, law, psychology, and more. It also covers datasets evaluating specific disciplines on a smaller scale and those comprehensively assessing disciplinary capabilities encompassing a wide range of subjects. The paper emphasizes the multifaceted abilities of LLMs in natural language understanding (NLU) tasks, covering fundamental comprehension of grammatical structures to advanced semantic reasoning and context handling.

Additionally, the paper covers datasets designed to gauge LLMs' proficiency in logical reasoning and inference, including multi-step reasoning, decision reasoning, deductive reasoning, and mathematical reasoning. It also explores datasets evaluating LLMs' capabilities in handling coding-related tasks, specifically code interpretation, code generation, code correction, and code optimization.

The paper addresses the evaluation datasets designed to gauge the capabilities of pre-trained base models after fine-tuning with instructions from previously unseen tasks. It also delves into legal evaluation datasets, medical evaluation datasets, financial evaluation datasets, societal norms assessment datasets, and data on evaluating the factual accuracy of LLMs.

In summary, the paper comprehensively evaluates the various types of traditional NLP datasets and their roles and applications in the context of LLMs, offering a comprehensive overview of the current landscape of LLM dataset evaluation.
The paper provides an extensive exploration of Large Language Model (LLM) datasets, categorizing them across five dimensions: pre-training corpora, fine-tuning instruction datasets, preference datasets, evaluation datasets, and traditional Natural Language Processing (NLP) datasets. The paper highlights numerous specific datasets within each dimension, presenting details of their scale, source, and application. It also discusses the challenges faced by these datasets and outlines future development directions.

The paper presents a categorization of LLM datasets by examining pre-training corpora and fine-tuning instruction datasets, including examples such as Common Crawl, Project Gutenberg, and WMT65. It discusses the critical role these datasets play in the development of LLMs, emphasizing the lack of a comprehensive overview and thorough analysis of these datasets. Furthermore, the paper introduces specific tasks, such as sentiment analysis, semantic matching, text generation, text translation, text summarization, and text classification. It provides an overview of classic sentiment analysis datasets like IMDB, Sentiment140, SST-2, and EPRSTMT, as well as datasets for semantic matching such as MRPC, QQP, PAWS, AFQMC, and LCQMC. Additionally, the paper outlines datasets for text generation, text translation, and text summarization and presents key datasets for text classification and text quality evaluation tasks. The paper also discusses the challenges and potential directions for future dataset development in four key areas: pre-training corpora, fine-tuning instruction datasets, preference datasets, and evaluation datasets.

The paper provides detailed insights into various LLM datasets, including their source, scale, and application in different NLP tasks. It emphasizes the importance of these datasets in the development and training of LLMs and highlights the challenges and areas for improvement in the current landscape of LLM datasets. The paper concludes with a hope that this survey will serve as a valuable reference for researchers in academia and industry, as well as practitioners engaged with LLMs. Overall, the paper offers a comprehensive overview of LLM datasets, highlighting their crucial role in the development of LLMs and providing insights into their current challenges and future directions for improvement.
The paper discusses the critical role of Large Language Model (LLM) datasets in the development of LLMs and identifies the lack of a comprehensive overview and thorough analysis of these datasets. The paper provides an extensive exploration of various LLM datasets, categorizing them into different domains and subdomains, including medical, legal, financial, social, safety, and more. Each dataset is described in terms of its purpose, content, size, and evaluation methodology. For example, in the medical domain, datasets such as Baize, MedDialog, and Huatuo-26M are used for training models for medical dialogues and question answering. In the legal domain, datasets like DISC-Law-SFT, HanFei 1.0, and LawGPT zh are used for legal comprehension and reasoning tasks. The financial domain includes datasets like BBF-CFLEB, FinanceIQ, and FinEval, focusing on evaluating financial knowledge and reasoning abilities.

The paper also discusses the evaluation of LLMs in different areas, such as language comprehension, code generation, safety, bias, and factual accuracy. Datasets like LAiW, L-Eval, AgentBench, and FACTOR are used to evaluate LLMs' proficiency in handling extensive text, various language tasks, AI agent capabilities, and factual accuracy, respectively. The evaluation datasets serve as benchmarks to assess LLMs' performance across a wide range of tasks and domains. The paper provides detailed insights into the categorization and evaluation of LLM datasets, encompassing various domains and subdomains, to facilitate comprehensive analysis and understanding of their role in the development and evaluation of LLMs.

The paper extensively explores Large Language Model (LLM) datasets, highlighting their critical role in the development of LLMs. The study categorizes the datasets from various perspectives and domains, such as evaluating factual accuracy, dynamic QA benchmarking, testing hallucinatory behaviors, assessing authenticity, fairness, evaluating language capabilities, assessing reading comprehension, checking cross-lingual sentence classification, and scrutinizing various NLP tasks. It examines the datasets from multiple domains, including text summarization, mathematical capability, sentiment analysis, text translation, news article topic classification, text summarization, and semantic matching. The datasets range from traditional NLP datasets like SQuAD, XNLI, and WMT, to more novel datasets like SuperGLUE, DuoRC, and WebNLG. The diverse collection of datasets provides ample opportunities for training and evaluating LLMs across a broad spectrum of tasks and domains, ensuring the robustness and reliability of the models for real-world applications. This comprehensive exploration of LLM datasets offers an invaluable resource for the development and evaluation of language models.

The paper discusses the critical role of Large Language Model (LLM) datasets in the development of LLMs, highlighting the lack of comprehensive overviews and thorough analyses of these datasets. It categorizes the fundamental aspects of LLM datasets based on pre-training corpora and instruction fine-tuning datasets. The article provides an extensive overview of various datasets designed for tasks such as Chinese spell checking, grammar correction, text proofreading, and NER (Named Entity Recognition) in both Chinese and English. The highlighted datasets include YACLC, CSpider, DuSQL, MBPP, Spider, CLUENER, CoNLL2003, Few-NERD, MSRA, OntoNotes 5.0, Resume, Taobao NER, Weibo NER, WUNT2017, Youku NER, Dialogue RE, DocRED, FewRel, TACRED, CSL, METS-CoV, QED, each serving various NLP tasks. For instance, the YACLC dataset encompasses Chinese text samples and is utilized for tasks like grammar correction and fluency improvement. Furthermore, the article presents datasets like FewRel, TACRED, CSL, METS-CoV, and QED, each with a specific focus on challenging NLP tasks and applications, such as relation extraction, scientific literature analysis, medical entity recognition in COVID-19 related social media texts, and Q&A scenarios. Overall, the paper provides a comprehensive overview of a wide range of LLM datasets and their applications, shedding light on their significance in advancing language models and natural language processing tasks.

Reference: https://arxiv.org/abs/2402.180...