Scientific Large Language Models (Sci-LLMs)

This repository collects papers on scientific large language models, particularly in the domains of biology and chemistry.

😎 Welcome to recommend missing papers through Adding Issues or Pull Requests.

🔔 News

💥 [2024/07] We have updated our survey paper by incorporating the latest related works. Please refer to the revised version on arXiv.
💥 [2024/01] Our survey paper 'Scientific Large Language Models: A Survey on Biological & Chemical Domains' has been released on arXiv.

Sci-LLMs-Scopes In this survey, we focus on scientific languages (i.e., textual, molecular, protein and genomic languages), as well as their combination (i.e., multimodal language).

🌟 Contents

Scientific Large Language Models (Sci-LLMs)

📖 Textual Scientific Large Language Models (Text-Sci-LLMs)

Medical

2019.04 ClinicalBERT: Modeling Clinical Notes and Predicting Hospital Readmission, arXiv, Code
2022.02 GatorTron: A Large Clinical Language Model to Unlock Patient Information from Unstructured Electronic Health Records, arXiv, Model
2022.12 BioMedLM, .stanford.edu, huggingface
2023.05 A Study of Generative Large Language Model for Medical Research and Healthcare (GatorTronGPT), arXiv, Code
2023.11 MEDITRON-70B: Scaling Medical Pretraining for Large Language Models, arXiv, Code
2024.03 Small Language Models Learn Enhanced Reasoning Skills from Medical Textbooks (Meerkat), arXiv
2023.06 ClinicalGPT: Large Language Models Finetuned with Diverse Medical Data and Comprehensive Evaluation, arXiv
2023.10 Qilin-Med: Multi-stage Knowledge Injection Advanced Medical Large Language Model, arXiv, Code
2023.03 ChatDoctor: A Medical Chat Model Fine-Tuned on a Large Language Model Meta-AI (LLaMA) Using Medical Domain Knowledge, arXiv, Code
2023.04 HuaTuo: Tuning LLaMA Model with Chinese Medical Knowledge, arXiv, Code
2023.05 HuatuoGPT, towards Taming Language Model to Be a Doctor, arXiv, Code
2023.04 Baize: An Open-Source Chat Model with Parameter-Efficient Tuning on Self-Chat Data, arXiv, Code
2023.08 Zhongjing: Enhancing the Chinese Medical Capabilities of Large Language Model through Expert Feedback and Real-world Multi-turn Dialogue, arXiv, Code
2023.04 PMC-LLaMA: Towards Building Open-source Language Models for Medicine, arXiv, Code
2023.09 CPLLM: Clinical Prediction with Large Language Models, arXiv, Code
2023.05 Towards Expert-Level Medical Question Answering with Large Language Models(Med-PaLM 2), Google Research, arXiv
2023.05 Clinical Camel: An Open Expert-Level Medical Language Model with Dialogue-Based Knowledge Encoding, arXiv Code
2023.04 DoctorGLM: Fine-tuning your Chinese Doctor is not a Herculean Task, arXiv, Code
2023.10 BianQue: Balancing the Questioning and Suggestion Ability of Health LLMs with Multi-turn Health Conversations Polished by ChatGPT, arXiv, Code
2024.01 Medical mT5: An Open-Source Multilingual Text-to-Text LLM for The Medical Domain, arXiv
2024.02 Me LLaMA: Foundation Large Language Models for Medical Applications, arXiv, Code
2024.02 BiMediX: Bilingual Medical Mixture of Experts LLM, arXiv, Code, Hugging Face

Biology

2019.04 BioELMo: Probing Biomedical Embeddings from Language Models, arXiv, Code
2019.05 BioBERT: a pre-trained biomedical language representation model for biomedical text mining, arXiv, Code
2019.07 Transfer Learning in Biomedical Natural Language Processing: An Evaluation of BERT and ELMo on Ten Benchmarking Datasets, arXiv, Code
2020.10 BioMegatron: Larger Biomedical Domain Language Model, arXiv, Code
2020.10 Domain-Specific Language Model Pretraining for Biomedical Natural Language Processing, arXiv, Hugging Face
2021.06 BioM-Transformers: Building Large Biomedical Language Models with BERT, ALBERT and ELECTRA, ACL Anthology, Code
2022.03 LinkBERT: Pretraining Language Models with Document Links, arXiv, Code
2023.03 BioGPT: Generative Pre-trained Transformer for Biomedical Text Generation and Mining, arXiv, Code
2023.08 BioMedGPT: Open Multimodal Generative Pre-trained Transformer for BioMedicine, arXiv, Code
2023.09 BioinspiredLLM: Conversational Large Language Model for the Mechanics of Biological and Bio-Inspired Materials, arXiv
2024.02 BioMistral: BioMistral: A Collection of Open-Source Pretrained Large Language Models for Medical Domains, arXiv, Code

Chemistry

2021.06 Automated Chemical Reaction Extraction from Scientific Literature. Journal of Chemical Information and Modeling, Code
2021.09 MatSciBERT: A materials domain language model for text mining and information extraction, npj Computational Materials, Code
2022.09 A general-purpose material property data extraction pipeline from large polymer corpora using natural language processing, npj Computational Materials, Hugging Face
2024.01 ChemDFM: Dialogue Foundation Model for Chemistry, arXiv, Model
2024.02 ChemLLM: A Chemical Large Language Model, arXiv, Model
2024.02 LlaSMol: Advancing Large Language Models for Chemistry with a Large-Scale, Comprehensive, High-Quality Instruction Tuning Dataset, arXiv, Page, Model, Dataset
2024.02 PharmaGPT: Domain-Specific Large Language Models for Bio-Pharmaceutical and Chemistry, arXiv

Comprehensive

2019.09 SciBERT: A Pretrained Language Model for Scientific Text, arXiv, Code
2023.05 The Diminishing Returns of Masked Language Models to Science, arXiv, Hugging Face
2023.08 DARWIN Series: Domain Specific Large Language Models for Natural Science, arXiv, Code
2024.01 SciGLM: Training Scientific Language Models with Self-Reflective Instruction Annotation and Tuning, arXiv, GitHub
2024.03 Uni-SMART: Universal Science Multimodal Analysis and Research Transformer, arXiv
2024.05 INDUS: Effective and Efficient Language Models for Scientific Applications,arXiv

Datasets and Benchmarks

The MIMIC dataset, 2016.05. mimic-code, Data Descriptor: MIMIC-III, a freely accessible critical care database, Scientific Data
eICU-CRD. 2019.04. The eICU Collaborative Research Database, a freely available multi-center database for critical care research, Scientific Data
cMedQA2, 2018.11. Multi-Scale Attentive Interaction Networks for Chinese Medical Question Answer Selection, IEEE Access