This is an actively updated list of practical guide resources for Medical Large Language Models (Medical LLMs). It's based on our survey paper: A Survey of Large Language Models in Medicine: Principles, Applications, and Challenges.
If you want to add your work or model to this list, please do not hesitate to email [email protected] and [email protected] or pull requests. Markdown format:
- [**Name of Conference or Journal Year**] Paper Name. [[pdf]](link) [[code]](link)
This repository aims to provide an overview of the progress, application, and challenge of LLMs in medicine, aiming to promote further research and exploration in this interdisciplinary field.
[2023-12-11] We have updated our survey version 2. Please check it out!
[2023-12-04] We updated the outline image and radar chart evaluation of Medical LLM.
[2023-11-09] We released the repository.
- Practical Guide for Building Pipeline
- Practical Guide for Medical Data
- Downstream Biomedical Tasks
- Practical Guide for Clinical Applications
- Practical Guide for Challenges
- Practical Guide for Future Directions
- Acknowledgement
- Citation
- BioBERT: A pre-trained biomedical language representation model for biomedical text mining. 2020. paper
- PubMedBERT:Domain-specific language model pretraining for biomedical natural language processing. 2021. paper
- SciBERT:A pretrained language model for scientific text. 2019. paper
- ClinicalBERT:Publicly available clinical BERT embeddings. 2019. paper
- BlueBERT:Transfer learning in biomedical natural language processing: an evaluation of BERT and ELMo on ten benchmarking datasets. 2019. paper
- BioCPT:Contrastive pre-trained transformers with large-scale pubmed search logs for zero-shot biomedical information retrieval. 2023. paper
- BioGPT:generative pre-trained transformer for biomedical text generation and mining. 2022. paper
- OphGLM:Training an Ophthalmology Large Language-and-Vision Assistant based on Instructions and Dialogue. 2023. paper
- GatorTron:A large language model for electronic health records. 2022. paper
- GatorTronGPT:A Study of Generative Large Language Model for Medical Research and Healthcare. 2023. paper
- MEDITRON-70B: Scaling Medical Pretraining for Large Language Models. 2023. paper
- ChatGLM-Med:ChatGLM-Med: 基于中文医学知识的ChatGLM模型微调. 2023. github
- DoctorGLM:Fine-tuning your chinese doctor is not a herculean task. 2023. paper
- BianQue:Balancing the Questioning and Suggestion Ability of Health LLMs with Multi-turn Health Conversations Polished by ChatGPT. 2023. paper
- ClinicalGPT:Large Language Models Finetuned with Diverse Medical Data and Comprehensive Evaluation. 2023. paper
- Qilin-Med:Multi-stage Knowledge Injection Advanced Medical Large Language Model. 2023. paper
- Qilin-Med-VL:Towards Chinese Large Vision-Language Model for General Healthcare. 2023. paper
- Zhongjing:Enhancing the Chinese medical capabilities of large language model through expert feedback and real-world multi-turn dialogue. 2023. paper
- ChatDoctor:A Medical Chat Model Fine-Tuned on a Large Language Model Meta-AI (LLaMA) Using Medical Domain Knowledge. 2023. paper
- BenTsao:Tuning llama model with chinese medical knowledge. 2023. paper
- HuatuoGPT: HuatuoGPT, towards Taming Language Model to Be a Doctor. 2023. paper
- LLaVA-Med: Training a large language-and-vision assistant for biomedicine in one day. 2023. paper
- Baize-healthcare: An open-source chat model with parameter-efficient tuning on self-chat data. 2023. paper
- Visual Med-Alpeca: A parameter-efficient biomedical llm with visual capabilities. 2023. Repo
- PMC-LLaMA: Further finetuning llama on medical papers. 2023. paper
- Clinical Camel: An Open-Source Expert-Level Medical Language Model with Dialogue-Based Knowledge Encoding. 2023. paper
- MedPaLM 2: Towards expert-level medical question answering with large language models. 2023. paper
- MedPaLM M: Towards generalist biomedical ai. 2023. paper
- CPLLM: Clinical Prediction with Large Language Models. 2023. paper
- AlpaCare: Instruction-tuned Large Language Models for Medical Application. 2023. paper
- From Beginner to Expert: Modeling Medical Knowledge into General LLMs. 2023. paper
- Taiyi: A Bilingual Fine-Tuned Large Language Model for Diverse Biomedical Tasks. 2023. paper
- DelD-GPT: Zero-shot medical text de-identification by gpt-4. 2023. paper
- ChatCAD: Interactive computer-aided diagnosis on medical image using large language models. 2023. paper
- Dr. Knows: Leveraging a medical knowledge graph into large language models for diagnosis prediction. 2023. paper
- MedPaLM: Large language models encode clinical knowledge. 2022. paper
- MedPrompt: Can Generalist Foundation Models Outcompete Special-Purpose Tuning? Case Study in Medicine. paper
- Enhancing Medical Task Performance in GPT-4V: A Comprehensive Study on Prompt Engineering Strategies. paper
- Drugs.com
- DrugBank
- NHS Health
- NHS Medicine
- Unified Medical Language System (UMLS)
- The Human Phenotype Ontology
- Center for Disease Control and Prevention
- National Institute for Health and Care Excellence
- World Health Organization
- PubMed: National Institutes of Health. PubMed Data. In National Library of Medicine. 2022. database
- Literature: Construction of the literature graph in semantic scholar. 2018. paper
- MIMIC-III: MIMIC-III, a freely accessible critical care database. 2016. paper
- PubMed: The pile: An 800gb dataset of diverse text for language modeling. 2020. paper
- MedDialog: Meddialog: Two large-scale medical dialogue datasets. 2020. paper
- EHRs: A large language model for electronic health records. 2022. paper
- EHRs: A Study of Generative Large Language Model for Medical Research and Healthcare. 2023. paper
- Guidelines A high-quality collection of clinical practice guidelines (CPGs) for the medical training of LLMs. dataset
- GAP-REPLAY Scaling Medical Pretraining for Large Language Models. 2023. paper
- cMeKG:Chinese Medical Knowledge Graph. 2023. github
- CMD.: Chinese medical dialogue data. 2023. repo
- BianQueCorpus: BianQue: Balancing the Questioning and Suggestion Ability of Health LLMs with Multi-turn Health Conversations Polished by ChatGPT. 2023. paper
- MD-EHR: ClinicalGPT: Large Language Models Finetuned with Diverse Medical Data and Comprehensive Evaluation. 2023. paper
- VariousMedQA: Multi-scale attentive interaction networks for chinese medical question answer selection. 2018. paper
- VariousMedQA: What disease does this patient have? a large-scale open domain question answering dataset from medical exams. 2021. paper
- MedDialog: Meddialog: Two large-scale medical dialogue datasets. 2020. paper
- ChiMed: Qilin-Med: Multi-stage Knowledge Injection Advanced Medical Large Language Model. 2023. paper
- ChiMed-VL: Qilin-Med-VL: Towards Chinese Large Vision-Language Model for General Healthcare. 2023. paper
- Healthcare Magic: Healthcare Magic. platform
- ICliniq: ICliniq. platform
- Hybrid SFT: HuatuoGPT, towards Taming Language Model to Be a Doctor. 2023. paper
- PMC-15M: Large-scale domain-specific pretraining for biomedical vision-language processing. 2023. paper
- MedQuAD: A question-entailment approach to question answering. 2019. paper
- VariousMedQA: Visual med-alpaca: A parameter-efficient biomedical llm with visual capabilities. 2023. repo
- CMtMedQA:Zhongjing: Enhancing the Chinese medical capabilities of large language model through expert feedback and real-world multi-turn dialogue. 2023. paper
- MTB: Med-flamingo: a multimodal medical few-shot learner. 2023. paper
- PMC-OA: Pmc-clip: Contrastive language-image pre-training using biomedical documents. 2023. paper
- Medical Meadow: MedAlpaca--An Open-Source Collection of Medical Conversational AI Models and Training Data. 2023. paper
- Literature: S2ORC: The semantic scholar open research corpus. 2019. paper
- MedC-I: Pmc-llama: Further finetuning llama on medical papers. 2023. paper
- ShareGPT: Sharegpt. 2023. platform
- PubMed: National Institutes of Health. PubMed Data. In National Library of Medicine. 2022. database
- MedQA: What disease does this patient have? a large-scale open domain question answering dataset from medical exams. 2021. paper
- MultiMedQA: Towards expert-level medical question answering with large language models. 2023. paper
- MultiMedBench: Towards generalist biomedical ai. 2023. paper
- MedInstruct-52: Instruction-tuned Large Language Models for Medical Application. 2023. paper
- eICU-CRD: The eicu collaborative research database, a freely available multi-center database for critical care research. 2018. paper
- MIMIC-IV: MIMIC-IV, a freely accessible electronic health record dataset. 2023. paper database
- PMC-Patients: 167k open patient summaries. 2023. paper database
- PubMed: National Institutes of Health. PubMed Data. In National Library of Medicine. database
- PMC: National Institutes of Health. PubMed Central Data. In National Library of Medicine. database
- CORD-19: Cord-19: The covid-19 open research dataset 2020. paper
- MentSum: Mentsum: A resource for exploring summarization of mental health online posts 2022. paper
- MeQSum: On the summarization of consumer health questions 2019. paper
- MultiCochrane: Multilingual Simplification of Medical Texts 2023. paper
- AutoMeTS: AutoMeTS: the autocomplete for medical text simplification 2020. paper
- BioASQ-QA: BioASQ-QA: A manually curated corpus for Biomedical Question Answering 2023. paper
- emrQA: emrqa: A large corpus for question answering on electronic medical records 2018. paper
- CliCR: CliCR: a dataset of clinical case reports for machine reading comprehension 2018. paper
- PubMedQA: Pubmedqa: A dataset for biomedical research question answering 2019. paper
- COVID-QA: COVID-QA: A question answering dataset for COVID-19 2020. paper
- MASH-QA: Question answering with long multiple-span answers 2020. paper
- Health-QA: A hierarchical attention retrieval model for healthcare question answering 2019. paper
- MedQA: What disease does this patient have? a large-scale open domain question answering dataset from medical exams 2021. paper
- MedMCQA: Medmcqa: A large-scale multi-subject multi-choice dataset for medical domain question answering 2022. paper
- MMLU (Clinical Knowledge): Measuring massive multitask language understanding 2020. paper
- MMLU (College Medicine): Measuring massive multitask language understanding 2020. paper
- MMLU (Professional Medicine): Measuring massive multitask language understanding 2020. paper
- NCBI Disease: NCBI disease corpus: a resource for disease name recognition and concept normalization 2014. paper
- JNLPBA: Introduction to the bio-entity recognition task at JNLPBA 2004. paper
- GENIA: GENIA corpus--a semantically annotated corpus for bio-textmining 2003. paper
- BC5CDR: BioCreative V CDR task corpus: a resource for chemical disease relation extraction 2016. paper
- BC4CHEMD: The CHEMDNER corpus of chemicals and drugs and its annotation principles 2015. paper
- BioRED: BioRED: a rich biomedical relation extraction dataset 2022. paper
- CMeEE: Cblue: A chinese biomedical language understanding evaluation benchmark 2021. paper
- NLM-Chem-BC7: NLM-Chem-BC7: manually annotated full-text resources for chemical entity annotation and indexing in biomedical articles 2022. paper
- ADE: Development of a benchmark corpus to support the automatic extraction of drug-related adverse effects from medical case reports 2012. paper
- 2012 i2b2: Evaluating temporal relations in clinical text: 2012 i2b2 challenge 2013. paper
- 2014 i2b2/UTHealth (Track 1): Annotating longitudinal clinical narratives for de-identification: The 2014 i2b2/UTHealth corpus 2015. paper
- 2018 n2c2 (Track 2): 2018 n2c2 shared task on adverse drug events and medication extraction in electronic health records 2020. paper
- Cadec: Cadec: A corpus of adverse drug event annotations 2015. paper
- DDI: Semeval-2013 task 9: Extraction of drug-drug interactions from biomedical texts (ddiextraction 2013) 2013. paper
- PGR: A silver standard corpus of human phenotype-gene relations 2019. paper
- EU-ADR: The EU-ADR corpus: annotated drugs, diseases, targets, and their relationships 2012. paper
- BC5CDR: BioCreative V CDR task corpus: a resource for chemical disease relation extraction 2016. paper
- BioRED: BioRED: a rich biomedical relation extraction dataset 2022. paper
- ADE: Development of a benchmark corpus to support the automatic extraction of drug-related adverse effects from medical case reports 2012. paper
- 2018 n2c2 (Track 2): 2018 n2c2 shared task on adverse drug events and medication extraction in electronic health records 2020. paper
- 2010 i2b2/VA: 2010 i2b2/VA challenge on concepts, assertions, and relations in clinical text 2011. paper
- ChemProt: Overview of the BioCreative VI chemical-protein interaction Track 2017. database
- GDA: Renet: A deep learning approach for extracting gene-disease associations from literature 2019. paper
- DDI: Semeval-2013 task 9: Extraction of drug-drug interactions from biomedical texts (ddiextraction 2013) 2013. paper
- GAD: The genetic association database 2004. paper
- 2012 i2b2: Evaluating temporal relations in clinical text: 2012 i2b2 challenge 2013. paper
- PGR: A silver standard corpus of human phenotype-gene relations 2019. paper
- EU-ADR: The EU-ADR corpus: annotated drugs, diseases, targets, and their relationships 2012. paper
- ADE: Development of a benchmark corpus to support the automatic extraction of drug-related adverse effects from medical case reports 2012. paper
- 2014 i2b2/UTHealth (Track 2): Annotating longitudinal clinical narratives for de-identification: The 2014 i2b2/UTHealth corpus 2015. paper
- HoC: Automatic semantic classification of scientific literature according to the hallmarks of cancer 2016. paper
- OHSUMED: OHSUMED: An interactive retrieval evaluation and new large test collection for research 1994. paper
- WNUT-2020 Task 2: WNUT-2020 task 2: identification of informative COVID-19 english tweets 2020. paper
- Medical Abstracts: Evaluating unsupervised text classification: zero-shot and similarity-based approaches 2022. paper
- MIMIC-III: MIMIC-III, a freely accessible critical care database 2016. paper
- MedNLI: Lessons from natural language inference in the clinical domain 2018. paper
- BioNLI: BioNLI: Generating a Biomedical NLI Dataset Using Lexico-semantic Constraints for Adversarial Examples 2022. paper
- MedSTS: MedSTS: a resource for clinical semantic textual similarity 2020. paper
- 2019 n2c2/OHNLP: The 2019 n2c2/ohnlp track on clinical semantic textual similarity: overview 2020. paper
- BIOSSES: BIOSSES: a semantic sentence similarity estimation system for the biomedical domain 2017. paper
- TREC-COVID: TREC-COVID: constructing a pandemic information retrieval test collection 2021. paper
- NFCorpus: A full-text learning to rank dataset for medical information retrieval 2016. paper
- BioASQ (BEIR): Beir: A heterogenous benchmark for zero-shot evaluation of information retrieval models 2021. paper
- Retrieve, Summarize, and Verify: How Will ChatGPT Affect Information Seeking from the Medical Literature? paper
- Designing a Deep Learning-Driven Resource-Efficient Diagnostic System for Metastatic Breast Cancer: Reducing Long Delays of Clinical Diagnosis and Improving Patient Survival in Developing Countries. 2023. paper
- AI in health and medicine. 2022. paper
- Large language models in medicine. 2023. paper
- Leveraging a medical knowledge graph into large language models for diagnosis prediction. 2023. paper
- Chatcad: Interactive computer-aided diagnosis on medical image using large language models. 2023. paper
- Applying large language model artificial intelligence for retina International Classification of Diseases (ICD) coding. 2023. paper
- PLM-ICD: automatic ICD coding with pretrained language models. 2022. paper
- Using ChatGPT to write patient clinic letters. 2023. paper
- ChatGPT: the future of discharge summaries?. 2023. paper
- Chatcad: Interactive computer-aided diagnosis on medical image using large language models. 2023. paper
- Can GPT-4V (ision) Serve Medical Applications? Case Studies on GPT-4V for Multimodal Medical Diagnosis. 2023. paper
- Qilin-Med-VL: Towards Chinese Large Vision-Language Model for General Healthcare. 2023. paper
- Customizing General-Purpose Foundation Models for Medical Report Generation. 2023. paper
- Towards generalist foundation model for radiology. 2023. paper
- Clinical Text Summarization: Adapting Large Language Models Can Outperform Human Experts. 2023. paper
- MAIRA-1: A specialised large multimodal model for radiology report generation. 2023. paper
- Consensus, dissensus and synergy between clinicians and specialist foundation models in radiology report generation. 2023. paper
- Large Language Models in Medical Education: Opportunities, Challenges, and Future Directions. 2023. paper
- The Advent of Generative Language Models in Medical Education. 2023. paper
- The impending impacts of large language models on medical education. 2023. paper
- A Nested U-Structure for Instrument Segmentation in Robotic Surgery. 2023. paper
- The multi-trip autonomous mobile robot scheduling problem with time windows in a stochastic environment at smart hospitals. 2023. paper
- Advanced robotics for medical rehabilitation. 2016. paper
- GRID: Scene-Graph-based Instruction-driven Robotic Task Planning. 2023. paper
- Trust in Construction AI-Powered Collaborative Robots: A Qualitative Empirical Analysis. 2023. paper
- Machine translation of standardised medical terminology using natural language processing: A Scoping Review. 2023. paper
- The Advent of Generative Language Models in Medical Education. 2023. paper
- The impending impacts of large language models on medical education. 2023. paper
- ChatCounselor: A Large Language Models for Mental Health Support. 2023. paper
- Tell me, what are you most afraid of? Exploring the Effects of Agent Representation on Information Disclosure in Human-Chatbot Interaction, 2023, paper
- A Brief Wellbeing Training Session Delivered by a Humanoid Social Robot: A Pilot Randomized Controlled Trial. 2023. paper
- Real conversations with artificial intelligence: A comparison between human–human online conversations and human–chatbot conversations. 2015. paper
- Benefits and Harms of Large Language Models in Digital Mental Health. 2023. paper
- PsyChat: A Client-Centric Dialogue System for Mental Health Support. 2023. paper
- Survey of hallucination in natural language generation. 2023. paper
- Med-halt: Medical domain hallucination test for large language models. 2023. paper
- A survey of hallucination in large foundation models. 2023. paper
- Selfcheckgpt: Zero-resource black-box hallucination detection for generative large language models. 2023. paper
- Retrieval augmentation reduces hallucination in conversation. 2021. paper
- Chain-of-verification reduces hallucination in large language models. 2023. paper
- What disease does this patient have? a large-scale open domain question answering dataset from medical exams. 2021. paper
- Truthfulqa: Measuring how models mimic human falsehoods. 2021. paper
- HaluEval: A Large-Scale Hallucination Evaluation Benchmark for Large Language Models. 2023. paper
- Textbooks Are All You Need. 2023. paper
- Model Dementia: Generated Data Makes Models Forget. 2023. paper
- Detecting Edit Failures In Large Language Models: An Improved Specificity Benchmark. 2023. paper
- Editing Large Language Models: Problems, Methods, and Opportunities. 2023. paper
- Retrieval-augmented generation for knowledge-intensive nlp tasks. 2020. paper
- Aligning ai with shared human values. 2020. paper
- Training a helpful and harmless assistant with reinforcement learning from human feedback. 2022. paper
- Improving alignment of dialogue agents via targeted human judgements. 2022. paper
- Webgpt: Browser-assisted question-answering with human feedback. 2021. paper
- Languages are rewards: Hindsight finetuning using human feedback. 2023. paper
- ChatGPT utility in healthcare education, research, and practice: systematic review on the promising perspectives and valid concerns. 2023. paper
- ChatGPT listed as author on research papers: many scientists disapprove. 2023. paper
- A Survey of Large Language Models for Healthcare: from Data, Technology, and Applications to Accountability and Ethics. 2023. paper
- Multi-step jailbreaking privacy attacks on chatgpt. 2023. paper
- Jailbroken: How does llm safety training fail?. 2023. paper
- "Do Anything Now": Characterizing and Evaluating In-The-Wild Jailbreak Prompts on Large Language Models. 2023. paper
- A comprehensive benchmark study on biomedical text generation and mining with ChatGPT. 2023. paper
- Creation and adoption of large language models in medicine. 2023. paper
- Large Language Models in Sport Science & Medicine: Opportunities, Risks and Considerations. 2023. paper
- VisionFM: a Multi-Modal Multi-Task Vision Foundation Model for Generalist Ophthalmic Artificial Intelligence. 2023. paper
- Creation and adoption of large language models in medicine. 2023. paper
- ChatGPT and Physicians' Malpractice Risk. 2023. paper
- A Survey on Multimodal Large Language Models. 2023. paper
- Mm-react: Prompting chatgpt for multimodal reasoning and action. 2023. paper
- ChatGPT for shaping the future of dentistry: the potential of multi-modal large language model. 2023. paper
- Frozen Language Model Helps ECG Zero-Shot Learning. 2023. paper
- Exploring and Characterizing Large Language Models For Embedded System Development and Debugging. 2023. paper
- Holistic Evaluation of GPT-4V for Biomedical Imaging. 2023. paper
- The Rise and Potential of Large Language Model Based Agents: A Survey. 2023. paper
- MedAgents: Large Language Models as Collaborators for Zero-shot Medical Reasoning. 2023. paper code
- GeneGPT: Augmenting Large Language Models with Domain Tools for Improved Access to Biomedical Information. 2023. paper code
- OpenMedCalc: Augmentation of ChatGPT with Clinician-Informed Tools Improves Performance on Medical Calculation Tasks. 2023. paper
- LLMsPracticalGuide. The codebase we built upon and it is a comprehensive LLM survey.
- LargeAISurvey. Large AI Models in Health Informatics: Applications, Challenges, and the Future.
- Nature Medicine. Large language models in medicine. 2023.
Please consider citing 📑 our papers if our repository is helpful to your work, thanks sincerely!
@article{zhou2023survey,
title={A Survey of Large Language Models in Medicine: Principles, Applications, and Challenges},
author={Hongjian Zhou, Fenglin Liu, Boyang Gu, Xinyu Zou, Jinfa Huang, Jinge Wu, Yiru Li, Sam S. Chen, Peilin Zhou, Junling Liu, Yining Hua, Chengfeng Mao, Xian Wu, Yefeng Zheng, Lei Clifton, Zheng Li, Jiebo Luo, David A. Clifton},
journal={arXiv preprint 2311.05112}
year={2023}
}