An open medical vision-language model that handles medical text, 2D images, 3D CT/MRI volumes and surgical videos, trained on public data.
87834PythonApache-2.0Updated 2 months ago
ZJU4HealthCare/HealthGPT
A medical multimodal large language model family: HealthGPT-Pro for understanding medical text, 2D images and 3D volumes, and HealthGPT for unified comprehension and generation.
- Stars
- 1,660
- Forks
- 242
- Open issues
- 13
- Last commit
- 3 months ago
What is it?
HealthGPT Series is a medical multimodal large language model family made of two sub-projects. HealthGPT-Pro, built on Qwen3-VL, targets medical understanding and analysis across medical text, 2D images such as CT, MRI, X-ray, fundus and histopathology, and 3D medical volumes. It is released in 4B, 8B and 27B sizes on Hugging Face and ModelScope.
HealthGPT is the unified model in the series for medical comprehension and generation. It uses heterogeneous low-rank adaptation (H-LoRA) and a three-stage learning strategy so a pretrained LLM can follow both visual comprehension and generation instructions. The project was accepted to ICML 2025 as a Spotlight paper.
The repository includes the model weights, a dataset release (VL-Health), and benchmark results compared against models such as Lingshu and Hulu-Med on medical QA and vision-language tasks. Setup and inference details live in the sub-project directories.
Key features
- 01Medical text, 2D image and 3D volume understanding in one model (HealthGPT-Pro).
- 02Covers 14 medical modalities including CT, MRI, fundus, dermoscopy and histopathology.
- 03Unified comprehension and generation via heterogeneous low-rank adaptation (H-LoRA).
- 04Released weights in 4B, 8B and 27B sizes on Hugging Face and ModelScope.
- 05Includes the VL-Health dataset and benchmark comparisons on medical QA and VQA tasks.
Use cases
- Researching medical vision-language models that combine report understanding with image analysis.
- Evaluating medical multimodal LLMs against benchmarks such as VQA-RAD, SLAKE and PathVQA.
- Building medical question-answering tools over text and imaging data in a research setting.
- Studying unified comprehension and generation with parameter-efficient adapters in medical imaging.
Quick start
- 1
Open the HealthGPT-Pro README for setup and inference details
- 2
Or open the HealthGPT README for its setup and inference steps
Things to know
- Setup and inference instructions live in the subrepository READMEs, not in this top-level README.
- The HealthGPT-Pro variants range up to 27B parameters, which implies substantial hardware requirements.
Topics
FAQ
- What is HealthGPT?
- A medical multimodal large language model family: HealthGPT-Pro for understanding medical text, 2D images and 3D volumes, and HealthGPT for unified comprehension and generation. HealthGPT Series is a medical multimodal large language model family made of two sub-projects. HealthGPT-Pro, built on Qwen3-VL, targets medical understanding and analysis across medical text, 2D images such as CT, MRI, X-ray, fundus and histopathology, and 3D medical volumes. It is released in 4B, 8B and 27B sizes on Hugging Face and ModelScope.
- Is HealthGPT free to use? What license does it have?
- HealthGPT is open source under the Apache-2.0 license. Check the license terms before using it in commercial products.
- Is HealthGPT still maintained?
- Updated within 6 months. The last commit to HealthGPT was on Jul 31, 2026.
- What can I use HealthGPT for?
- Researching medical vision-language models that combine report understanding with image analysis. Evaluating medical multimodal LLMs against benchmarks such as VQA-RAD, SLAKE and PathVQA. Building medical question-answering tools over text and imaging data in a research setting. Studying unified comprehension and generation with parameter-efficient adapters in medical imaging.
- How do I install HealthGPT?
- Open the HealthGPT-Pro README for setup and inference details Or open the HealthGPT README for its setup and inference steps
- Where can I get HealthGPT?
- The source code and installation instructions are on GitHub: https://github.com/ZJU4HealthCare/HealthGPT
Summary generated by AI from the project's README. GitHub stats refresh daily.
Related projects
Unified framework for efficiently fine-tuning 100+ LLMs and VLMs with a zero-code CLI and Web UI.
75.6K9.3KPythonApache-2.0Updated 2 days ago
Open-source multimodal models (MiniCPM-V 4.7 and MiniCPM-o 4.5) for understanding images, video, documents and speech on local and on-device hardware.
26.9K2.1KPythonApache-2.0Updated 5 days ago
A fine-tuning, training, inference and deployment framework for 600+ large language models and 400+ multimodal models, covering SFT, DPO and GRPO.
15.9K1.7KPythonApache-2.0Updated 2 hours ago
A 9B-parameter open-weight multimodal model for visual understanding, spatial reasoning, agentic tool use, and embodied-AI research, built on Qwen3.5 and C-RADIOv4-H.
3.1K540PythonApache-2.0Updated last month
NEO is a family of native vision-language models that process pixels and words in one encoder-free dense architecture, with training and evaluation code.
90433PythonApache-2.0Updated 3 months ago