A tiny open-source vision-language model (about 65M parameters) that you can train from scratch on a single GPU in roughly two hours.
8.8K+14 this week988PythonApache-2.0Updated 11 days ago
2U1/Qwen-VL-Series-Finetune
Training scripts for fine-tuning the Qwen-VL vision-language model family (Qwen2-VL, Qwen2.5-VL, Qwen3-VL, Qwen3.5) with Hugging Face and Liger-Kernel.
- Stars
- 1,980
- Forks
- 223
- Open issues
- 57
- Last commit
- 2 months ago
What is it?
This project provides training scripts for adapting Qwen-VL series vision-language models to custom data. It covers Qwen2-VL, Qwen2.5-VL, Qwen3-VL and Qwen3.5, using Hugging Face Transformers and Liger-Kernel rather than a separate training framework.
The scripts support full fine-tuning, LoRA and QLoRA, DoRA, DeepSpeed ZeRO, and freezing or unfreezing the vision tower and language model layers separately. Training data follows the LLaVA conversation format and can include single-image, multi-image, video and mixed-modality samples. Beyond supervised fine-tuning, the repo includes DPO, GRPO with custom reward functions, and classification training with several loss options.
It also documents reasoning-trace training for supported Qwen3-VL Thinking and Qwen3.5 models, generation-based evaluation during training, a Gradio demo for inference, and a Docker image with a prebuilt environment.
Key features
- 01Full fine-tuning, LoRA, QLoRA and DoRA for Qwen-VL models.
- 02Training on single-image, multi-image, video and mixed-modality datasets in LLaVA format.
- 03Preference optimization (DPO) and GRPO with user-defined reward functions.
- 04Classification fine-tuning with cross-entropy or focal loss variants.
- 05Separate learning rates and freezing controls for vision and language modules.
- 06DeepSpeed ZeRO support and Liger-Kernel acceleration.
Use cases
- Adapting a Qwen2.5-VL or Qwen3-VL model to a domain-specific image question-answering dataset.
- Training a video understanding model from clips paired with text descriptions.
- Building an image classifier on top of a Qwen-VL backbone.
- Running reinforcement-learning style tuning with custom rewards on a vision-language model.
Quick start
- 1
Install the Python packages from requirements.txt
pip install -r requirements.txt -f https://download.pytorch.org/whl/cu128 - 2
Install qwen-vl-utils
pip install qwen-vl-utils - 3
Install flash-attn after the other packages
pip install flash-attn --no-build-isolation
Things to know
- Training setup assumes Ubuntu with a recent Nvidia GPU and CUDA 12.8.
- Flash Attention 2 is unstable on Qwen3.5, so the README recommends disabling it there.
- QLoRA cannot be combined with vision training or Liger-Kernel.
Topics
#multimodal#vision-language-model#fine-tuning#lora#qwen#deepspeed#llm-training#qwen2-5-vl#qwen2-vl#qwen3-5
FAQ
- What is Qwen-VL-Series-Finetune?
- Training scripts for fine-tuning the Qwen-VL vision-language model family (Qwen2-VL, Qwen2.5-VL, Qwen3-VL, Qwen3.5) with Hugging Face and Liger-Kernel. This project provides training scripts for adapting Qwen-VL series vision-language models to custom data. It covers Qwen2-VL, Qwen2.5-VL, Qwen3-VL and Qwen3.5, using Hugging Face Transformers and Liger-Kernel rather than a separate training framework.
- Is Qwen-VL-Series-Finetune free to use? What license does it have?
- Qwen-VL-Series-Finetune is open source under the Apache-2.0 license. Check the license terms before using it in commercial products.
- Is Qwen-VL-Series-Finetune still maintained?
- Updated within 6 months. The last commit to Qwen-VL-Series-Finetune was on Sep 9, 2026.
- What can I use Qwen-VL-Series-Finetune for?
- Adapting a Qwen2.5-VL or Qwen3-VL model to a domain-specific image question-answering dataset. Training a video understanding model from clips paired with text descriptions. Building an image classifier on top of a Qwen-VL backbone. Running reinforcement-learning style tuning with custom rewards on a vision-language model.
- How do I install Qwen-VL-Series-Finetune?
- Install the Python packages from requirements.txt: pip install -r requirements.txt -f https://download.pytorch.org/whl/cu128 Install qwen-vl-utils: pip install qwen-vl-utils Install flash-attn after the other packages: pip install flash-attn --no-build-isolation
- Where can I get Qwen-VL-Series-Finetune?
- The source code and installation instructions are on GitHub: https://github.com/2U1/Qwen-VL-Series-Finetune
Summary generated by AI from the project's README. GitHub stats refresh daily.
Related projects
A small educational project for running vision-language model inference on Apple Silicon with MLX, scoring candidate answers about one image.
327+1 this week30PythonMITUpdated 2 months ago
Turns your exported chat history into a fine-tuned LLM that mimics your style, then deploys it as a chatbot on Telegram, Discord, Slack or WeChat.
18.4K+25 this week1.5KPythonAGPL-3.0Well maintainedUpdated 2 days ago
A PaddlePaddle-based Transformers-style library for training large language and vision-language models, with a Hugging Face-compatible API.
13K+1 this week2.2KPythonApache-2.0Well maintainedUpdated 17 days ago
A Chinese-language tutorial collection for deploying, using and fine-tuning open-source LLMs and multimodal models on Linux, with full, LoRA and P-tuning guides.
32.7K+71 this week3.1KJupyter NotebookApache-2.0Updated 9 days ago
A medical multimodal large language model family: HealthGPT-Pro for understanding medical text, 2D images and 3D volumes, and HealthGPT for unified comprehension and generation.
1.7K+1 this week242PythonApache-2.0Updated 3 months ago