Repolode
Submit
PaddlePaddle

PaddlePaddle/PaddleFormers

A PaddlePaddle-based Transformers-style library for training large language and vision-language models, with a Hugging Face-compatible API.

View on GitHub
Stars
12,970
Forks
2,184
Open issues
244
Last commit
16 days ago

What is it?

PaddleFormers is a model library built on the PaddlePaddle deep learning framework that aims to offer the same kind of model interfaces and features that Hugging Face Transformers provides for PyTorch users. It covers training for large language models (LLMs) and vision-language models (VLMs), with a model zoo that includes DeepSeek-V3/V4, GLM-4.5, Qwen2, Qwen3, Qwen3-VL and ERNIE-4.5 families.

The library supports distributed training strategies such as tensor, pipeline and expert parallelism, along with automatic mixed precision and FP8 training. Post-training workflows include continued pretraining, SFT, LoRA, DPO and DPO-LoRA. Checkpoints are saved in Safetensors format, so they can be loaded by tools such as vLLM, SGLang and FastDeploy. It also provides a command-line tool, paddleformers-cli, for launching training from YAML configs.

The project documents adaptation to several domestic accelerators, including Kunlunxin P800, Iluvatar BI-V150 and Metax C550, as well as Moore Threads GPUs.

Key features

  • 01Hugging Face-style AutoModel and AutoTokenizer APIs on PaddlePaddle
  • 02Tensor, pipeline and expert parallelism for large model training
  • 03Pretraining, SFT, LoRA, DPO and DPO-LoRA workflows
  • 04Safetensors checkpoints compatible with vLLM, SGLang and FastDeploy
  • 05Support for Function Call and Thinking-style training data

Use cases

  • Fine-tuning open LLMs such as Qwen3 or GLM-4.5 on custom data with SFT or LoRA
  • Running preference alignment with DPO on PaddlePaddle infrastructure
  • Training or adapting vision-language models like Qwen3-VL or PaddleOCR-VL
  • Deploying domestic accelerator hardware for LLM post-training

Quick start

  1. 1

    Install the tokenizer and processor package (no training dependencies)

    python -m pip install paddleformers
  2. 2

    Load a tokenizer and run text generation with the Hugging Face-style API

  3. 3

    Clone the repository to get the example training configs

    git clone https://github.com/PaddlePaddle/PaddleFormers.git
  4. 4

    Run SFT training with the example config

    paddleformers-cli train ./examples/config/sft/full.yaml

Things to know

  • GPU training requires CUDA 13.0 or newer and PaddleFleet 0.3 or newer.
  • Other hardware such as Kunlunxin XPU, Iluvatar, Metax and MUSA needs separate vendor-specific installation guides.

Topics

#llm#paddlepaddle#transformers#distributed-training#fine-tuning#vision-language-model#lora#model

FAQ

What is PaddleFormers?
A PaddlePaddle-based Transformers-style library for training large language and vision-language models, with a Hugging Face-compatible API. PaddleFormers is a model library built on the PaddlePaddle deep learning framework that aims to offer the same kind of model interfaces and features that Hugging Face Transformers provides for PyTorch users. It covers training for large language models (LLMs) and vision-language models (VLMs), with a model zoo that includes DeepSeek-V3/V4, GLM-4.5, Qwen2, Qwen3, Qwen3-VL and ERNIE-4.5 families.
Is PaddleFormers free to use? What license does it have?
PaddleFormers is open source under the Apache-2.0 license. Check the license terms before using it in commercial products.
Is PaddleFormers still maintained?
Actively maintained. The last commit to PaddleFormers was on Oct 21, 2026.
What can I use PaddleFormers for?
Fine-tuning open LLMs such as Qwen3 or GLM-4.5 on custom data with SFT or LoRA Running preference alignment with DPO on PaddlePaddle infrastructure Training or adapting vision-language models like Qwen3-VL or PaddleOCR-VL Deploying domestic accelerator hardware for LLM post-training
How do I install PaddleFormers?
Install the tokenizer and processor package (no training dependencies): python -m pip install paddleformers Load a tokenizer and run text generation with the Hugging Face-style API Clone the repository to get the example training configs: git clone https://github.com/PaddlePaddle/PaddleFormers.git Run SFT training with the example config: paddleformers-cli train ./examples/config/sft/full.yaml
Where can I get PaddleFormers?
The source code and installation instructions are on GitHub: https://github.com/PaddlePaddle/PaddleFormers

Summary generated by AI from the project's README. GitHub stats refresh daily.

Related projects