Repolode
Submit
DLLXW

DLLXW/baby-llama2-chinese

A from-scratch pretraining and SFT pipeline for a small Chinese LLaMA2-style model, runnable on a single 24G GPU.

View on GitHub
Stars
2,923
Forks
354
Open issues
42
Last commit
2 months ago

What is it?

Baby-Llama2-Chinese is a learning-oriented project that builds a small Chinese LLaMA2-style language model end to end. It covers corpus tokenization, pretraining, supervised fine-tuning (SFT) on instruction datasets, and inference. The stated goal is to produce a chat model with basic Chinese question-answering ability on a single 24G GPU.

The pipeline uses the ChatGLM2-6B tokenizer, whose 64,793-token vocabulary fits in uint16 storage, and processes pretraining corpora such as Wikipedia-zh, BaiduBaiKe, C4_zh and WuDaoCorpora into binary files. Pretraining runs through torchrun on multiple GPUs, and SFT scripts build prompt-answer samples with loss computed only on the answer. Pretrained checkpoints from 92M to 218M parameters, along with several SFT variants for general and medical chat, are published for download.

The repository also includes data cleaning scripts for short-text filtering and MinHash or SimHash deduplication, plus a unified infer.py that loads historical checkpoints and detects model dimensions automatically. A reward model and reinforcement learning stage are listed as planned but not yet implemented.

Key features

  • 01Covers tokenizer setup, pretraining, and full-parameter SFT in one repository.
  • 02Pretrains a Chinese model from 92M to 218M parameters on a single 24G GPU.
  • 03Preprocesses corpora into uint16 binary files using the ChatGLM2-6B tokenizer.
  • 04Includes MinHash and SimHash deduplication scripts for corpus cleaning.
  • 05Provides infer.py that loads pretrain and SFT checkpoints with auto-detected dimensions.
  • 06Publishes pretrained and fine-tuned weights for download.

Use cases

  • Learning how a small LLaMA-style model is pretrained and fine-tuned from scratch.
  • Experimenting with Chinese pretraining corpora on limited GPU hardware.
  • Comparing how model size and vocabulary size affect pretraining loss.
  • Building a small domain-specific chat model, such as a medical QA model, from existing SFT data.

Quick start

  1. 1

    Download the tokenized pretraining corpus from the Baidu Netdisk link (about 118G)

  2. 2

    Place the downloaded data in the ./data directory and run the data processing script

    python data_process.py
  3. 3

    Run pretraining with torchrun on 4 GPUs (example setup from the README)

    torchrun --standalone --nproc_per_node=4 pretrain.py
  4. 4

    Run SFT fine-tuning

    python sft.py

Things to know

  • The README's example pretraining setup assumes multiple GPUs, and the author notes the 4x3090 hardware limits model and corpus size.
  • The full pretraining corpus is about 118G and must be downloaded from Baidu Netdisk.
  • Most data and model downloads are hosted on Baidu Netdisk, which can be inconvenient outside China.
  • Documentation is primarily in Chinese, and the reinforcement learning stage is still marked as to-do.

Topics

#llm#pretraining#fine-tuning#chinese-nlp#llama#pytorch#sft#tokenizer

FAQ

What is baby-llama2-chinese?
A from-scratch pretraining and SFT pipeline for a small Chinese LLaMA2-style model, runnable on a single 24G GPU. Baby-Llama2-Chinese is a learning-oriented project that builds a small Chinese LLaMA2-style language model end to end. It covers corpus tokenization, pretraining, supervised fine-tuning (SFT) on instruction datasets, and inference. The stated goal is to produce a chat model with basic Chinese question-answering ability on a single 24G GPU.
Is baby-llama2-chinese free to use? What license does it have?
baby-llama2-chinese is open source under the MIT license. Check the license terms before using it in commercial products.
Is baby-llama2-chinese still maintained?
Updated within 6 months. The last commit to baby-llama2-chinese was on Aug 13, 2026.
What can I use baby-llama2-chinese for?
Learning how a small LLaMA-style model is pretrained and fine-tuned from scratch. Experimenting with Chinese pretraining corpora on limited GPU hardware. Comparing how model size and vocabulary size affect pretraining loss. Building a small domain-specific chat model, such as a medical QA model, from existing SFT data.
How do I install baby-llama2-chinese?
Download the tokenized pretraining corpus from the Baidu Netdisk link (about 118G) Place the downloaded data in the ./data directory and run the data processing script: python data_process.py Run pretraining with torchrun on 4 GPUs (example setup from the README): torchrun --standalone --nproc_per_node=4 pretrain.py Run SFT fine-tuning: python sft.py
Where can I get baby-llama2-chinese?
The source code and installation instructions are on GitHub: https://github.com/DLLXW/baby-llama2-chinese

Summary generated by AI from the project's README. GitHub stats refresh daily.

Related projects