Repolode
Submit
jingyaogong

jingyaogong/minimind

An ultra-small open-source LLM project that teaches you to train a 64M-parameter language model from scratch in about 2 hours on a single GPU.

View on GitHubWebsite
Stars
64,986
Forks
8,511
Open issues
67
Last commit
12 days ago

What is it?

MiniMind is a from-scratch LLM training project built around a very small model, roughly 64M parameters for the dense version and about 198M total (64M active) for the MoE version. It covers the full pipeline in PyTorch: tokenizer training, pretraining, supervised fine-tuning, LoRA, DPO, PPO, GRPO, CISPO, tool calling, adaptive thinking, and knowledge distillation. The core algorithms are implemented natively rather than wrapped in high-level libraries, so the code can be read and modified line by line.

The project targets developers who want to understand how an LLM is built instead of only calling one through an API or fine-tuning an existing model. The authors report a single RTX 3090 SFT epoch of about 2 hours, and the repository ships pre-processed datasets and pretrained weights on ModelScope and Hugging Face. Trained models can be run with eval scripts, a Streamlit WebUI, or an OpenAI-compatible API server, and export paths to llama.cpp, vLLM and Ollama are provided.

The repository also includes experimental branches for a diffusion language model and linear attention, plus sibling projects for vision and omni-modal variants. Its model architecture is aligned with the Qwen3 family naming and configuration.

Key features

  • 01Trains a roughly 64M-parameter dense model and a MoE variant from scratch on a single consumer GPU.
  • 02Implements pretraining, SFT, LoRA, DPO, PPO, GRPO and CISPO in native PyTorch.
  • 03Supports tool calling and switchable thinking through the chat template.
  • 04Provides OpenAI-compatible API serving and a Streamlit chat WebUI.
  • 05Offers released datasets and checkpoints, with export paths to llama.cpp, vLLM and Ollama.

Use cases

  • Learning how a language model is built end to end by reading and running the training code.
  • Reproducing a small-scale LLM training pipeline on a single GPU for research or teaching.
  • Experimenting with RL post-training methods such as DPO, PPO and GRPO on a small model.
  • Prototyping domain-specific behavior by training LoRA adapters on your own dialogue data.

Quick start

  1. 1

    Clone the repository

    git clone --depth 1 https://github.com/jingyaogong/minimind
  2. 2

    Install dependencies

    cd minimind && pip install -r requirements.txt -i https://mirrors.aliyun.com/pypi/simple
  3. 3

    Download the minimind-3 model

    modelscope download --model gongjy/minimind-3 --local_dir ./minimind-3
  4. 4

    Run command-line inference with the model

    python eval_llm.py --load_from ./minimind-3

Things to know

  • Training from scratch is designed for a single GPU, and the README's 2-hour timing was measured on an NVIDIA 3090.
  • The model is very small (about 64M parameters), so its output quality is limited.
  • Its 6,400-token tokenizer compresses Chinese text less efficiently than larger tokenizers such as Qwen2.

In collections

Reinforcement learning frameworks & training environments

Topics

#llm#pytorch#training-from-scratch#fine-tuning#reinforcement-learning#moe#tutorial#artificial-intelligence#large-language-model

FAQ

What is minimind?
An ultra-small open-source LLM project that teaches you to train a 64M-parameter language model from scratch in about 2 hours on a single GPU. MiniMind is a from-scratch LLM training project built around a very small model, roughly 64M parameters for the dense version and about 198M total (64M active) for the MoE version. It covers the full pipeline in PyTorch: tokenizer training, pretraining, supervised fine-tuning, LoRA, DPO, PPO, GRPO, CISPO, tool calling, adaptive thinking, and knowledge distillation. The core algorithms are implemented natively rather than wrapped in high-level libraries, so the code can be read and modified line by line.
Is minimind free to use? What license does it have?
minimind is open source under the Apache-2.0 license. Check the license terms before using it in commercial products.
Is minimind still maintained?
Actively maintained. The last commit to minimind was on Oct 27, 2026.
What can I use minimind for?
Learning how a language model is built end to end by reading and running the training code. Reproducing a small-scale LLM training pipeline on a single GPU for research or teaching. Experimenting with RL post-training methods such as DPO, PPO and GRPO on a small model. Prototyping domain-specific behavior by training LoRA adapters on your own dialogue data.
How do I install minimind?
Clone the repository: git clone --depth 1 https://github.com/jingyaogong/minimind Install dependencies: cd minimind && pip install -r requirements.txt -i https://mirrors.aliyun.com/pypi/simple Download the minimind-3 model: modelscope download --model gongjy/minimind-3 --local_dir ./minimind-3 Run command-line inference with the model: python eval_llm.py --load_from ./minimind-3
Where can I get minimind?
The source code and installation instructions are on GitHub: https://github.com/jingyaogong/minimind

Summary generated by AI from the project's README. GitHub stats refresh daily.

Related projects