FareedKhan-dev/train-llm-from-scratch

What is it?

Key features

• 01Builds attention, MLP and transformer blocks from scratch in PyTorch.
• 02Includes a byte-level BPE tokenizer implemented in the repo.
• 03Covers SFT with an assistant-only loss mask on chat data.
• 04Implements reward modeling, DPO/ORPO/KTO, PPO and GRPO.
• 05Offers a tiny preset that trains on a laptop CPU without a GPU.

Use cases

• Learning how a transformer language model and its training loop work step by step.
• Pretraining a small LLM on a single GPU or a laptop as an experiment.
• Studying post-training methods like DPO, PPO and GRPO in a readable codebase.
• Comparing classic and modern transformer components on the same pipeline.

Quick start

1. 1

   Clone the repository

   git clone https://github.com/FareedKhan-dev/train-llm-from-scratch.git
2. 2

   Enter the project folder

   cd train-llm-from-scratch
3. 3

   Install the project in editable mode

   pip install -e .
4. 4

   Train a tiny model on a laptop CPU after preparing data

   python scripts/train_transformer.py --preset tiny --arch modern

Things to know

• !Large model configurations need a GPU, and a billion-parameter model will not fit on a free T4.
• !Some stages depend on public datasets that must be downloaded first.

Topics

FAQ

What is train-llm-from-scratch?

A from-scratch PyTorch walkthrough and codebase for training a transformer LLM, covering pretraining, SFT, reward modeling, DPO/PPO/GRPO and generation. This repository implements a transformer language model from scratch in plain PyTorch, following the original "Attention is All You Need" design, and walks through the full path from raw text to a chat-style model. It includes a byte-level BPE tokenizer, a classic GPT-2 style decoder and a modern variant with RoPE, RMSNorm, SwiGLU, grouped-query attention and Mixture of Experts, all written by hand rather than using Hugging Face trainers.

Is train-llm-from-scratch free to use? What license does it have?

train-llm-from-scratch is open source under the MIT license. Check the license terms before using it in commercial products.

Is train-llm-from-scratch still maintained?

Actively maintained. The last commit to train-llm-from-scratch was on Oct 2, 2026.

What can I use train-llm-from-scratch for?

Learning how a transformer language model and its training loop work step by step. Pretraining a small LLM on a single GPU or a laptop as an experiment. Studying post-training methods like DPO, PPO and GRPO in a readable codebase. Comparing classic and modern transformer components on the same pipeline.

How do I install train-llm-from-scratch?

Clone the repository: git clone https://github.com/FareedKhan-dev/train-llm-from-scratch.git Enter the project folder: cd train-llm-from-scratch Install the project in editable mode: pip install -e . Train a tiny model on a laptop CPU after preparing data: python scripts/train_transformer.py --preset tiny --arch modern

Where can I get train-llm-from-scratch?

The source code and installation instructions are on GitHub: https://github.com/FareedKhan-dev/train-llm-from-scratch