raiyanyahya/how-to-train-your-gpt

What is it?

Key features

• 01Twelve sequential chapters covering tokenization through inference
• 02Line-by-line comments explaining what each piece of code does and why
• 03Implements RoPE, RMSNorm, SwiGLU, pre-norm blocks and weight tying
• 04Custom training loop with AdamW, cosine warmup and mixed precision
• 05Inference with KV cache, temperature, top-k/top-p sampling and beam search
• 06Runnable main.py with a small CPU-friendly default config

Use cases

• Python developers who want to understand how ChatGPT-style models work internally
• Students learning Transformer attention and training from the ground up
• Engineers comparing architectural choices such as RoPE vs learned positions or RMSNorm vs LayerNorm
• Teachers preparing a hands-on lab on building a small language model

Quick start

1. 1

   Clone the repository

   git clone https://github.com/raiyanyahya/how-to-train-your-gpt.git
2. 2

   Create and activate a virtual environment (Mac/Linux)

   python -m venv gpt_env && source gpt_env/bin/activate
3. 3

   Install dependencies from the requirements file

   pip install -r requirements.txt
4. 4

   Run the training script (tiny config by default)

   python main.py

Things to know

• !The default config is a tiny model, and the 151M-parameter GPT-2 scale configuration needs a GPU.
• !This is a learning resource and is labelled for learning only, not for production models.

Topics

FAQ

What is how-to-train-your-gpt?

A 12-chapter, commented tutorial that walks you through building and training a small LLaMA-style GPT from scratch in Python and PyTorch. This is a learning project that teaches how a modern decoder-only language model works by building one from scratch. It covers the full pipeline: a BPE tokenizer, embeddings, rotary position encoding (RoPE), multi-head attention with a causal mask, transformer blocks using RMSNorm and SwiGLU, a training loop with AdamW and cosine warmup, and an inference engine with KV cache and sampling options.

Is how-to-train-your-gpt free to use? What license does it have?

how-to-train-your-gpt is open source under the MIT license. Check the license terms before using it in commercial products.

Is how-to-train-your-gpt still maintained?

Updated within 6 months. The last commit to how-to-train-your-gpt was on Aug 30, 2026.

What can I use how-to-train-your-gpt for?

Python developers who want to understand how ChatGPT-style models work internally Students learning Transformer attention and training from the ground up Engineers comparing architectural choices such as RoPE vs learned positions or RMSNorm vs LayerNorm Teachers preparing a hands-on lab on building a small language model

How do I install how-to-train-your-gpt?

Clone the repository: git clone https://github.com/raiyanyahya/how-to-train-your-gpt.git Create and activate a virtual environment (Mac/Linux): python -m venv gpt_env && source gpt_env/bin/activate Install dependencies from the requirements file: pip install -r requirements.txt Run the training script (tiny config by default): python main.py

Where can I get how-to-train-your-gpt?

The source code and installation instructions are on GitHub: https://github.com/raiyanyahya/how-to-train-your-gpt