Repolode
Submit
rasbt

rasbt/LLMs-from-scratch

Companion code for the book Build a Large Language Model (From Scratch), implementing a GPT-like LLM in PyTorch step by step.

View on GitHubWebsite
Stars
106,975
Forks
16,469
Open issues
4
Last commit
19 days ago

What is it?

This repository contains the code for developing, pretraining, and finetuning a GPT-like large language model in PyTorch, without relying on external LLM libraries. It is organized by book chapter, covering text data preparation and tokenization, attention mechanisms, building a GPT model, pretraining on unlabeled data, finetuning for classification, and instruction finetuning.

The code is designed to run on a conventional laptop within a reasonable time, and it uses a GPU automatically when one is available. Beyond the main chapters, bonus folders explore topics such as BPE tokenizers, KV cache, grouped-query and multi-head latent attention, mixture-of-experts, DPO preference tuning, LoRA, and from-scratch implementations of architectures like Llama 3.2, Qwen3, and Gemma 3. Loading weights of larger pretrained models for finetuning is also included.

The project is the official code companion to the Manning book, and a video course and a sequel on reasoning models are also available.

Key features

  • 01Step-by-step implementation of a GPT-like model in PyTorch without external LLM libraries
  • 02Covers tokenization, attention, pretraining, classification finetuning and instruction finetuning
  • 03Runs on conventional laptops and uses a GPU automatically when available
  • 04Bonus material on BPE, KV cache, MoE, DPO and LoRA
  • 05From-scratch code for architectures such as Llama 3.2, Qwen3 and Gemma 3

Use cases

  • Learning how LLMs work internally by writing the code yourself
  • Following along with the Build a Large Language Model book or its video course
  • Experimenting with pretraining a small GPT model on a laptop for educational purposes
  • Studying modern attention variants and architectures through readable PyTorch code

Quick start

  1. 1

    Clone the repository with a shallow copy

    git clone --depth 1 https://github.com/rasbt/LLMs-from-scratch.git
  2. 2

    Follow the setup guide in the setup folder to install Python and the required packages

  3. 3

    Open the main chapter notebooks, such as ch02/01_main-chapter-code/ch02.ipynb

Things to know

  • The metadata lists the license as NOASSERTION, so the license terms are unclear and should be checked before reuse.
  • This is companion code for a book, so it is a set of learning notebooks and scripts rather than a packaged library or application.

Topics

#llm#pytorch#deep-learning#gpt#transformers#education#finetuning#pretraining#ai#artificial-intelligence

FAQ

What is LLMs-from-scratch?
Companion code for the book Build a Large Language Model (From Scratch), implementing a GPT-like LLM in PyTorch step by step. This repository contains the code for developing, pretraining, and finetuning a GPT-like large language model in PyTorch, without relying on external LLM libraries. It is organized by book chapter, covering text data preparation and tokenization, attention mechanisms, building a GPT model, pretraining on unlabeled data, finetuning for classification, and instruction finetuning.
Is LLMs-from-scratch free to use? What license does it have?
LLMs-from-scratch is open source under the Other license. Check the license terms before using it in commercial products.
Is LLMs-from-scratch still maintained?
Actively maintained. The last commit to LLMs-from-scratch was on Oct 2, 2026.
What can I use LLMs-from-scratch for?
Learning how LLMs work internally by writing the code yourself Following along with the Build a Large Language Model book or its video course Experimenting with pretraining a small GPT model on a laptop for educational purposes Studying modern attention variants and architectures through readable PyTorch code
How do I install LLMs-from-scratch?
Clone the repository with a shallow copy: git clone --depth 1 https://github.com/rasbt/LLMs-from-scratch.git Follow the setup guide in the setup folder to install Python and the required packages Open the main chapter notebooks, such as ch02/01_main-chapter-code/ch02.ipynb
Where can I get LLMs-from-scratch?
The source code and installation instructions are on GitHub: https://github.com/rasbt/LLMs-from-scratch

Summary generated by AI from the project's README. GitHub stats refresh daily.

Related projects