Repolode
Submit
karpathy

karpathy/nanochat

A minimal, hackable harness that covers the full LLM pipeline from tokenization and pretraining to finetuning, evaluation and chat, runnable on a single GPU node.

View on GitHub
Stars
58,786
Forks
8,229
Open issues
122
Last commit
last month

What is it?

nanochat is an experimental harness for training large language models, aimed at running on a single GPU node with a small, readable codebase. It covers tokenization, pretraining, finetuning, evaluation and inference, and the project includes a simple CLI for chatting with the models you train.

The main control is a single complexity dial, `--depth`, which sets the number of transformer layers. Width, attention heads, learning rates, training horizon and weight decay are derived automatically so that models at each depth are compute-optimal. The project's stated goal is to reproduce GPT-2-level capability for a few tens of dollars of cloud GPU time, with a public leaderboard tracking training time to that target.

The repository provides scripts for speedrun training on an 8xH100 node, scaling-law and model-series experiments, a CPU/Apple Silicon example with a much smaller model, and documented precision handling across bf16, fp16 and fp32.

Key features

  • 01Covers tokenization, pretraining, finetuning, evaluation and inference in one codebase.
  • 02Model size is set by a single --depth parameter, with other hyperparameters computed automatically.
  • 03Includes a CLI for chatting with trained models.
  • 04Runs on a single GPU, with gradient accumulation fallback when only one GPU is available.
  • 05Supports CUDA GPUs, with a CPU and MPS path using a smaller model.
  • 06Provides a GPT-2 speedrun leaderboard tracking wall-clock training time.

Use cases

  • Learning how a full LLM training pipeline fits together by reading a small codebase.
  • Running small-scale pretraining experiments on a single GPU.
  • Testing architecture or optimizer changes with quick, depth-controlled runs.
  • Training a small chat model end to end on a limited budget.

Quick start

  1. 1

    Install uv dependencies with GPU extra

    uv sync --extra gpu
  2. 2

    Activate the virtual environment

    source .venv/bin/activate
  3. 3

    Run the speedrun training script (designed for an 8XH100 node, about 1.5 hours)

    bash runs/speedrun.sh
  4. 4

    Chat with the trained model over the CLI

    python -m scripts.chat_cli

Things to know

  • Full GPT-2 speedrun is designed for an 8XH100 GPU node and costs roughly $48 or more in cloud GPU time.
  • Pretraining needs substantial GPU resources; GPUs with less than 80GB VRAM require tuning batch size.
  • Most of the code is PyTorch, but non-CUDA backends such as MPS and XPU have not been fully tested by the author.

Topics

#llm#pretraining#finetuning#pytorch#gpt#deep-learning#training

FAQ

What is nanochat?
A minimal, hackable harness that covers the full LLM pipeline from tokenization and pretraining to finetuning, evaluation and chat, runnable on a single GPU node. nanochat is an experimental harness for training large language models, aimed at running on a single GPU node with a small, readable codebase. It covers tokenization, pretraining, finetuning, evaluation and inference, and the project includes a simple CLI for chatting with the models you train.
Is nanochat free to use? What license does it have?
nanochat is open source under the MIT license. Check the license terms before using it in commercial products.
Is nanochat still maintained?
Updated within 6 months. The last commit to nanochat was on Sep 7, 2026.
What can I use nanochat for?
Learning how a full LLM training pipeline fits together by reading a small codebase. Running small-scale pretraining experiments on a single GPU. Testing architecture or optimizer changes with quick, depth-controlled runs. Training a small chat model end to end on a limited budget.
How do I install nanochat?
Install uv dependencies with GPU extra: uv sync --extra gpu Activate the virtual environment: source .venv/bin/activate Run the speedrun training script (designed for an 8XH100 node, about 1.5 hours): bash runs/speedrun.sh Chat with the trained model over the CLI: python -m scripts.chat_cli
Where can I get nanochat?
The source code and installation instructions are on GitHub: https://github.com/karpathy/nanochat

Summary generated by AI from the project's README. GitHub stats refresh daily.

Related projects