Repolode
Submit
skyzh

skyzh/tiny-llm

A hands-on course for systems engineers that builds a small Qwen3 LLM inference stack on Apple Silicon with MLX, from matmul up to a mini vLLM and coding agent.

View on GitHubWebsite
Stars
5,687
Forks
524
Open issues
12
Last commit
15 hours ago

What is it?

tiny-llm is a course for systems engineers who want to understand LLM inference end to end. Students implement the path that loads a Qwen3 model, turns tokens into logits, and generates text, starting from MLX array operations and adding kernels and serving machinery as the model needs them.

The course is built on MLX arrays and the MLX extension runtime without high-level neural-network layers. When a chapter teaches an operator, the learner implements it in Python, C++, or Metal, while MLX serves as the correctness oracle and performance baseline. The code is kept small enough to read in full.

The four-week path covers a single-request model with KV cache and quantized weights, a mini vLLM with continuous batching, chunked prefill, and paged KV cache, and a Week 4 coding agent built step by step with validated edits and checkpoints. The course targets Apple Silicon machines, and the book is published online with a reference solution package for tests.

Key features

  • 01Builds a Qwen3 model from MLX array operations including attention, RoPE, GQA, and RMSNorm
  • 02Implements KV cache, quantized matvec, and fused kernels in Python, C++, or Metal
  • 03Covers continuous batching, chunked prefill, and paged KV cache in a mini vLLM
  • 04Includes a reference solution and test suite for checking each chapter
  • 05Ends with a step-by-step coding agent with validated edits and resumable checkpoints

Use cases

  • Systems engineers who want to connect LLM inference equations to memory traffic and KV-cache growth
  • Developers learning how vLLM-style batching and paged attention work internally
  • Engineers running local model experiments on an Apple Silicon Mac

Quick start

  1. 1

    Read the environment setup guide in the book

  2. 2

    Install dependencies with PDM

    pdm install -v
  3. 3

    Verify the installation

    pdm run check-installation
  4. 4

    Run the Week 1 reference solution tests

    pdm run test-refsol -- -- -k week_1

Things to know

  • The course is built on MLX and is designed for Apple silicon Macs.
  • Week 2 and Week 4 chapters are still in progress and not all are finished.
  • The Week 4 coding agent can run commands and edit files, so use a disposable workspace without secrets.

In collections

Run LLMs locallyOpen-source Mac apps

Topics

#llm-inference#mlx#apple-silicon#qwen#kv-cache#course#llm-serving#large-language-model#llm#python

FAQ

What is tiny-llm?
A hands-on course for systems engineers that builds a small Qwen3 LLM inference stack on Apple Silicon with MLX, from matmul up to a mini vLLM and coding agent. tiny-llm is a course for systems engineers who want to understand LLM inference end to end. Students implement the path that loads a Qwen3 model, turns tokens into logits, and generates text, starting from MLX array operations and adding kernels and serving machinery as the model needs them.
Is tiny-llm free to use? What license does it have?
tiny-llm is open source under the Apache-2.0 license. Check the license terms before using it in commercial products.
Is tiny-llm still maintained?
Actively maintained. The last commit to tiny-llm was on Nov 7, 2026.
What can I use tiny-llm for?
Systems engineers who want to connect LLM inference equations to memory traffic and KV-cache growth Developers learning how vLLM-style batching and paged attention work internally Engineers running local model experiments on an Apple Silicon Mac
How do I install tiny-llm?
Read the environment setup guide in the book Install dependencies with PDM: pdm install -v Verify the installation: pdm run check-installation Run the Week 1 reference solution tests: pdm run test-refsol -- -- -k week_1
Where can I get tiny-llm?
The source code and installation instructions are on GitHub: https://github.com/skyzh/tiny-llm

Summary generated by AI from the project's README. GitHub stats refresh daily.

Related projects