Repolode
Submit
tairov

tairov/llama2.mojo

Llama 2 style transformer inference written in pure Mojo, in a single file, using SIMD vectorization and multi-threading.

View on GitHubWebsite
Stars
2,128
Forks
139
Open issues
0
Last commit
last month

What is it?

llama2.mojo runs inference for small Llama-architecture models entirely in Mojo, with no Python runtime in the inference path. It is a port of the author's earlier llama2.py and is organized around Mojo's SIMD and vectorization primitives, with a parallelize-based worker pool for multi-threaded execution.

The project loads Karpathy's stories models (260K to 110M parameters) and TinyLlama-1.1B-Chat. Users pass a checkpoint file, a tokenizer file, a prompt, and sampling options such as temperature, step count, seed, and number of workers. The README includes benchmark tables comparing throughput against llama2.c, llama.cpp, and llama2.py on Apple M1 Max and x86 machines.

The code is MIT licensed and requires the Mojo compiler along with the MAX package, which provides the parallelize function.

Key features

  • 01Runs Llama 2 architecture models in a single Mojo source file.
  • 02Uses SIMD and vectorization primitives for matrix operations.
  • 03Supports multi-threaded inference with a configurable number of workers.
  • 04Loads stories 260K to 110M checkpoints and TinyLlama-1.1B-Chat.
  • 05Exposes command-line options for prompt, temperature, seed, and step count.

Use cases

  • Studying how a transformer forward pass maps onto SIMD and multi-core CPU code.
  • Experimenting with small Llama checkpoints on a local CPU.
  • Comparing language implementations of the same inference workload.
  • Learning the Mojo language through a complete, readable numerical program.

Quick start

  1. 1

    Install the Mojo compiler and MAX with uv

    uv venv --python 3.13 ~/.modular-venv && uv pip install --python ~/.modular-venv/bin/python mojo "max[all]"
  2. 2

    Clone the repository

    git clone https://github.com/tairov/llama2.mojo.git
  3. 3

    Download the stories15M model

    wget https://huggingface.co/karpathy/tinyllamas/resolve/main/stories15M.bin
  4. 4

    Run the model

    mojo llama2.mojo stories15M.bin -s 100 -n 256 -t 0.5 -i "Once upon a time"

Things to know

  • Requires a C compiler on the host to link Mojo executables.
  • Depends on the Mojo 1.1 toolchain and MAX, which are version-specific.

In collections

Run LLMs locally

Topics

#llm-inference#transformer#simd#cpu-inference#parallel-computing#llama#performance#inference#llama2#modular

FAQ

What is llama2.mojo?
Llama 2 style transformer inference written in pure Mojo, in a single file, using SIMD vectorization and multi-threading. llama2.mojo runs inference for small Llama-architecture models entirely in Mojo, with no Python runtime in the inference path. It is a port of the author's earlier llama2.py and is organized around Mojo's SIMD and vectorization primitives, with a parallelize-based worker pool for multi-threaded execution.
Is llama2.mojo free to use? What license does it have?
llama2.mojo is open source under the MIT license. Check the license terms before using it in commercial products.
Is llama2.mojo still maintained?
Updated within 6 months. The last commit to llama2.mojo was on Sep 30, 2026.
What can I use llama2.mojo for?
Studying how a transformer forward pass maps onto SIMD and multi-core CPU code. Experimenting with small Llama checkpoints on a local CPU. Comparing language implementations of the same inference workload. Learning the Mojo language through a complete, readable numerical program.
How do I install llama2.mojo?
Install the Mojo compiler and MAX with uv: uv venv --python 3.13 ~/.modular-venv && uv pip install --python ~/.modular-venv/bin/python mojo "max[all]" Clone the repository: git clone https://github.com/tairov/llama2.mojo.git Download the stories15M model: wget https://huggingface.co/karpathy/tinyllamas/resolve/main/stories15M.bin Run the model: mojo llama2.mojo stories15M.bin -s 100 -n 256 -t 0.5 -i "Once upon a time"
Where can I get llama2.mojo?
The source code and installation instructions are on GitHub: https://github.com/tairov/llama2.mojo

Summary generated by AI from the project's README. GitHub stats refresh daily.

Related projects