Repolode
Submit
b4rtaz

b4rtaz/distributed-llama

A C++ tool that splits LLM inference across multiple home devices over Ethernet to speed up generation.

View on GitHub
Stars
3,115
Forks
258
Open issues
48
Last commit
11 days ago

What is it?

Distributed Llama connects several ordinary computers, such as home devices or Raspberry Pi boards, into one cluster for running large language models. A root node loads the model and weights, sends each worker its slice of the network, and synchronizes state, while worker nodes process their own portions. Tensor parallelism and high-speed synchronization over Ethernet are used so that adding devices increases inference speed.

The project runs on Linux, macOS and Windows, with optimizations for ARM and x86_64 AVX2 CPUs. It ships a single-command setup that downloads supported Llama, DeepSeek R1 Distill and Qwen 3 models in Q40 quantization, and it also includes a converter for Hugging Face models. Commands cover benchmarking inference, a CLI chat, running a worker, and an API server.

The cluster size is limited to 2^n nodes, and the maximum node count is tied to the model's number of KV heads. Only q40 with q80 buffer type and f32 with f32 buffer type are supported for quantization. Experimental Vulkan and GPU paths are documented separately.

Key features

  • 01Splits model computation across multiple devices using tensor parallelism.
  • 02Synchronizes across nodes over standard Ethernet.
  • 03Runs on Linux, macOS and Windows, with ARM and x86_64 AVX2 optimizations.
  • 04Provides a CLI chat, an inference benchmark, a worker mode and an API server.
  • 05Offers a one-command setup for several Llama, DeepSeek and Qwen 3 models.

Use cases

  • Running a 70B-class Llama model across several home computers or mini PCs.
  • Experimenting with LLM inference on Raspberry Pi clusters.
  • Serving a local OpenAI-style API from a small group of machines on a LAN.

Quick start

  1. 1

    Install Python 3 and a C++ compiler, then clone the repository

  2. 2

    Set up the root node with a single command (downloads the model and tokenizer)

    python launch.py llama3_2_1b_instruct_q40

Things to know

  • Node count must be 1, 2, 4, or 2^n, and the maximum is the number of model KV heads.
  • Only q40 models with q80 buffer type and f32 models with f32 buffer type are supported.
  • Vulkan support is experimental.

In collections

Run LLMs locally

Topics

#llm-inference#distributed-computing#tensor-parallelism#llama#cpp#edge-ai#distributed-llm#llama2#llama3#llm

FAQ

What is distributed-llama?
A C++ tool that splits LLM inference across multiple home devices over Ethernet to speed up generation. Distributed Llama connects several ordinary computers, such as home devices or Raspberry Pi boards, into one cluster for running large language models. A root node loads the model and weights, sends each worker its slice of the network, and synchronizes state, while worker nodes process their own portions. Tensor parallelism and high-speed synchronization over Ethernet are used so that adding devices increases inference speed.
Is distributed-llama free to use? What license does it have?
distributed-llama is open source under the MIT license. Check the license terms before using it in commercial products.
Is distributed-llama still maintained?
Actively maintained. The last commit to distributed-llama was on Oct 14, 2026.
What can I use distributed-llama for?
Running a 70B-class Llama model across several home computers or mini PCs. Experimenting with LLM inference on Raspberry Pi clusters. Serving a local OpenAI-style API from a small group of machines on a LAN.
How do I install distributed-llama?
Install Python 3 and a C++ compiler, then clone the repository Set up the root node with a single command (downloads the model and tokenizer): python launch.py llama3_2_1b_instruct_q40
Where can I get distributed-llama?
The source code and installation instructions are on GitHub: https://github.com/b4rtaz/distributed-llama

Summary generated by AI from the project's README. GitHub stats refresh daily.

Related projects