Repolode
Submit
lyogavin

lyogavin/airllm

Python library that runs large language models layer by layer so 70B-class models can infer on a single 4GB GPU, without quantization.

View on GitHub
Stars
39,483
Forks
4,026
Open issues
167
Last commit
2 days ago

What is it?

AirLLM is a Python library for running large language models on GPUs with very little VRAM. Instead of loading the whole model at once, it keeps only one decoder layer on the GPU at a time, so the memory needed depends on layer size rather than total parameter count. The project reports running Llama 3.x 70B on about 4GB, DeepSeek-V3 (671B) on about 12GB, and newer releases such as Qwen3-235B and Kimi K3 on single cards.

Models are loaded through a single AutoModel.from_pretrained call that accepts a Hugging Face repo ID or a local path. Optional block-wise 4-bit or 8-bit compression can further reduce disk loading time and speed up inference. The library also supports macOS on Apple silicon via MLX, and CPU inference. Recent versions add LoRA-based fine-tuning that streams frozen base weights from disk.

The main trade-off is speed: because layers are read from disk for each pass, the original model is first split into per-layer files, which needs substantial free disk space.

Key features

  • 01Runs 70B-class models on a single 4GB GPU by keeping one layer in VRAM at a time.
  • 02Loads models from a Hugging Face repo ID or local path through one AutoModel call.
  • 03Offers optional 4-bit and 8-bit block-wise compression to speed up loading.
  • 04Supports macOS on Apple silicon through MLX, plus CPU inference.
  • 05Includes LoRA fine-tuning that streams frozen base weights from disk.

Use cases

  • Trying large open models like Llama 70B on a consumer GPU with limited VRAM.
  • Running a very large MoE model such as DeepSeek-V3 on a single workstation card.
  • Fine-tuning a big model with LoRA on a small GPU.
  • Running large models locally on an Apple silicon Mac.

Quick start

  1. 1

    Install the airllm package

    pip install airllm
  2. 2

    Load a model by Hugging Face repo ID and run inference (see the Quickstart code in the README)

    model = AutoModel.from_pretrained("Qwen/Qwen3-32B")

Things to know

  • Inference first decomposes the original model into layer-wise shards, so you need sufficient disk space in the Hugging Face cache.
  • Some newer models need specific transformers versions or extra packages, such as Kimi K3 requiring compressed-tensors and flash-attn.
  • MacOS support is limited to Apple silicon and requires installing mlx and torch.

Topics

#llm#inference#gpu#low-vram#pytorch#fine-tuning#lora#quantization#chinese-llm#chinese-nlp

FAQ

What is airllm?
Python library that runs large language models layer by layer so 70B-class models can infer on a single 4GB GPU, without quantization. AirLLM is a Python library for running large language models on GPUs with very little VRAM. Instead of loading the whole model at once, it keeps only one decoder layer on the GPU at a time, so the memory needed depends on layer size rather than total parameter count. The project reports running Llama 3.x 70B on about 4GB, DeepSeek-V3 (671B) on about 12GB, and newer releases such as Qwen3-235B and Kimi K3 on single cards.
Is airllm free to use? What license does it have?
airllm is open source under the Apache-2.0 license. Check the license terms before using it in commercial products.
Is airllm still maintained?
Actively maintained. The last commit to airllm was on Oct 19, 2026.
What can I use airllm for?
Trying large open models like Llama 70B on a consumer GPU with limited VRAM. Running a very large MoE model such as DeepSeek-V3 on a single workstation card. Fine-tuning a big model with LoRA on a small GPU. Running large models locally on an Apple silicon Mac.
How do I install airllm?
Install the airllm package: pip install airllm Load a model by Hugging Face repo ID and run inference (see the Quickstart code in the README): model = AutoModel.from_pretrained("Qwen/Qwen3-32B")
Where can I get airllm?
The source code and installation instructions are on GitHub: https://github.com/lyogavin/airllm

Summary generated by AI from the project's README. GitHub stats refresh daily.

Related projects