Repolode
Submit
microsoft

microsoft/BitNet

Official CPU/GPU inference framework from Microsoft for 1-bit (ternary, 1.58-bit) large language models, built on llama.cpp.

View on GitHub
Stars
41,213
Forks
3,836
Open issues
375
Last commit
3 months ago

What is it?

bitnet.cpp is an inference framework for 1-bit and ternary LLMs such as BitNet b1.58. It provides optimized kernels (I2_S, TL1, TL2) that run these models on x86 and ARM CPUs, with an official GPU kernel for accelerated deployment. The project is built on the llama.cpp codebase and uses GGUF model files.

The framework targets local and edge deployment of large models. Its published results report speedups over full-precision baselines on CPUs and substantial energy reductions, and it can run a 100B-parameter BitNet b1.58 model on a single CPU. Official models include BitNet-b1.58-2B-4T and 1-bit embedding models, and several community and third-party 1.58-bit models are supported.

Setup uses a Python script that downloads a Hugging Face model, converts it, and builds the runtime with CMake and clang. Included tools cover conversational inference, end-to-end benchmarking, and conversion from safetensors checkpoints.

Key features

  • 01Optimized CPU kernels (I2_S, TL1, TL2) for 1.58-bit model inference on x86 and ARM.
  • 02Official GPU inference kernel for 1-bit models.
  • 03Conversational mode for instruct models via a command-line script.
  • 04Conversion from safetensors checkpoints to GGUF format.
  • 05Built-in end-to-end benchmark script for measuring generation speed.

Use cases

  • Running a 1-bit LLM locally on a laptop or server CPU without a GPU.
  • Deploying small, energy-efficient language models on edge devices.
  • Benchmarking ternary model inference speed against full-precision models.
  • Experimenting with BitNet-style embedding models on CPUs.

Quick start

  1. 1

    Clone the repository with submodules

    git clone --recursive https://github.com/microsoft/BitNet.git
  2. 2

    Enter the project directory

    cd BitNet
  3. 3

    Create a conda environment and install Python dependencies

    pip install -r requirements.txt
  4. 4

    Download the model and build the environment for inference

    python setup_env.py -md models/BitNet-b1.58-2B-4T -q i2_s

Things to know

  • Requires a C++ toolchain with clang 18 or newer and CMake 3.22 or newer to build from source.
  • Windows users must install Visual Studio 2022 with C++ and Clang components.
  • The README says NPU support is not yet available.

Topics

#llm#inference#quantization#1-bit#cpu-inference#ternary-weights#edge-ai#llama-cpp

FAQ

What is BitNet?
Official CPU/GPU inference framework from Microsoft for 1-bit (ternary, 1.58-bit) large language models, built on llama.cpp. bitnet.cpp is an inference framework for 1-bit and ternary LLMs such as BitNet b1.58. It provides optimized kernels (I2_S, TL1, TL2) that run these models on x86 and ARM CPUs, with an official GPU kernel for accelerated deployment. The project is built on the llama.cpp codebase and uses GGUF model files.
Is BitNet free to use? What license does it have?
BitNet is open source under the MIT license. Check the license terms before using it in commercial products.
Is BitNet still maintained?
Updated within 6 months. The last commit to BitNet was on Jul 27, 2026.
What can I use BitNet for?
Running a 1-bit LLM locally on a laptop or server CPU without a GPU. Deploying small, energy-efficient language models on edge devices. Benchmarking ternary model inference speed against full-precision models. Experimenting with BitNet-style embedding models on CPUs.
How do I install BitNet?
Clone the repository with submodules: git clone --recursive https://github.com/microsoft/BitNet.git Enter the project directory: cd BitNet Create a conda environment and install Python dependencies: pip install -r requirements.txt Download the model and build the environment for inference: python setup_env.py -md models/BitNet-b1.58-2B-4T -q i2_s
Where can I get BitNet?
The source code and installation instructions are on GitHub: https://github.com/microsoft/BitNet

Summary generated by AI from the project's README. GitHub stats refresh daily.

Related projects