Repolode
Submit
ridgerchu

ridgerchu/matmulfreellm

Open-source implementation of MatMul-free language models that replace matrix multiplications with ternary weights and additive operations, compatible with Hugging Face Transformers.

View on GitHub
Stars
3,101
Forks
204
Open issues
25
Last commit
2 months ago

What is it?

MatMul-Free LM is a language model architecture that removes matrix multiplication (MatMul) operations from the model, aiming to reduce the compute cost of large language models. This repository provides a PyTorch implementation that plugs into the Hugging Face Transformers library, so models can be initialized through AutoModel and used with the standard generation APIs.

The project includes pretrained checkpoints at 370M, 1.3B and 2.7B parameters, with the 370M and 1.3B models trained on 15B and 100B tokens respectively. Kernels are written in Triton and the package depends on PyTorch 2.0 or later and einops. The code is adapted from flash-linear-attention and corresponds to the Scalable MatMul-free Language Modeling paper (arXiv 2406.02528).

Key features

  • 01Language model architecture with no matrix multiplication operations
  • 02Compatible with Hugging Face AutoModel and generate APIs
  • 03Pretrained models available at 370M, 1.3B and 2.7B parameters
  • 04Triton kernels built on flash-linear-attention code
  • 05Installable directly from the GitHub repository

Use cases

  • Researchers studying efficient language model architectures without MatMul
  • Experimenting with ternary-weight LLMs on GPUs using Transformers tooling
  • Pretraining or fine-tuning MatMul-free models on custom data
  • Exploring neuromorphic or low-compute hardware targets for LLMs

Quick start

  1. 1

    Install the required dependencies: PyTorch 2.0 or later, Triton 2.2 or later, and einops

  2. 2

    Install the library directly from GitHub

    pip install -U git+https://github.com/ridgerchu/matmulfreellm

Things to know

  • Requires a CUDA GPU for the provided generation example, which calls .cuda() and .half().
  • Depends on Triton, which adds a non-trivial GPU-specific dependency stack.

Topics

#large-language-model#llm#transformers#triton#pytorch#efficient-inference#ternary-weights#linear-transformer

FAQ

What is matmulfreellm?
Open-source implementation of MatMul-free language models that replace matrix multiplications with ternary weights and additive operations, compatible with Hugging Face Transformers. MatMul-Free LM is a language model architecture that removes matrix multiplication (MatMul) operations from the model, aiming to reduce the compute cost of large language models. This repository provides a PyTorch implementation that plugs into the Hugging Face Transformers library, so models can be initialized through AutoModel and used with the standard generation APIs.
Is matmulfreellm free to use? What license does it have?
matmulfreellm is open source under the Apache-2.0 license. Check the license terms before using it in commercial products.
Is matmulfreellm still maintained?
Updated within 6 months. The last commit to matmulfreellm was on Sep 6, 2026.
What can I use matmulfreellm for?
Researchers studying efficient language model architectures without MatMul Experimenting with ternary-weight LLMs on GPUs using Transformers tooling Pretraining or fine-tuning MatMul-free models on custom data Exploring neuromorphic or low-compute hardware targets for LLMs
How do I install matmulfreellm?
Install the required dependencies: PyTorch 2.0 or later, Triton 2.2 or later, and einops Install the library directly from GitHub: pip install -U git+https://github.com/ridgerchu/matmulfreellm
Where can I get matmulfreellm?
The source code and installation instructions are on GitHub: https://github.com/ridgerchu/matmulfreellm

Summary generated by AI from the project's README. GitHub stats refresh daily.

Related projects