Repolode
Submit
state-spaces

state-spaces/mamba

Mamba is a PyTorch sequence-modeling architecture built on selective state space models, offering a linear-time alternative to Transformers for language and other sequence tasks.

View on GitHub
Stars
18,939
Forks
1,831
Open issues
640
Last commit
3 months ago

What is it?

Mamba is a sequence modeling architecture based on selective state space models (SSMs). It targets information-dense data such as language, where earlier subquadratic models have trailed Transformers, and uses a hardware-aware implementation inspired by FlashAttention.

The repository provides the core selective SSM operator, the Mamba and Mamba-2 blocks, a Mamba-3 block, and a complete language model wrapper that stacks blocks with a language modeling head. Installation supports CUDA selective scan builds as well as a pure Python core package, and AMD ROCm builds are covered with a patch note. Pretrained checkpoints from 130M to 2.8B parameters are hosted on Hugging Face, along with generation and benchmarking scripts and lm-evaluation-harness instructions.

The code is intended for researchers and engineers who want to train or run SSM-based sequence models and compare them against Transformer baselines.

Key features

  • 01Selective state space layer with a hardware-aware scan implementation
  • 02Mamba, Mamba-2, and Mamba-3 block modules with a shared PyTorch interface
  • 03Complete language model wrapper combining Mamba blocks with an LM head
  • 04Pretrained checkpoints from 130M to 2.8B parameters on Hugging Face
  • 05Optional CUDA selective scan extension, with a pure Python core install
  • 06Generation benchmark script and zero-shot evaluation instructions

Use cases

  • Training long-sequence language models as an alternative to Transformers
  • Researching state space model architectures and their efficiency trade-offs
  • Running inference with pretrained Mamba checkpoints for text generation
  • Benchmarking SSM generation latency and throughput against Transformer baselines

Quick start

  1. 1

    Install PyTorch first

  2. 2

    Install the core mamba-ssm package

    pip install mamba-ssm --no-build-isolation

Things to know

  • Requires Linux and an NVIDIA GPU with CUDA 11.6+ for the CUDA selective scan path.
  • Core install omits the CUDA selective_scan_cuda extension unless you opt in with environment flags.
  • Installation needs your existing CUDA-enabled PyTorch, so setup can be fiddly.

Topics

#state-space-model#sequence-modeling#pytorch#language-model#deep-learning#cuda#transformer-alternative

FAQ

What is mamba?
Mamba is a PyTorch sequence-modeling architecture built on selective state space models, offering a linear-time alternative to Transformers for language and other sequence tasks. Mamba is a sequence modeling architecture based on selective state space models (SSMs). It targets information-dense data such as language, where earlier subquadratic models have trailed Transformers, and uses a hardware-aware implementation inspired by FlashAttention.
Is mamba free to use? What license does it have?
mamba is open source under the Apache-2.0 license. Check the license terms before using it in commercial products.
Is mamba still maintained?
Updated within 6 months. The last commit to mamba was on Jul 22, 2026.
What can I use mamba for?
Training long-sequence language models as an alternative to Transformers Researching state space model architectures and their efficiency trade-offs Running inference with pretrained Mamba checkpoints for text generation Benchmarking SSM generation latency and throughput against Transformer baselines
How do I install mamba?
Install PyTorch first Install the core mamba-ssm package: pip install mamba-ssm --no-build-isolation
Where can I get mamba?
The source code and installation instructions are on GitHub: https://github.com/state-spaces/mamba

Summary generated by AI from the project's README. GitHub stats refresh daily.

Related projects