Repolode
Submit
BlinkDL

BlinkDL/RWKV-LM

Official training and reference code for RWKV, an RNN-style language model architecture that can also be trained in parallel like a GPT transformer.

View on GitHub
Stars
14,757
Forks
1,025
Open issues
159
Last commit
23 days ago

What is it?

RWKV-LM is the main training and reference repository for RWKV, a language model architecture that is a recurrent neural network but can be trained in parallel like a transformer. The project is currently at RWKV-7 "Goose", and the README describes the design as attention-free, linear in time, and constant in memory, with no KV cache. Inference only carries a fixed-size hidden state from token to token, so memory use does not grow with context length.

The repository includes the RWKV-7 reference training implementation (train_temp), fast CUDA kernels, demo scripts for GPT-like, RNN and combined modes, and older RWKV-4, 5 and 6 code. It also documents data preparation from jsonl into binidx format, DeepSpeed and PyTorch Lightning training setups, and initialization and weight decay rules for reproducing stable training.

The linked ecosystem includes an RWKV pip package, GGUF weights, mobile and WebGPU inference projects, and fine-tuning tools such as PEFT and RLHF implementations maintained by other contributors.

Key features

  • 01Parallel GPT-style training with RNN-style constant-memory inference.
  • 02Provides the RWKV-7 reference training implementation and CUDA kernels.
  • 03Includes demo scripts for GPT, RNN and combined inference modes.
  • 04Documents data preparation from jsonl to binidx and DeepSpeed-based training.
  • 05Gives initialization and weight decay guidance for stable from-scratch training.

Use cases

  • Researching attention-free or linear-time sequence models.
  • Training a custom RWKV language model on a single GPU with the reference code.
  • Running long-context inference where KV cache memory is a constraint.
  • Exploring RNN-based alternatives to transformers for edge or on-device deployment.

Quick start

  1. 1

    Install PyTorch with CUDA 12.1 wheels

    pip install torch --upgrade --extra-index-url https://download.pytorch.org/whl/cu121
  2. 2

    Install training dependencies (keep pytorch-lightning at 1.9.5)

    pip install pytorch-lightning==1.9.5 deepspeed wandb ninja --upgrade
  3. 3

    Change into the RWKV-7 reference training directory

    cd RWKV-v7/train_temp/
  4. 4

    Download MiniPile data to train_temp/data, then run the preparation script

    sh ./demo-training-prepare.sh

Things to know

  • Training expects an NVIDIA CUDA GPU; the default config needs about 7 GB of VRAM.
  • The reference training code pins pytorch-lightning to 1.9.5 and relies on DeepSpeed and CUDA kernels.
  • The README says the FLA RWKV-7 implementation is not aligned with the reference and performs worse, and there is no simple drop-in RWKV-7 layer.

Topics

#rnn#language-model#linear-attention#pytorch#llm#deep-learning#transformer-alternative#attention-mechanism#chatgpt#gpt

FAQ

What is RWKV-LM?
Official training and reference code for RWKV, an RNN-style language model architecture that can also be trained in parallel like a GPT transformer. RWKV-LM is the main training and reference repository for RWKV, a language model architecture that is a recurrent neural network but can be trained in parallel like a transformer. The project is currently at RWKV-7 "Goose", and the README describes the design as attention-free, linear in time, and constant in memory, with no KV cache. Inference only carries a fixed-size hidden state from token to token, so memory use does not grow with context length.
Is RWKV-LM free to use? What license does it have?
RWKV-LM is open source under the Apache-2.0 license. Check the license terms before using it in commercial products.
Is RWKV-LM still maintained?
Actively maintained. The last commit to RWKV-LM was on Sep 28, 2026.
What can I use RWKV-LM for?
Researching attention-free or linear-time sequence models. Training a custom RWKV language model on a single GPU with the reference code. Running long-context inference where KV cache memory is a constraint. Exploring RNN-based alternatives to transformers for edge or on-device deployment.
How do I install RWKV-LM?
Install PyTorch with CUDA 12.1 wheels: pip install torch --upgrade --extra-index-url https://download.pytorch.org/whl/cu121 Install training dependencies (keep pytorch-lightning at 1.9.5): pip install pytorch-lightning==1.9.5 deepspeed wandb ninja --upgrade Change into the RWKV-7 reference training directory: cd RWKV-v7/train_temp/ Download MiniPile data to train_temp/data, then run the preparation script: sh ./demo-training-prepare.sh
Where can I get RWKV-LM?
The source code and installation instructions are on GitHub: https://github.com/BlinkDL/RWKV-LM

Summary generated by AI from the project's README. GitHub stats refresh daily.

Related projects