mit-han-lab/streaming-llm

What is it?

Key features

• 01Keeps initial attention-sink tokens plus a recent-token window in the KV cache.
• 02Lets models run on inputs far longer than their training length without fine-tuning.
• 03Includes a streaming Llama chatbot demo script.
• 04Provides perplexity evaluation code.
• 05Supports Llama-2, MPT, Falcon, and Pythia.

Use cases

• Running a long-lived chatbot or daily assistant that must keep responding across many conversation turns.
• Studying attention sink behavior and KV cache management in LLM research.
• Reproducing the perplexity results from the StreamingLLM paper.

Quick start

1. 1

   Create and activate a conda environment with Python 3.8

   conda create -yn streaming python=3.8
2. 2

   Activate the environment

   conda activate streaming
3. 3

   Install PyTorch and the pinned Python dependencies

   pip install torch torchvision torchaudio
4. 4

   Run the streaming Llama chatbot demo on GPU 0

   CUDA_VISIBLE_DEVICES=0 python examples/run_streaming_llama.py --enable_streaming

Things to know

• !Setup requires a specific older Python version (3.8) and a pinned transformers version (4.33.0).
• !Requires a GPU for the demo and runs on pretrained models such as Llama-2 that you must obtain separately.
• !The model only attends to recent tokens and attention sinks, so it does not expand context or give long-term memory.
• !The repository's last push was in July 2024, so maintenance is limited.

In collections

Topics

FAQ

What is streaming-llm?

Research code for StreamingLLM, a method that lets pretrained LLMs process effectively unbounded text streams by keeping attention sink tokens plus a recent-token window. StreamingLLM is a research project from MIT Han Lab, accompanying the ICLR 2024 paper on efficient streaming language models with attention sinks. It addresses two problems in long multi-round LLM use: the memory cost of caching every past Key and Value state during decoding, and the degraded output of models once inputs exceed their training sequence length.

Is streaming-llm free to use? What license does it have?

streaming-llm is open source under the MIT license. Check the license terms before using it in commercial products.

Is streaming-llm still maintained?

No commits for 6+ months. The last commit to streaming-llm was on Jul 11, 2024.

What can I use streaming-llm for?

Running a long-lived chatbot or daily assistant that must keep responding across many conversation turns. Studying attention sink behavior and KV cache management in LLM research. Reproducing the perplexity results from the StreamingLLM paper.

How do I install streaming-llm?

Create and activate a conda environment with Python 3.8: conda create -yn streaming python=3.8 Activate the environment: conda activate streaming Install PyTorch and the pinned Python dependencies: pip install torch torchvision torchaudio Run the streaming Llama chatbot demo on GPU 0: CUDA_VISIBLE_DEVICES=0 python examples/run_streaming_llama.py --enable_streaming

Where can I get streaming-llm?

The source code and installation instructions are on GitHub: https://github.com/mit-han-lab/streaming-llm