Repolode
Submit
microsoft

microsoft/LLMLingua

A Python library from Microsoft that compresses LLM prompts and KV cache by up to 20x by dropping non-essential tokens, with minimal loss in performance.

View on GitHubWebsite
Stars
6,767
Forks
444
Open issues
125
Last commit
last month

What is it?

LLMLingua is a prompt compression toolkit from Microsoft Research for large language models. It uses a small, trained language model to find and remove tokens that contribute little to meaning, so long prompts become shorter while the key information is kept. The project claims compression of up to 20x with minimal performance loss, and it also covers KV cache compression to speed up inference.

The repository spans a series of methods. LLMLingua is the original approach, LongLLMLingua targets long-context and retrieval-augmented (RAG) scenarios and addresses the lost-in-the-middle problem, and LLMLingua-2 is a task-agnostic compressor built as a token classifier with a BERT-level encoder. A SecurityLingua variant is also included for detecting jailbreak intent. Structured prompt compression lets users tag sections of a prompt with different compression rates.

Usage is through a Python package installed with pip, exposing a PromptCompressor class with compress_prompt and structured_compress_prompt methods. The project has been integrated into Prompt flow, LangChain and LlamaIndex.

Key features

  • 01Removes non-essential tokens from prompts to shorten them, with compression up to 20x.
  • 02Compresses KV cache to speed up LLM inference.
  • 03Provides LongLLMLingua for long-context and RAG prompts.
  • 04Provides LLMLingua-2, a faster task-agnostic compressor based on a BERT-level encoder.
  • 05Supports structured compression with per-section rates via tags.
  • 06Installable with pip and usable from Python.

Use cases

  • Fitting long documents or retrieved passages into a limited LLM context window in RAG pipelines.
  • Reducing API costs for applications that send long prompts to GPT-4 or similar models.
  • Compressing chain-of-thought or few-shot examples while preserving reasoning content.
  • Detecting jailbreak intent in user prompts with the SecurityLingua model.

Quick start

  1. 1

    Install LLMLingua from PyPI

    pip install llmlingua
  2. 2

    Import PromptCompressor and compress a prompt with the default model

Things to know

  • Compression quality depends on the chosen model, and some variants need a GPU for good speed.
  • Some model options require extra packages such as optimum and auto-gptq for quantized models.

Topics

#llm#prompt-compression#rag#kv-cache#inference-optimization#nlp#python

FAQ

What is LLMLingua?
A Python library from Microsoft that compresses LLM prompts and KV cache by up to 20x by dropping non-essential tokens, with minimal loss in performance. LLMLingua is a prompt compression toolkit from Microsoft Research for large language models. It uses a small, trained language model to find and remove tokens that contribute little to meaning, so long prompts become shorter while the key information is kept. The project claims compression of up to 20x with minimal performance loss, and it also covers KV cache compression to speed up inference.
Is LLMLingua free to use? What license does it have?
LLMLingua is open source under the MIT license. Check the license terms before using it in commercial products.
Is LLMLingua still maintained?
Updated within 6 months. The last commit to LLMLingua was on Sep 10, 2026.
What can I use LLMLingua for?
Fitting long documents or retrieved passages into a limited LLM context window in RAG pipelines. Reducing API costs for applications that send long prompts to GPT-4 or similar models. Compressing chain-of-thought or few-shot examples while preserving reasoning content. Detecting jailbreak intent in user prompts with the SecurityLingua model.
How do I install LLMLingua?
Install LLMLingua from PyPI: pip install llmlingua Import PromptCompressor and compress a prompt with the default model
Where can I get LLMLingua?
The source code and installation instructions are on GitHub: https://github.com/microsoft/LLMLingua

Summary generated by AI from the project's README. GitHub stats refresh daily.

Related projects