A Python library from Microsoft that compresses LLM prompts and KV cache by up to 20x by dropping non-essential tokens, with minimal loss in performance.
6.8K444PythonMITUpdated last month
#inference-optimization
Open-source projects tagged “inference-optimization”
Curated GitHub repositories related to inference-optimization, with AI summaries and key stats.