Repolode
Submit
zilliztech

zilliztech/GPTCache

A semantic cache library for LLM queries that stores responses and reuses them for similar questions to cut API costs and latency.

View on GitHubWebsite
Stars
8,213
Forks
597
Open issues
100
Last commit
last month

What is it?

GPTCache is a Python library that caches responses from large language model APIs. Instead of only matching identical prompts, it converts queries into embeddings and searches a vector store for semantically similar earlier questions, returning a stored answer when a close match is found. This reduces repeated API calls, token spending and response time for applications that receive many overlapping requests.

The library is modular. Each part can be swapped: LLM adapters (OpenAI, LangChain, Llama.cpp, Dolly and others), embedding generators (ONNX, OpenAI, Cohere, Hugging Face, SentenceTransformers and others), cache storage (SQLite, PostgreSQL, MySQL, DuckDB, DynamoDB and others), vector stores (Milvus, Zilliz Cloud, FAISS, Chroma, PGVector, Qdrant, Weaviate and others), and eviction policies such as LRU, LFU and FIFO. A server Docker image is also provided so applications in any language can use the cache over the network.

The project is under active development and its API may change between releases. The README notes that new LLM API shapes are no longer being added to the adapters, and recommends the get and set API for custom integrations.

Key features

  • 01Semantic matching of queries via embeddings and vector search, not just exact string matches.
  • 02Pluggable LLM adapters, including OpenAI ChatCompletion and LangChain integration.
  • 03Multiple storage backends such as SQLite, PostgreSQL, MySQL and DynamoDB.
  • 04Support for vector stores including Milvus, FAISS, Chroma, Qdrant and Weaviate.
  • 05Distributed caching through Redis or memcached for horizontally scaled deployments.
  • 06A server Docker image that lets applications in any language use the cache.

Use cases

  • Reducing OpenAI or other LLM API costs in a chatbot that receives many similar user questions.
  • Speeding up responses in a LangChain or LlamaIndex application by reusing earlier answers.
  • Testing LLM applications locally without calling the live model on every run.
  • Sharing one semantic cache across several service replicas with Redis.

Quick start

  1. 1

    Install the package from PyPI

    pip install gptcache
  2. 2

    Initialize the cache in your existing OpenAI code

Things to know

  • The API may change at any time because the project is under swift development.
  • The project no longer adds support for new LLM APIs or models; the README recommends its get and set API instead.
  • Requires Python 3.8.1 or higher.

Topics

#llm#semantic-cache#openai#vector-search#embedding#langchain#cost-optimization#aigc#autogpt#babyagi

FAQ

What is GPTCache?
A semantic cache library for LLM queries that stores responses and reuses them for similar questions to cut API costs and latency. GPTCache is a Python library that caches responses from large language model APIs. Instead of only matching identical prompts, it converts queries into embeddings and searches a vector store for semantically similar earlier questions, returning a stored answer when a close match is found. This reduces repeated API calls, token spending and response time for applications that receive many overlapping requests.
Is GPTCache free to use? What license does it have?
GPTCache is open source under the MIT license. Check the license terms before using it in commercial products.
Is GPTCache still maintained?
Updated within 6 months. The last commit to GPTCache was on Sep 22, 2026.
What can I use GPTCache for?
Reducing OpenAI or other LLM API costs in a chatbot that receives many similar user questions. Speeding up responses in a LangChain or LlamaIndex application by reusing earlier answers. Testing LLM applications locally without calling the live model on every run. Sharing one semantic cache across several service replicas with Redis.
How do I install GPTCache?
Install the package from PyPI: pip install gptcache Initialize the cache in your existing OpenAI code
Where can I get GPTCache?
The source code and installation instructions are on GitHub: https://github.com/zilliztech/GPTCache

Summary generated by AI from the project's README. GitHub stats refresh daily.

Related projects