Repolode
Submit
feyninc

feyninc/chonkie

A lightweight Python chunking library that splits text into chunks for RAG pipelines, with multiple chunkers, refiners and vector database integrations.

View on GitHubWebsite
Stars
4,801
Forks
364
Open issues
49
Last commit
2 days ago

What is it?

Chonkie is a Python library for splitting text into chunks, the preprocessing step that retrieval-augmented generation (RAG) pipelines need before embedding documents. It offers a range of chunkers, from fixed token-size splitting and sentence-based splitting to recursive, semantic, code and LLM-driven approaches, all callable through a simple interface.

Beyond chunking, Chonkie includes refineries that add overlap or embeddings to chunks, porters that export chunks to JSON or Hugging Face datasets, and handshakes that write chunks directly into vector stores such as Chroma, Qdrant, Pinecone, pgvector and Weaviate. A Pipeline API chains these steps together, with sync and async execution. The core install is kept small, and optional extras are added per feature.

Chonkie can also run as a self-hosted REST API through a `chonkie serve` command or Docker, and it ships an agent skill for AI coding assistants.

Key features

  • 01Multiple chunking strategies: token, sentence, recursive, semantic, code and LLM-based.
  • 02Refineries add context overlap or embeddings to chunks.
  • 03Porters export chunks to JSON or Hugging Face datasets.
  • 04Handshakes write chunks into vector databases such as Qdrant, Pinecone and pgvector.
  • 05Pipeline API supports chaining steps with sync and async runs.
  • 06Self-hosted REST API via CLI or Docker.

Use cases

  • Preparing documents for a RAG pipeline before embedding and storing them in a vector database.
  • Comparing different chunking strategies on the same corpus through one interface.
  • Chunking source code or markdown tables with structure-aware chunkers.
  • Running a shared chunking service for multiple applications via the REST API.

Quick start

  1. 1

    Install Chonkie with pip

    pip install chonkie
  2. 2

    Import a chunker, create it, and call it on your text

Things to know

  • Optional integrations and chunkers need extra installs such as chonkie[semantic] or chonkie[tiktoken].
  • The all-in-one install is marked as not recommended for production environments.

In collections

Natural language processing tools & libraries

Topics

#rag#chunking#text-splitter#embeddings#vector-database#llm#nlp#ai#chonkie#chunker

FAQ

What is chonkie?
A lightweight Python chunking library that splits text into chunks for RAG pipelines, with multiple chunkers, refiners and vector database integrations. Chonkie is a Python library for splitting text into chunks, the preprocessing step that retrieval-augmented generation (RAG) pipelines need before embedding documents. It offers a range of chunkers, from fixed token-size splitting and sentence-based splitting to recursive, semantic, code and LLM-driven approaches, all callable through a simple interface.
Is chonkie free to use? What license does it have?
chonkie is open source under the MIT license. Check the license terms before using it in commercial products.
Is chonkie still maintained?
Actively maintained. The last commit to chonkie was on Oct 20, 2026.
What can I use chonkie for?
Preparing documents for a RAG pipeline before embedding and storing them in a vector database. Comparing different chunking strategies on the same corpus through one interface. Chunking source code or markdown tables with structure-aware chunkers. Running a shared chunking service for multiple applications via the REST API.
How do I install chonkie?
Install Chonkie with pip: pip install chonkie Import a chunker, create it, and call it on your text
Where can I get chonkie?
The source code and installation instructions are on GitHub: https://github.com/feyninc/chonkie

Summary generated by AI from the project's README. GitHub stats refresh daily.

Related projects