Repolode
Submit
aryn-ai

aryn-ai/sycamore

Python framework for ETL, RAG and analytics on unstructured documents, with AI-powered partitioning, chunking and loading into vector and hybrid search databases.

View on GitHubWebsite
Stars
605
Forks
71
Open issues
56
Last commit
3 months ago

What is it?

Sycamore is a document processing engine for turning unstructured files such as PDFs, reports, presentations, transcripts and manuals into clean, chunked data. It handles complex layouts including embedded tables, figures and infographics, and it can apply LLM-based enrichment, transforms and cleaning before the data is stored.

Processing is organized around the DocSet abstraction, a functional, Python-based pipeline that runs on a scalable Ray backend. Document partitioning can use the hosted Aryn DocParse service or a locally run partitioner, and users choose which LLMs power the transforms.

The project provides connectors for loading results into vector and hybrid search stores such as OpenSearch, Elasticsearch, Pinecone, DuckDB, Qdrant and Weaviate, installable as pip extras. It supports Linux and macOS.

Key features

  • 01Partitions PDFs and images with tables, figures and infographics using a layout-detection model
  • 02DocSet abstraction for scalable, functional transformations of documents
  • 03LLM-powered user-defined functions for extraction, enrichment and cleaning
  • 04Loads chunks into OpenSearch, Elasticsearch, Pinecone, DuckDB, Qdrant and Weaviate
  • 05Scales processing on a Ray backend

Use cases

  • Preparing PDF reports and manuals for a RAG application
  • Building a hybrid search index over enterprise documents in OpenSearch
  • Extracting structured fields from transcripts or presentations with LLMs
  • Running document ETL jobs in a Jupyter notebook before loading a vector database

Quick start

  1. 1

    Install the core package with pip

    pip install sycamore-ai
  2. 2

    Optionally install a vector database connector, e.g. DuckDB

    pip install sycamore-ai[duckdb]
  3. 3

    Sign up for a free Aryn DocParse account and get an API key

Things to know

  • Runs on Linux and Mac OS according to the README.
  • Using the Aryn DocParse service requires signing up for an API key.

Topics

#rag#document-processing#etl#vector-database#llm#unstructured-data#semantic-search#ai#dataprep#information-retrieval

FAQ

What is sycamore?
Python framework for ETL, RAG and analytics on unstructured documents, with AI-powered partitioning, chunking and loading into vector and hybrid search databases. Sycamore is a document processing engine for turning unstructured files such as PDFs, reports, presentations, transcripts and manuals into clean, chunked data. It handles complex layouts including embedded tables, figures and infographics, and it can apply LLM-based enrichment, transforms and cleaning before the data is stored.
Is sycamore free to use? What license does it have?
sycamore is open source under the Apache-2.0 license. Check the license terms before using it in commercial products.
Is sycamore still maintained?
Updated within 6 months. The last commit to sycamore was on Aug 1, 2026.
What can I use sycamore for?
Preparing PDF reports and manuals for a RAG application Building a hybrid search index over enterprise documents in OpenSearch Extracting structured fields from transcripts or presentations with LLMs Running document ETL jobs in a Jupyter notebook before loading a vector database
How do I install sycamore?
Install the core package with pip: pip install sycamore-ai Optionally install a vector database connector, e.g. DuckDB: pip install sycamore-ai[duckdb] Sign up for a free Aryn DocParse account and get an API key
Where can I get sycamore?
The source code and installation instructions are on GitHub: https://github.com/aryn-ai/sycamore

Summary generated by AI from the project's README. GitHub stats refresh daily.

Related projects