Repolode
Submit
ucbepic

ucbepic/docetl

A Python system for running LLM-powered map-reduce and ETL pipelines over large structured and unstructured data, described in natural language.

View on GitHubWebsite
Stars
4,137
Forks
452
Open issues
47
Last commit
12 days ago

What is it?

DocETL is a system for processing large collections of structured and unstructured data with LLMs. Each operation is written in natural language, and the system provides operators such as map, reduce, filter, resolve, split, and gather, then orchestrates them and parallelizes the work across the dataset. The output is tabular data that can be queried in a database.

The pipeline can be written in three ways: a Python API, a YAML configuration file run through the docetl CLI, or the DocWrangler web UI for interactive prompt development. The project also includes an optimizer, MOAR, that automatically swaps models, rewrites prompts, decomposes operations, and replaces subtasks with code to improve accuracy and reduce cost.

DocETL works with any LLM provider whose API key is set in the environment. It is aimed at teams that would otherwise hand-write and tune chains of LLM calls for document analysis and similar data tasks.

Key features

  • 01Map, reduce, filter, resolve, split and gather operators for LLM-based data processing.
  • 02Natural-language operation definitions with output schemas.
  • 03Automatic pipeline optimization that swaps models and rewrites prompts.
  • 04Python API, YAML configuration with a CLI, and a DocWrangler web UI.
  • 05Parallel execution across the dataset with rate limit controls.

Use cases

  • Extracting structured fields such as complaints or priorities from large batches of support tickets.
  • Summarizing documents grouped by category with a map-then-reduce workflow.
  • Analyzing transcripts or unstructured text and returning results as tables for querying.

Quick start

  1. 1

    Install the package with pip

    pip install docetl
  2. 2

    Set your LLM provider API key (for example OpenAI)

    export OPENAI_API_KEY=your_key
  3. 3

    Write a pipeline YAML file and run it with the CLI

    docetl run pipeline.yaml

Things to know

  • Requires an API key from an LLM provider, which incurs usage costs.

Topics

#llm#etl#data-pipelines#document-processing#map-reduce#unstructured-data#agents#data#document-analysis#elt

FAQ

What is docetl?
A Python system for running LLM-powered map-reduce and ETL pipelines over large structured and unstructured data, described in natural language. DocETL is a system for processing large collections of structured and unstructured data with LLMs. Each operation is written in natural language, and the system provides operators such as map, reduce, filter, resolve, split, and gather, then orchestrates them and parallelizes the work across the dataset. The output is tabular data that can be queried in a database.
Is docetl free to use? What license does it have?
docetl is open source under the MIT license. Check the license terms before using it in commercial products.
Is docetl still maintained?
Actively maintained. The last commit to docetl was on Oct 9, 2026.
What can I use docetl for?
Extracting structured fields such as complaints or priorities from large batches of support tickets. Summarizing documents grouped by category with a map-then-reduce workflow. Analyzing transcripts or unstructured text and returning results as tables for querying.
How do I install docetl?
Install the package with pip: pip install docetl Set your LLM provider API key (for example OpenAI): export OPENAI_API_KEY=your_key Write a pipeline YAML file and run it with the CLI: docetl run pipeline.yaml
Where can I get docetl?
The source code and installation instructions are on GitHub: https://github.com/ucbepic/docetl

Summary generated by AI from the project's README. GitHub stats refresh daily.

Related projects