Repolode
Submit
datajuicer

datajuicer/data-juicer

A data processing system with 200+ composable operators for cleaning, filtering, deduplicating and synthesizing text, image, audio and video data for foundation model training.

View on GitHubWebsite
Stars
7,176
Forks
437
Open issues
65
Last commit
6 hours ago

What is it?

Data-Juicer is a data processing framework for preparing training and retrieval data for large language models and other foundation models. It provides modular operators for tasks such as cleaning, filtering, deduplication, synthetic data generation and profiling, covering text, image, audio, video and multimodal inputs.

Pipelines are defined as YAML recipes that can be versioned and shared, and can also be composed directly in Python by chaining operators on a dataset. The same pipeline can run on a single machine or scale out on Ray clusters, with partitioned execution, checkpoints and export to local, S3 or HDFS storage.

The project also includes analysis and visualization tools for dataset statistics, tracing of changed samples, and an operator plugin mechanism for third-party extensions.

Key features

  • 01200+ operators for text, image, audio, video and multimodal data
  • 02YAML recipes for reproducible, shareable pipelines
  • 03Python API for composing operators on datasets
  • 04Ray-based partitioned execution for large-scale processing
  • 05Dataset analysis, visualization and sample tracing
  • 06Local, S3 and HDFS loading and export

Use cases

  • Deduplicating and filtering web-scale pre-training corpora
  • Preparing instruction-tuning or synthetic datasets for LLMs
  • Building RAG indices with extraction, chunking and cleaning
  • Curating video or image datasets for multimodal model training

Quick start

  1. 1

    Install the package with uv

    uv pip install py-data-juicer
  2. 2

    Run the example pipeline from the config

    dj-process --config demos/process_simple/process.yaml

Things to know

  • Some advanced features such as GPU acceleration and large-scale Ray execution need extra setup and hardware.

Topics

#data-processing#llm#foundation-models#data-pipeline#multi-modal#ray#synthetic-data#deduplication#data#data-analysis

FAQ

What is data-juicer?
A data processing system with 200+ composable operators for cleaning, filtering, deduplicating and synthesizing text, image, audio and video data for foundation model training. Data-Juicer is a data processing framework for preparing training and retrieval data for large language models and other foundation models. It provides modular operators for tasks such as cleaning, filtering, deduplication, synthetic data generation and profiling, covering text, image, audio, video and multimodal inputs.
Is data-juicer free to use? What license does it have?
data-juicer is open source under the Apache-2.0 license. Check the license terms before using it in commercial products.
Is data-juicer still maintained?
Actively maintained. The last commit to data-juicer was on Oct 21, 2026.
What can I use data-juicer for?
Deduplicating and filtering web-scale pre-training corpora Preparing instruction-tuning or synthetic datasets for LLMs Building RAG indices with extraction, chunking and cleaning Curating video or image datasets for multimodal model training
How do I install data-juicer?
Install the package with uv: uv pip install py-data-juicer Run the example pipeline from the config: dj-process --config demos/process_simple/process.yaml
Where can I get data-juicer?
The source code and installation instructions are on GitHub: https://github.com/datajuicer/data-juicer

Summary generated by AI from the project's README. GitHub stats refresh daily.

Related projects