Repolode
Submit
OpenDCAI

OpenDCAI/DataFlow

A Python framework for generating, cleaning and preparing LLM training data using composable operators and pipelines, with a visual WebUI.

View on GitHubWebsite
Stars
8,249
Forks
1,200
Open issues
30
Last commit
3 days ago

What is it?

DataFlow is a data preparation system for large language models. It turns noisy sources such as PDFs, plain text and low-quality question-answer pairs into cleaner datasets for pre-training, supervised fine-tuning, reinforcement learning or retrieval-augmented generation. Work is organized as operators (single processing steps such as generation, evaluation, filtering or refinement) that are chained into reusable, shareable pipelines.

The project ships ready-made pipelines for text question-answer mining, reasoning enhancement, Text2SQL, knowledge base cleaning and agentic RAG data, along with an agent that can assemble pipelines from existing or newly written operators. Operators accept JSON, JSONL and CSV inputs and can run against API-based LLMs or local inference through vLLM. Installation is via pip (open-dataflow), Docker, or Google Colab, and the dataflow webui command launches a browser-based visual pipeline builder.

The repository also includes DataFlow-Instruct-10K, an instruction dataset produced with the framework, and benchmark tables comparing models fine-tuned on it against other datasets.

Key features

  • 01Operator-based design where each processing step is a reusable unit that can be chained into pipelines.
  • 02Built-in pipelines for text QA mining, reasoning enhancement, Text2SQL, knowledge base cleaning and agentic RAG.
  • 03Supports JSON, JSONL and CSV inputs and outputs.
  • 04Works with API-based LLMs and local inference via vLLM.
  • 05Browser-based visual WebUI launched with a single command.
  • 06Custom operators can be packaged and distributed through GitHub or PyPI.

Use cases

  • Turning raw PDFs or web text into question-answer pairs for supervised fine-tuning.
  • Generating and filtering instruction or reasoning datasets for model training.
  • Building Text2SQL training data with schema context and explanations.
  • Preparing cleaned knowledge base entries for retrieval-augmented generation.

Quick start

  1. 1

    Install the open-dataflow package from PyPI

  2. 2

    Launch the visual pipeline WebUI

    dataflow webui

Things to know

  • Many operators call LLM APIs or large models, so you need model access to run them.

Topics

#llm#data-cleaning#data-synthesis#data-pipelines#training-data#vllm#python#data#data-agent#data-processing

FAQ

What is DataFlow?
A Python framework for generating, cleaning and preparing LLM training data using composable operators and pipelines, with a visual WebUI. DataFlow is a data preparation system for large language models. It turns noisy sources such as PDFs, plain text and low-quality question-answer pairs into cleaner datasets for pre-training, supervised fine-tuning, reinforcement learning or retrieval-augmented generation. Work is organized as operators (single processing steps such as generation, evaluation, filtering or refinement) that are chained into reusable, shareable pipelines.
Is DataFlow free to use? What license does it have?
DataFlow is open source under the Apache-2.0 license. Check the license terms before using it in commercial products.
Is DataFlow still maintained?
Actively maintained. The last commit to DataFlow was on Oct 19, 2026.
What can I use DataFlow for?
Turning raw PDFs or web text into question-answer pairs for supervised fine-tuning. Generating and filtering instruction or reasoning datasets for model training. Building Text2SQL training data with schema context and explanations. Preparing cleaned knowledge base entries for retrieval-augmented generation.
How do I install DataFlow?
Install the open-dataflow package from PyPI Launch the visual pipeline WebUI: dataflow webui
Where can I get DataFlow?
The source code and installation instructions are on GitHub: https://github.com/OpenDCAI/DataFlow

Summary generated by AI from the project's README. GitHub stats refresh daily.

Related projects