Repolode
Submit
NVIDIA-NeMo

NVIDIA-NeMo/Curator

GPU-accelerated toolkit for building repeatable pipelines that filter, deduplicate and transform large text, image, video and audio datasets for AI training.

View on GitHub
Stars
1,818
Forks
351
Open issues
297
Last commit
6 hours ago

What is it?

NeMo Curator is a Python toolkit for preparing large datasets used to train AI models. It lets ML engineers and data teams assemble pipelines from stages that load, filter, deduplicate, classify and write data, with separate building blocks for text, image, video and audio.

Pipelines are built from stages that declare their own CPU, GPU and replica needs, and a pluggable executor scales them across resources. The same pipeline definition can run on a single machine or on a multi-node Ray cluster, including Slurm-managed HPC environments. The project documents an Xenna executor as the production default, with Ray-based backends available as experimental options.

The toolkit is part of the NVIDIA NeMo suite and is used in the data pipelines described for Nemotron and Nemotron-CC datasets. It is installed through uv or Docker, with a CUDA 12 path for GPU text workloads on Linux.

Key features

  • 01Modular stages for loading, filtering, deduplicating, classifying and writing data.
  • 02GPU-accelerated deduplication and classification using RAPIDS.
  • 03Runs the same pipeline on a laptop or a multi-node Ray cluster.
  • 04Modality plug-ins for text, image, video and audio.
  • 05Includes an OpenAI-compatible inference server for synthetic data workflows.

Use cases

  • Deduplicating and quality-filtering web text before LLM pre-training.
  • Filtering and embedding large image collections for training datasets.
  • Preparing video clips with scene detection and motion filtering.
  • Transcribing and filtering speech data for audio model training.

Quick start

  1. 1

    Install the uv package manager (the README's install script is piped to sh, so no command is given here)

  2. 2

    Create a virtual environment and install the CPU text extras

    uv venv && source .venv/bin/activate uv pip install "nemo-curator[text_cpu]"
  3. 3

    Verify the installation by printing the version

    python -c "import nemo_curator; print(nemo_curator.__version__)"

Things to know

  • The GPU text path requires CUDA 12 on Linux x86_64 and roughly 16 GB of GPU memory.
  • Standard pip install is not supported for the CUDA 12 text extra; it needs a special uv override file.
  • Audio and video workflows need FFmpeg, which is best handled with a custom container build.
  • NVIDIA no longer publishes new NeMo Curator container images on NGC starting with 26.09.

Topics

#data-curation#llm#data-processing#deduplication#gpu#ray#data-quality#python#data#data-prep

FAQ

What is Curator?
GPU-accelerated toolkit for building repeatable pipelines that filter, deduplicate and transform large text, image, video and audio datasets for AI training. NeMo Curator is a Python toolkit for preparing large datasets used to train AI models. It lets ML engineers and data teams assemble pipelines from stages that load, filter, deduplicate, classify and write data, with separate building blocks for text, image, video and audio.
Is Curator free to use? What license does it have?
Curator is open source under the Apache-2.0 license. Check the license terms before using it in commercial products.
Is Curator still maintained?
Actively maintained. The last commit to Curator was on Oct 21, 2026.
What can I use Curator for?
Deduplicating and quality-filtering web text before LLM pre-training. Filtering and embedding large image collections for training datasets. Preparing video clips with scene detection and motion filtering. Transcribing and filtering speech data for audio model training.
How do I install Curator?
Install the uv package manager (the README's install script is piped to sh, so no command is given here) Create a virtual environment and install the CPU text extras: uv venv && source .venv/bin/activate uv pip install "nemo-curator[text_cpu]" Verify the installation by printing the version: python -c "import nemo_curator; print(nemo_curator.__version__)"
Where can I get Curator?
The source code and installation instructions are on GitHub: https://github.com/NVIDIA-NeMo/Curator

Summary generated by AI from the project's README. GitHub stats refresh daily.

Related projects