Repolode
Submit
Lightning-AI

Lightning-AI/litData

Python library that optimizes datasets into a chunked format for fast streaming from cloud or local storage during model training.

View on GitHub
Stars
617
Forks
111
Open issues
53
Last commit
6 days ago

What is it?

LitData addresses slow model training caused by data loading. Large datasets, millions of small files, or slow cloud reads can leave GPUs waiting, and LitData provides tools to preprocess datasets into its own chunked binary format that streams efficiently from S3, GCS, Azure, R2, Hugging Face Hub, or local disk.

The library offers several paths. Raw files such as images, audio, or JSONL can be streamed directly with StreamingRawDataset and a normal PyTorch DataLoader, with no conversion step. For maximum throughput, optimize() converts data into chunks that StreamingDataset and StreamingDataLoader read, supporting shuffling, drop_last, and resumable streaming. A map() operator distributes general processing jobs such as resizing images, creating embeddings, or running inference across multiple machines.

LitData also includes helpers for Hugging Face datasets, typed media wrappers for images, audio, video and PDFs, custom serializers, and PyTorch Geometric graph support.

Key features

  • 01Converts datasets into a chunked binary format optimized for streaming.
  • 02Streams from S3, GCS, Azure, R2, Hugging Face Hub, and local paths.
  • 03Streams raw files such as images or audio without a preprocessing step.
  • 04Supports pausing and resuming data streaming mid-epoch.
  • 05Provides a map operator for parallel processing across machines.

Use cases

  • Training on a dataset too large to fit on local disk, streamed from cloud storage.
  • Replacing millions of small image files with chunked data to reduce I/O overhead.
  • Distributed preprocessing such as image resizing or embedding generation before training.
  • Loading Hugging Face datasets in a faster format for repeated training runs.

Quick start

  1. 1

    Install LitData with pip

    pip install litdata
  2. 2

    Stream existing files directly with StreamingRawDataset, or optimize datasets into chunks for StreamingDataset

Things to know

  • Needs PyTorch 2.4 or newer.
  • Optimizing data requires a one-time preprocessing step before the fastest streaming.

Topics

#pytorch#data-loading#streaming#deep-learning#machine-learning#cloud-storage#data-processing

FAQ

What is litData?
Python library that optimizes datasets into a chunked format for fast streaming from cloud or local storage during model training. LitData addresses slow model training caused by data loading. Large datasets, millions of small files, or slow cloud reads can leave GPUs waiting, and LitData provides tools to preprocess datasets into its own chunked binary format that streams efficiently from S3, GCS, Azure, R2, Hugging Face Hub, or local disk.
Is litData free to use? What license does it have?
litData is open source under the Apache-2.0 license. Check the license terms before using it in commercial products.
Is litData still maintained?
Actively maintained. The last commit to litData was on Oct 15, 2026.
What can I use litData for?
Training on a dataset too large to fit on local disk, streamed from cloud storage. Replacing millions of small image files with chunked data to reduce I/O overhead. Distributed preprocessing such as image resizing or embedding generation before training. Loading Hugging Face datasets in a faster format for repeated training runs.
How do I install litData?
Install LitData with pip: pip install litdata Stream existing files directly with StreamingRawDataset, or optimize datasets into chunks for StreamingDataset
Where can I get litData?
The source code and installation instructions are on GitHub: https://github.com/Lightning-AI/litData

Summary generated by AI from the project's README. GitHub stats refresh daily.

Related projects