Repolode
Submit
huggingface

huggingface/datasets

A Python library from Hugging Face for loading public AI datasets from the Hub with one line and preprocessing them efficiently with Arrow-backed tools.

View on GitHubWebsite
Stars
22,076
Forks
3,549
Open issues
1,529
Last commit
2 days ago

What is it?

Datasets is a lightweight Python library with two main jobs. It provides one-line loaders for many public datasets hosted on the Hugging Face Hub, covering text, image, audio, video, PDF and NIfTI medical data, and returns them ready for NumPy, Pandas, PyTorch, TensorFlow, JAX or Polars workflows. It also offers preprocessing tools for both Hub datasets and local files in formats such as CSV, JSON, Parquet, XML, WAV, JPEG and PDF.

Datasets are stored with Apache Arrow and memory mapping, so large data does not need to fit in RAM. Operations like map() can run in parallel with num_proc, and processed results are cached and reused. Streaming mode iterates over remote data without a full download, which suits datasets larger than local disk.

The library also supports reading and writing Hugging Face Storage Buckets, loading AI agent traces, and building FAISS or Elasticsearch indexes for similarity search.

Key features

  • 01Load Hub datasets with a single load_dataset() call
  • 02Streaming mode iterates over remote data without a full download
  • 03Apache Arrow memory-mapped storage keeps large datasets out of RAM
  • 04Parallel map() processing with automatic caching of results
  • 05Reads CSV, JSON, Parquet, XML, text and multimedia files, including audio, image, video, PDF and NIfTI
  • 06Converts to NumPy, Pandas, Polars, PyTorch, TensorFlow and JAX formats

Use cases

  • Fine-tuning or evaluating an NLP model on a public benchmark dataset
  • Streaming a very large image or speech corpus during training
  • Preprocessing and tokenizing local CSV or Parquet data for model training
  • Building a similarity search index over embeddings with FAISS

Quick start

  1. 1

    Install the library with pip

    pip install datasets
  2. 2

    Load a dataset from the Hub in Python

    python -c "from datasets import load_dataset; squad_dataset = load_dataset('rajpurkar/squad'); print(squad_dataset['train'][0])"

Things to know

  • Some features need optional extras, such as audio, vision, PDF or NIfTI support.
  • Many datasets on the Hub are provided by third parties, so licenses and terms vary per dataset.

Topics

#machine-learning#nlp#datasets#data-preprocessing#pytorch#apache-arrow#deep-learning#huggingface#ai#artificial-intelligence

FAQ

What is datasets?
A Python library from Hugging Face for loading public AI datasets from the Hub with one line and preprocessing them efficiently with Arrow-backed tools. Datasets is a lightweight Python library with two main jobs. It provides one-line loaders for many public datasets hosted on the Hugging Face Hub, covering text, image, audio, video, PDF and NIfTI medical data, and returns them ready for NumPy, Pandas, PyTorch, TensorFlow, JAX or Polars workflows. It also offers preprocessing tools for both Hub datasets and local files in formats such as CSV, JSON, Parquet, XML, WAV, JPEG and PDF.
Is datasets free to use? What license does it have?
datasets is open source under the Apache-2.0 license. Check the license terms before using it in commercial products.
Is datasets still maintained?
Actively maintained. The last commit to datasets was on Oct 19, 2026.
What can I use datasets for?
Fine-tuning or evaluating an NLP model on a public benchmark dataset Streaming a very large image or speech corpus during training Preprocessing and tokenizing local CSV or Parquet data for model training Building a similarity search index over embeddings with FAISS
How do I install datasets?
Install the library with pip: pip install datasets Load a dataset from the Hub in Python: python -c "from datasets import load_dataset; squad_dataset = load_dataset('rajpurkar/squad'); print(squad_dataset['train'][0])"
Where can I get datasets?
The source code and installation instructions are on GitHub: https://github.com/huggingface/datasets

Summary generated by AI from the project's README. GitHub stats refresh daily.

Related projects