Repolode
Submit
Eventual-Inc

Eventual-Inc/Daft

A high-performance data engine in Python with a Rust core for processing images, audio, video, embeddings and structured data at scale.

View on GitHubWebsite
Stars
5,829
Forks
586
Open issues
434
Last commit
3 days ago

What is it?

Daft is a data engine for AI and multimodal workloads. It lets you process images, audio, video, embeddings and structured tables in one DataFrame-style framework, so you do not need separate tools for each data type.

Daft includes built-in AI operations such as running LLM prompts, generating embeddings and classifying data with OpenAI, Hugging Face Transformers or custom models. The engine is written in Rust with a Python API, and it starts on a single machine and can scale out to distributed clusters on Ray or Kubernetes.

It reads data directly from S3, GCS, Iceberg, Delta Lake, Hugging Face and Unity Catalog. Installation is via pip and requires Python 3.10 or higher.

Key features

  • 01Processes images, audio, video, embeddings and structured data in one framework.
  • 02Runs LLM prompts, embedding generation and classification at scale.
  • 03Python API with a Rust execution engine.
  • 04Scales from a local machine to Ray or Kubernetes clusters.
  • 05Reads from S3, GCS, Iceberg, Delta Lake, Hugging Face and Unity Catalog.

Use cases

  • Generating embeddings or captions for large image or video collections.
  • Running batch LLM inference over text columns in a data pipeline.
  • Querying Iceberg or Parquet tables that sit alongside multimodal files.
  • Scaling a Python data preprocessing job from a laptop to a Ray cluster.

Quick start

  1. 1

    Install Daft from PyPI (Python 3.10 or higher)

    pip install daft
  2. 2

    Read the Quickstart guide in the docs to run your first example

Things to know

  • Requires Python 3.10 or higher.

Topics

#data-engineering#multimodal#embeddings#llm#distributed-computing#dataframe#parquet#ray#ai-engineering#ai-pipeline

FAQ

What is Daft?
A high-performance data engine in Python with a Rust core for processing images, audio, video, embeddings and structured data at scale. Daft is a data engine for AI and multimodal workloads. It lets you process images, audio, video, embeddings and structured tables in one DataFrame-style framework, so you do not need separate tools for each data type.
Is Daft free to use? What license does it have?
Daft is open source under the Apache-2.0 license. Check the license terms before using it in commercial products.
Is Daft still maintained?
Actively maintained. The last commit to Daft was on Nov 4, 2026.
What can I use Daft for?
Generating embeddings or captions for large image or video collections. Running batch LLM inference over text columns in a data pipeline. Querying Iceberg or Parquet tables that sit alongside multimodal files. Scaling a Python data preprocessing job from a laptop to a Ray cluster.
How do I install Daft?
Install Daft from PyPI (Python 3.10 or higher): pip install daft Read the Quickstart guide in the docs to run your first example
Where can I get Daft?
The source code and installation instructions are on GitHub: https://github.com/Eventual-Inc/Daft

Summary generated by AI from the project's README. GitHub stats refresh daily.

Related projects