Repolode
Submit
lance-format

lance-format/lance

An open lakehouse table and file format for multimodal AI, with vector search, full-text search, and fast random access on object storage.

View on GitHubWebsite
Stars
7,266
Forks
926
Open issues
1,197
Last commit
1 hour ago

What is it?

Lance is a file format, table format, and catalog spec for building a lakehouse on top of object storage for AI workloads. It stores images, videos, audio, text, and embeddings in one format, with blob encoding and lazy loading for multimodal data.

The project targets workloads that traditional Parquet-based lakehouses handle poorly, such as similarity search, random sampling for ML training, and adding backfilled feature columns without rewriting the whole table. It supports vector indexes, BM25 full-text search, SQL analytics on the same dataset, and automatic versioning with time travel, tags, and branches.

Lance can be written from Parquet with a few lines of Python and read through Pandas, Polars, DuckDB, PyArrow, and PyTorch. Integrations with Ray, Spark, Trino, and Flink are also listed, and the core is implemented in Rust with Python and Java bindings.

Key features

  • 01Vector similarity search with IVF_PQ indexes alongside full-text (BM25) and SQL queries.
  • 02Random access that the project benchmarks at up to 100x faster than Parquet for batched lookups.
  • 03Stores images, video, audio, text, and embeddings in one format with lazy blob loading.
  • 04Adds columns with backfilled values without rewriting the full table.
  • 05Automatic versioning with ACID transactions, time travel, tags, and branches.
  • 06Works with Pandas, Polars, DuckDB, PyArrow, and PyTorch.

Use cases

  • Building search engines or feature stores that combine vector, keyword, and SQL filters.
  • Large-scale ML training that needs fast random sampling over image or video datasets.
  • Managing embeddings and multimodal assets in an object-storage lakehouse.
  • Adding engineered features to an existing dataset without a full rewrite.

Quick start

  1. 1

    Install the Python package pylance

    pip install pylance
  2. 2

    Convert a Parquet dataset to Lance and read it back in Python

Things to know

  • Lance releases frequently, so pin data_storage_version for deterministic writes across mixed Lance versions.
  • The 'next' file format alias is unstable and should not be used for production data.

In collections

Vector databasesComputer vision libraries & toolsData science courses & tools

Topics

#vector-database#multimodal#data-format#apache-arrow#duckdb#embeddings#machine-learning#lakehouse#computer-vision#data-analysis

FAQ

What is lance?
An open lakehouse table and file format for multimodal AI, with vector search, full-text search, and fast random access on object storage. Lance is a file format, table format, and catalog spec for building a lakehouse on top of object storage for AI workloads. It stores images, videos, audio, text, and embeddings in one format, with blob encoding and lazy loading for multimodal data.
Is lance free to use? What license does it have?
lance is open source under the Apache-2.0 license. Check the license terms before using it in commercial products.
Is lance still maintained?
Actively maintained. The last commit to lance was on Nov 7, 2026.
What can I use lance for?
Building search engines or feature stores that combine vector, keyword, and SQL filters. Large-scale ML training that needs fast random sampling over image or video datasets. Managing embeddings and multimodal assets in an object-storage lakehouse. Adding engineered features to an existing dataset without a full rewrite.
How do I install lance?
Install the Python package pylance: pip install pylance Convert a Parquet dataset to Lance and read it back in Python
Where can I get lance?
The source code and installation instructions are on GitHub: https://github.com/lance-format/lance

Summary generated by AI from the project's README. GitHub stats refresh daily.

Related projects