Repolode
Submit
apache

apache/datafusion-comet

A Spark accelerator that runs Spark SQL and DataFrame operators on Apache Arrow columnar data via a Rust engine built on DataFusion, with no code changes.

View on GitHubWebsite
Stars
1,315
Forks
413
Open issues
867
Last commit
38 minutes ago

What is it?

Apache DataFusion Comet is an accelerator for Apache Spark. It replaces Spark operators and expressions with implementations that process Apache Arrow columnar data directly, so query execution avoids the per-row overhead of Spark's traditional row-based engine. Most operators run as native Rust code built on Apache DataFusion, while others run in the JVM on Arrow batches.

Comet covers Parquet scans (including Iceberg tables read from Spark), Arrow-IPC columnar shuffle with hash and range partitioning, hash, sort-merge and broadcast joins, hash aggregates, and a large set of Spark expressions. It supports Spark 3.4, 3.5, 4.0, 4.1 and 4.2, and is installed by adding the matching jar to the Spark classpath and enabling the plugin.

The project targets existing Spark SQL, DataFrame and PySpark workloads that should run faster on the same hardware, without GPUs or other specialized accelerators.

Key features

  • 01Native Rust execution of Spark operators and expressions built on Apache DataFusion
  • 02Native Parquet scan and accelerated Iceberg table reads from Spark
  • 03Arrow-IPC columnar shuffle with hash and range partitioning
  • 04Hash, sort-merge and broadcast join support
  • 05Keeps Scala and Java scalar UDFs in the Comet pipeline via whole-stage codegen
  • 06Works with existing Spark SQL, DataFrame and PySpark code without changes

Use cases

  • Reducing cloud or on-prem cost of existing Spark SQL and ETL jobs
  • Speeding up Spark queries over Parquet or Iceberg data lakes
  • Running large analytical workloads such as TPC-DS on commodity CPU clusters

Topics

#spark#arrow#datafusion#rust#query-acceleration#columnar#parquet#data-engineering

FAQ

What is datafusion-comet?
A Spark accelerator that runs Spark SQL and DataFrame operators on Apache Arrow columnar data via a Rust engine built on DataFusion, with no code changes. Apache DataFusion Comet is an accelerator for Apache Spark. It replaces Spark operators and expressions with implementations that process Apache Arrow columnar data directly, so query execution avoids the per-row overhead of Spark's traditional row-based engine. Most operators run as native Rust code built on Apache DataFusion, while others run in the JVM on Arrow batches.
Is datafusion-comet free to use? What license does it have?
datafusion-comet is open source under the Apache-2.0 license. Check the license terms before using it in commercial products.
Is datafusion-comet still maintained?
Actively maintained. The last commit to datafusion-comet was on Nov 6, 2026.
What can I use datafusion-comet for?
Reducing cloud or on-prem cost of existing Spark SQL and ETL jobs Speeding up Spark queries over Parquet or Iceberg data lakes Running large analytical workloads such as TPC-DS on commodity CPU clusters
Where can I get datafusion-comet?
The source code and installation instructions are on GitHub: https://github.com/apache/datafusion-comet

Summary generated by AI from the project's README. GitHub stats refresh daily.

Related projects