Repolode
Submit
lakehq

lakehq/sail

A drop-in Apache Spark replacement written in Rust that runs existing PySpark code over the Spark Connect protocol.

Open-source alternative to:Databricks

View on GitHubWebsite
Stars
3,488
Forks
240
Open issues
294
Last commit
20 hours ago

What is it?

Sail is a distributed compute engine that stands in for Apache Spark. Clients connect to it using the Spark Connect protocol, so existing PySpark DataFrame and Spark SQL code can run against Sail without modification. It is written entirely in Rust and does not depend on the JVM.

Sail is built on Apache Arrow and Apache DataFusion, using a columnar in-memory format and vectorized execution. It covers batch processing, stream processing, and Python UDF, UDAF, UDWF, and UDTF workloads. It can run as a local server started from the sail command or the pysail Python package, or in cluster mode on Kubernetes.

For data access, Sail provides native support for Delta Lake and Apache Iceberg tables and integrates with catalogs such as Iceberg REST, AWS Glue, Unity Catalog, and Hive Metastore. It reads and writes data from S3, Azure, Google Cloud Storage, HDFS, Cloudflare R2, HTTP, and local file systems. A compatibility check script is included as an experimental aid for finding PySpark functions that Sail does not yet implement.

Key features

  • 01Speaks the Spark Connect protocol, so PySpark clients connect without code changes.
  • 02Supports Spark SQL and the Spark DataFrame API, including Python UDF, UDAF, UDWF, and UDTF.
  • 03Native read and write support for Delta Lake and Apache Iceberg tables.
  • 04Integrates with Iceberg REST, AWS Glue, Unity Catalog, and Hive Metastore catalogs.
  • 05Runs without the JVM, built on Apache Arrow and DataFusion.
  • 06Deployable as a local server or as a cluster on Kubernetes.

Use cases

  • Moving existing PySpark ETL jobs to a lighter-weight engine without rewriting code.
  • Running batch analytics on Delta Lake or Iceberg tables stored in S3 or Azure.
  • Reducing cluster cost and startup time for Spark-style workloads on Kubernetes.
  • Developing and testing Spark code locally with a single-command server.

Quick start

  1. 1

    Install Sail from PyPI

    pip install pysail
  2. 2

    Install PySpark client

    pip install "pyspark-client"
  3. 3

    Start the local Sail server

    sail spark server --port 50051
  4. 4

    Connect PySpark to the server at sc://localhost:50051 and run your code unchanged

Things to know

  • The PySpark compatibility check script is experimental and does not verify behavioral parity with Spark.
  • Production deployment on Kubernetes requires building a Docker image and writing manifests.

In collections

Databases & SQL tools

Topics

#spark#data-engineering#rust#lakehouse#delta-lake#apache-iceberg#sql#distributed-computing#apache-spark#arrow

FAQ

What is sail?
A drop-in Apache Spark replacement written in Rust that runs existing PySpark code over the Spark Connect protocol. Sail is a distributed compute engine that stands in for Apache Spark. Clients connect to it using the Spark Connect protocol, so existing PySpark DataFrame and Spark SQL code can run against Sail without modification. It is written entirely in Rust and does not depend on the JVM.
Is sail free to use? What license does it have?
sail is open source under the Apache-2.0 license. Check the license terms before using it in commercial products.
Is sail still maintained?
Actively maintained. The last commit to sail was on Nov 6, 2026.
What can I use sail for?
Moving existing PySpark ETL jobs to a lighter-weight engine without rewriting code. Running batch analytics on Delta Lake or Iceberg tables stored in S3 or Azure. Reducing cluster cost and startup time for Spark-style workloads on Kubernetes. Developing and testing Spark code locally with a single-command server.
How do I install sail?
Install Sail from PyPI: pip install pysail Install PySpark client: pip install "pyspark-client" Start the local Sail server: sail spark server --port 50051 Connect PySpark to the server at sc://localhost:50051 and run your code unchanged
Where can I get sail?
The source code and installation instructions are on GitHub: https://github.com/lakehq/sail

Summary generated by AI from the project's README. GitHub stats refresh daily.

Related projects