Repolode
Submit
apache

apache/spark

A unified analytics engine for large-scale data processing, with APIs in Scala, Java, Python and R.

View on GitHubWebsite
Stars
44,283
Forks
29,463
Open issues
692
Last commit
4 hours ago

What is it?

Apache Spark is a distributed engine for processing large datasets. It exposes high-level APIs in Scala, Java, Python and R (R is deprecated), and runs general computation graphs on an optimized execution engine.

On top of the core engine, Spark ships several libraries: Spark SQL for SQL queries and DataFrames, the pandas API on Spark for running pandas-style code at cluster scale, MLlib for machine learning, GraphX for graph processing, and Structured Streaming for stream processing.

The quickest way to try it is the bundled spark-shell (Scala) or pyspark (Python) shell. Examples can be submitted to a cluster by setting the master URL, for example to a Spark standalone cluster or to YARN.

Key features

  • 01Unified APIs in Scala, Java, Python and R
  • 02Spark SQL for SQL queries and DataFrames
  • 03pandas API on Spark for pandas-style workloads at scale
  • 04MLlib for machine learning and GraphX for graph processing
  • 05Structured Streaming for stream processing

Use cases

  • Running large batch ETL and analytics jobs over data in HDFS or other Hadoop-compatible storage
  • Scaling existing pandas code to datasets too large for a single machine
  • Building machine learning pipelines on big data with MLlib
  • Processing continuous data streams with Structured Streaming

Things to know

  • The R API is deprecated according to the README.
  • The README badge indicates it requires Java 17 or newer.

In collections

Databases & SQL tools

Topics

#big-data#distributed-computing#sql#machine-learning#stream-processing#scala#python#data-engineering#java#jdbc

FAQ

What is spark?
A unified analytics engine for large-scale data processing, with APIs in Scala, Java, Python and R. Apache Spark is a distributed engine for processing large datasets. It exposes high-level APIs in Scala, Java, Python and R (R is deprecated), and runs general computation graphs on an optimized execution engine.
Is spark free to use? What license does it have?
spark is open source under the Apache-2.0 license. Check the license terms before using it in commercial products.
Is spark still maintained?
Actively maintained. The last commit to spark was on Oct 22, 2026.
What can I use spark for?
Running large batch ETL and analytics jobs over data in HDFS or other Hadoop-compatible storage Scaling existing pandas code to datasets too large for a single machine Building machine learning pipelines on big data with MLlib Processing continuous data streams with Structured Streaming
Where can I get spark?
The source code and installation instructions are on GitHub: https://github.com/apache/spark

Summary generated by AI from the project's README. GitHub stats refresh daily.

Related projects