Repolode
Submit
microsoft

microsoft/SynapseML

An open-source library that adds distributed machine learning, deep learning and Azure AI services to Apache Spark pipelines.

View on GitHubWebsite
Stars
5,254
Forks
869
Open issues
144
Last commit
3 days ago

What is it?

SynapseML (formerly MMLSpark) is a library for building large-scale machine learning pipelines on Apache Spark. It exposes composable APIs that share the same style as Spark MLlib, so its estimators and transformers can be dropped into existing Spark workflows. Covered tasks include text analytics, computer vision, anomaly detection, gradient boosting with LightGBM, and Vowpal Wabbit.

Beyond training, it includes Spark Serving for exposing Spark computations as web services, ONNX-based distributed model inference, and HTTP-on-Spark for calling arbitrary web APIs from a cluster. It also provides wrappers for Azure Cognitive Services so they can be called at scale from Spark DataFrames.

The library is published as JVM artifacts for specific Spark versions (Spark 3.5 on Scala 2.12, Spark 4.0 and 4.1 on Scala 2.13), with Python wrappers on PyPI. It runs on Microsoft Fabric, Azure Synapse, Databricks, EMR, HDInsight and standalone Spark.

Key features

  • 01Estimators and transformers that follow the SparkML/MLlib API
  • 02LightGBM gradient boosting and Vowpal Wabbit text models on Spark
  • 03Spark Serving to expose Spark computations as low-latency web services
  • 04Distributed ONNX model inference on Spark
  • 05Wrappers for Azure Cognitive Services called from DataFrames
  • 06Isolation Forest for distributed anomaly detection

Use cases

  • Training gradient boosted models on very large datasets in an existing Spark cluster
  • Running text analytics or vision services over large tables stored in a data lake
  • Deploying Spark pipelines as HTTP endpoints for online scoring
  • Scoring ONNX models at scale inside Spark jobs

Topics

#apache-spark#machine-learning#distributed-computing#deep-learning#lightgbm#onnx#big-data#pyspark#ai#azure

FAQ

What is SynapseML?
An open-source library that adds distributed machine learning, deep learning and Azure AI services to Apache Spark pipelines. SynapseML (formerly MMLSpark) is a library for building large-scale machine learning pipelines on Apache Spark. It exposes composable APIs that share the same style as Spark MLlib, so its estimators and transformers can be dropped into existing Spark workflows. Covered tasks include text analytics, computer vision, anomaly detection, gradient boosting with LightGBM, and Vowpal Wabbit.
Is SynapseML free to use? What license does it have?
SynapseML is open source under the MIT license. Check the license terms before using it in commercial products.
Is SynapseML still maintained?
Actively maintained. The last commit to SynapseML was on Nov 4, 2026.
What can I use SynapseML for?
Training gradient boosted models on very large datasets in an existing Spark cluster Running text analytics or vision services over large tables stored in a data lake Deploying Spark pipelines as HTTP endpoints for online scoring Scoring ONNX models at scale inside Spark jobs
Where can I get SynapseML?
The source code and installation instructions are on GitHub: https://github.com/microsoft/SynapseML

Summary generated by AI from the project's README. GitHub stats refresh daily.

Related projects