Repolode
Submit
Netflix

Netflix/maestro

Netflix's general-purpose workflow orchestrator, offered as a workflow-as-a-service for scheduling and running data and ML pipelines.

Open-source alternative to:Apache Airflow

View on GitHubWebsite
Stars
3,853
Forks
311
Open issues
39
Last commit
yesterday

What is it?

Maestro is a workflow orchestrator originally built at Netflix to run data and machine learning pipelines. Users define workflows as directed acyclic graphs (DAGs) of jobs, then schedule, start and monitor them through a server API. The project is written in Java and runs on Spring Boot.

The server exposes a versioned REST API for creating workflows, starting runs and querying instances and step runs. Workflows can be pushed as YAML or JSON, and a Python SDK (maestro-sdk) provides a builder for defining workflows in code. A sample build, run and trigger flow is included in the setup instructions.

The repository includes an AWS module that runs locally with LocalStack for SQS and SNS, plus a Kubernetes option for executing jobs. An optional maestro-extensions service consumes server events to provide views such as flattened foreach steps.

Key features

  • 01Defines workflows as DAGs of jobs, submitted as YAML or JSON.
  • 02Exposes a versioned REST API to create, start and inspect workflows and runs.
  • 03Provides a Python SDK for building and pushing workflows.
  • 04Supports AWS integration through SQS and SNS event handling.
  • 05Can execute jobs on Kubernetes.
  • 06Offers an extensions service for foreach flattening views.

Use cases

  • Scheduling recurring data engineering ETL or ELT pipelines.
  • Orchestrating machine learning training and batch scoring workflows.
  • Running large numbers of data jobs with a managed, API-driven workflow service.
  • Evaluating a Netflix-style workflow engine for an internal data platform.

Quick start

  1. 1

    Install prerequisites: Git, Java 21, Gradle and Docker

  2. 2

    Build the project

    ./gradlew build
  3. 3

    Run the server

    ./gradlew bootRun
  4. 4

    Create a sample workflow via the API

    curl --header "user: tester" -X POST 'http://127.0.0.1:8080/api/v3/workflows' -H "Content-Type: application/json" -d @maestro-server/src/test/resources/samples/sample-dag-test-1.json

Things to know

  • Requires Java 21, Gradle and Docker to build and run.
  • Designed as a large-scale internal platform; self-hosting requires setup effort.
  • Some features such as AWS mode need Docker Compose and LocalStack for local testing.

Topics

#workflow-orchestration#data-pipelines#dag#java#spring-boot#batch-processing#etl#scheduler#agentic-workflow#analytics

FAQ

What is maestro?
Netflix's general-purpose workflow orchestrator, offered as a workflow-as-a-service for scheduling and running data and ML pipelines. Maestro is a workflow orchestrator originally built at Netflix to run data and machine learning pipelines. Users define workflows as directed acyclic graphs (DAGs) of jobs, then schedule, start and monitor them through a server API. The project is written in Java and runs on Spring Boot.
Is maestro free to use? What license does it have?
maestro is open source under the Apache-2.0 license. Check the license terms before using it in commercial products.
Is maestro still maintained?
Actively maintained. The last commit to maestro was on Nov 5, 2026.
What can I use maestro for?
Scheduling recurring data engineering ETL or ELT pipelines. Orchestrating machine learning training and batch scoring workflows. Running large numbers of data jobs with a managed, API-driven workflow service. Evaluating a Netflix-style workflow engine for an internal data platform.
How do I install maestro?
Install prerequisites: Git, Java 21, Gradle and Docker Build the project: ./gradlew build Run the server: ./gradlew bootRun Create a sample workflow via the API: curl --header "user: tester" -X POST 'http://127.0.0.1:8080/api/v3/workflows' -H "Content-Type: application/json" -d @maestro-server/src/test/resources/samples/sample-dag-test-1.json
Where can I get maestro?
The source code and installation instructions are on GitHub: https://github.com/Netflix/maestro

Summary generated by AI from the project's README. GitHub stats refresh daily.

Related projects