Repolode
Submit
spotify

spotify/luigi

Python package for building complex pipelines of batch jobs, with dependency resolution, workflow management, failure handling and a web visualiser.

View on GitHub
Stars
18,784
Forks
2,474
Open issues
192
Last commit
8 days ago

What is it?

Luigi is a Python package for chaining many batch tasks into pipelines. You define each task and its dependencies in Python code, and Luigi resolves the dependency graph, runs tasks in order, and handles failures so that long-running workflows can be restarted without redoing completed work.

A task can be a Hadoop or Hive job, a Pig script, a Spark job, a database table dump, an scp file transfer, or an arbitrary Python function. Luigi ships with built-in support for Hadoop and HDFS, and its file system operations on local files are atomic, so a failed run does not leave partial output behind.

Luigi includes a web server with a visualiser for searching and filtering tasks and viewing the dependency graph, where completed and pending tasks are shown in different colours. It is installed with pip and supports TOML-based configuration as an optional extra.

Key features

  • 01Resolves task dependencies and runs tasks in the correct order.
  • 02Built-in support for Hadoop, HDFS, Hive, Pig and Spark jobs.
  • 03Atomic file system operations so failed runs don't leave partial data.
  • 04Web visualiser for searching tasks and viewing the dependency graph.
  • 05Tasks are defined in Python, so dependencies can be computed programmatically.

Use cases

  • Orchestrating daily or hourly data pipelines made of Hadoop, Hive or Spark jobs.
  • Chaining database exports, transformations and machine learning training steps.
  • Restarting long multi-step batch workflows after a failure without rerunning finished tasks.

Quick start

  1. 1

    Install Luigi from PyPI

    pip install luigi
  2. 2

    Install Luigi with TOML config support (optional)

    pip install luigi[toml]

Things to know

  • Built for batch pipelines, so it is not suited to real-time or streaming workloads.
  • Tasks are defined in Python code, so teams without Python experience may find it harder to adopt.

Topics

#orchestration-framework#python#workflow#batch-processing#data-pipeline#hadoop#scheduling#luigi

FAQ

What is luigi?
Python package for building complex pipelines of batch jobs, with dependency resolution, workflow management, failure handling and a web visualiser. Luigi is a Python package for chaining many batch tasks into pipelines. You define each task and its dependencies in Python code, and Luigi resolves the dependency graph, runs tasks in order, and handles failures so that long-running workflows can be restarted without redoing completed work.
Is luigi free to use? What license does it have?
luigi is open source under the Apache-2.0 license. Check the license terms before using it in commercial products.
Is luigi still maintained?
Actively maintained. The last commit to luigi was on Oct 14, 2026.
What can I use luigi for?
Orchestrating daily or hourly data pipelines made of Hadoop, Hive or Spark jobs. Chaining database exports, transformations and machine learning training steps. Restarting long multi-step batch workflows after a failure without rerunning finished tasks.
How do I install luigi?
Install Luigi from PyPI: pip install luigi Install Luigi with TOML config support (optional): pip install luigi[toml]
Where can I get luigi?
The source code and installation instructions are on GitHub: https://github.com/spotify/luigi

Summary generated by AI from the project's README. GitHub stats refresh daily.

Related projects