Repolode
Submit
apache

apache/airflow

Apache Airflow is a platform to programmatically author, schedule, and monitor workflows defined as Python code (Dags).

View on GitHubWebsite
Stars
47,223
Forks
18,089
Open issues
1,877
Last commit
4 hours ago

What is it?

Apache Airflow lets you define workflows as code. Each workflow is a Dag made of tasks with dependencies, and the scheduler runs those tasks on a set of workers in the order the dependencies require. Because pipelines are ordinary Python, they can be generated dynamically, parameterized with Jinja templating, and kept under version control and tests.

The project provides a web UI for viewing all Dags, their run status, a grid view across time, a dependency graph for a given run, backfills over date ranges, and the source code of each Dag. Command line utilities cover operations on Dags. Airflow is installed from PyPI or Docker images and supports Kubernetes and a range of databases.

Airflow is aimed at batch-style, mostly static pipelines rather than true streaming. It is widely used for ETL and ELT jobs, as well as for orchestrating machine learning and LLM pipeline steps such as data preparation, model invocation and evaluation. Tasks are expected to be idempotent, and large data volumes are better handled by external services.

Key features

  • 01Define workflows as Python code, with dynamic Dag generation and Jinja templating.
  • 02Scheduler runs tasks across a set of workers while respecting dependencies.
  • 03Web UI with grid, graph, and asset views plus run status and backfill support.
  • 04Built-in operators, extensible through provider distributions.
  • 05Command line utilities for managing Dags and operations.
  • 06Installable via pip or Docker images, with Kubernetes support.

Use cases

  • Scheduling and monitoring nightly ETL or ELT data pipelines.
  • Orchestrating machine learning training, evaluation, and deployment steps.
  • Coordinating steps of an LLM or AI pipeline such as data prep and model calls.

Quick start

  1. 1

    Install the apache-airflow package from PyPI, as described in the README's Installing from PyPI section

  2. 2

    Follow the official getting started guide to run Airflow

Things to know

  • Windows is not natively supported; use WSL2 or Linux containers.
  • SQLite is for testing only and should not be used in production.
  • Installing from PyPI can be tricky because of its dependency handling.

Topics

#workflow-orchestration#data-pipelines#scheduler#python#etl#mlops#airflow#apache#apache-airflow#automation

FAQ

What is airflow?
Apache Airflow is a platform to programmatically author, schedule, and monitor workflows defined as Python code (Dags). Apache Airflow lets you define workflows as code. Each workflow is a Dag made of tasks with dependencies, and the scheduler runs those tasks on a set of workers in the order the dependencies require. Because pipelines are ordinary Python, they can be generated dynamically, parameterized with Jinja templating, and kept under version control and tests.
Is airflow free to use? What license does it have?
airflow is open source under the Apache-2.0 license. Check the license terms before using it in commercial products.
Is airflow still maintained?
Actively maintained. The last commit to airflow was on Oct 20, 2026.
What can I use airflow for?
Scheduling and monitoring nightly ETL or ELT data pipelines. Orchestrating machine learning training, evaluation, and deployment steps. Coordinating steps of an LLM or AI pipeline such as data prep and model calls.
How do I install airflow?
Install the apache-airflow package from PyPI, as described in the README's Installing from PyPI section Follow the official getting started guide to run Airflow
Where can I get airflow?
The source code and installation instructions are on GitHub: https://github.com/apache/airflow

Summary generated by AI from the project's README. GitHub stats refresh daily.

Related projects