Repolode
Submit
apache

apache/hamilton

A lightweight Python library for defining data transformations as testable, self-documenting DAGs that run wherever Python runs.

View on GitHubWebsite
Stars
2,615
Forks
222
Open issues
189
Last commit
2 days ago

What is it?

Apache Hamilton is a Python library for organizing data transformations as directed acyclic graphs. You write ordinary Python functions, and each function's parameter names declare its dependencies on other functions. Hamilton reads those definitions and builds the graph automatically, so the code stays readable and can be visualized.

The DAG definition is separated from its execution. The same transformations can run in a script, a notebook, an Airflow task, or a FastAPI server. Features include function modifiers such as @config.when() for switching behavior between environments, @check_output for validating outputs, and a SchemaValidator adapter that tracks schemas of pandas, polars and Ibis dataframes.

The project includes an optional UI for cataloging DAGs, profiling datasets and tracking executions with lineage, which can run as a local server or be self-hosted with Docker. The project is an Apache Software Foundation incubating project, originally started at Stitch Fix.

Key features

  • 01Defines transformations as plain Python functions with dependencies declared through parameter names.
  • 02Automatically builds and visualizes the DAG from the function definitions.
  • 03Function modifiers like @config.when() reduce duplicated code across environments.
  • 04@check_output and SchemaValidator validate outputs and dataframe schemas.
  • 05Optional UI tracks executions, lineage and dataset profiles.

Use cases

  • Structuring ETL pipelines or feature engineering code in pandas, polars or Spark-based workflows.
  • Reusing the same transformation code across a notebook, an Airflow DAG and a FastAPI service.
  • Building RAG or LLM data preparation steps that need lineage and validation.
  • Moving data science code from notebooks to production modules with testable functions.

Quick start

  1. 1

    Install the library with the visualization extra

    pip install "apache-hamilton[visualization]"
  2. 2

    Install Graphviz separately on your system for visualizations

  3. 3

    Start the Apache Hamilton UI server

    hamilton ui

Things to know

  • Graphviz must be installed separately for DAG visualizations.
  • The project is undergoing Apache incubation, so it has not yet been fully endorsed by the ASF.

In collections

Data science courses & tools

Topics

#dag#etl#data-engineering#dataframe#pandas#lineage#feature-engineering#python#data-analysis#data-science

FAQ

What is hamilton?
A lightweight Python library for defining data transformations as testable, self-documenting DAGs that run wherever Python runs. Apache Hamilton is a Python library for organizing data transformations as directed acyclic graphs. You write ordinary Python functions, and each function's parameter names declare its dependencies on other functions. Hamilton reads those definitions and builds the graph automatically, so the code stays readable and can be visualized.
Is hamilton free to use? What license does it have?
hamilton is open source under the Apache-2.0 license. Check the license terms before using it in commercial products.
Is hamilton still maintained?
Actively maintained. The last commit to hamilton was on Oct 21, 2026.
What can I use hamilton for?
Structuring ETL pipelines or feature engineering code in pandas, polars or Spark-based workflows. Reusing the same transformation code across a notebook, an Airflow DAG and a FastAPI service. Building RAG or LLM data preparation steps that need lineage and validation. Moving data science code from notebooks to production modules with testable functions.
How do I install hamilton?
Install the library with the visualization extra: pip install "apache-hamilton[visualization]" Install Graphviz separately on your system for visualizations Start the Apache Hamilton UI server: hamilton ui
Where can I get hamilton?
The source code and installation instructions are on GitHub: https://github.com/apache/hamilton

Summary generated by AI from the project's README. GitHub stats refresh daily.

Related projects