Repolode
Submit
piercefreeman

piercefreeman/waymark

Python library for durable background workflows and actions that survive restarts and crashes, using Postgres as the only infrastructure.

View on GitHubWebsite
Stars
121
Forks
5
Open issues
92
Last commit
23 hours ago

What is it?

Waymark is a library for building background tasks that keep running through server restarts, task crashes, and long-running jobs. Its only infrastructure dependency is Postgres, and Python is the first supported language. You write plain async Python functions as actions, which run distributed on worker nodes, and a workflow that orchestrates them with ordinary control flow such as loops, conditionals, and parallel branches.

Workflows are compiled once from the Python source of their run() method into an intermediate representation, then executed by a Rust runtime that snapshots its state to Postgres as it progresses. A crashed workflow resumes where it left off, and the original workflow code is not re-executed during recovery. Because of this design, workflows must use supported patterns; non-deterministic calls like time or uuid need to be wrapped as explicit actions.

The project includes a built-in webapp served by the waymark-executor process, which shows workflow instances, their call timelines, and node load. It also supports per-call retry policies and timeouts, durable sleep, and cron or interval schedules. The project is pre-1.0, and its maintainers advise against production use until backwards compatibility is guaranteed.

Key features

  • 01Durable workflows that resume from their last state after crashes or restarts
  • 02Actions run on distributed workers, with per-call retry policies and timeouts
  • 03Ordinary Python control flow: loops, conditionals, and asyncio.gather for parallel work
  • 04Durable sleep that survives restarts, from seconds to days
  • 05Cron and interval schedules without extra infrastructure
  • 06Built-in webapp to inspect workflow instances and node load

Use cases

  • Sending batches of emails or calling third-party APIs with retries and fan-out
  • Running database jobs that must complete reliably despite deploys or crashes
  • Scheduling periodic data syncs without a separate task broker
  • Orchestrating multi-step async processes in a Postgres-backed Python app

Quick start

  1. 1

    Install the package with uv

    uv add waymark
  2. 2

    Start the node against your Postgres database with the webapp enabled

    export WAYMARK_DATABASE_URL=postgresql://postgres:postgres@localhost:5432/waymark export WAYMARK_HTTP_ENABLED=true uv run waymark-executor

Things to know

  • The README says not to use it in production yet, since the spec changes quickly and backwards compatibility is not guaranteed before 1.0.0.
  • Python is the only supported language today.
  • Not suited to latency-sensitive jobs needing sub-100ms handling or more than about 10k concurrently coordinated actions.

Topics

#workflow#background-jobs#postgres#task-queue#python#rust#scheduling#durable-execution

FAQ

What is waymark?
Python library for durable background workflows and actions that survive restarts and crashes, using Postgres as the only infrastructure. Waymark is a library for building background tasks that keep running through server restarts, task crashes, and long-running jobs. Its only infrastructure dependency is Postgres, and Python is the first supported language. You write plain async Python functions as actions, which run distributed on worker nodes, and a workflow that orchestrates them with ordinary control flow such as loops, conditionals, and parallel branches.
Is waymark free to use? What license does it have?
waymark is open source under the MIT license. Check the license terms before using it in commercial products.
Is waymark still maintained?
Actively maintained. The last commit to waymark was on Nov 6, 2026.
What can I use waymark for?
Sending batches of emails or calling third-party APIs with retries and fan-out Running database jobs that must complete reliably despite deploys or crashes Scheduling periodic data syncs without a separate task broker Orchestrating multi-step async processes in a Postgres-backed Python app
How do I install waymark?
Install the package with uv: uv add waymark Start the node against your Postgres database with the webapp enabled: export WAYMARK_DATABASE_URL=postgresql://postgres:postgres@localhost:5432/waymark export WAYMARK_HTTP_ENABLED=true uv run waymark-executor
Where can I get waymark?
The source code and installation instructions are on GitHub: https://github.com/piercefreeman/waymark

Summary generated by AI from the project's README. GitHub stats refresh daily.

Related projects