Repolode
Submit
m3dev

m3dev/gokart

A Python framework built on luigi for writing machine learning pipelines with automatic caching, reproducibility, and dependency management.

View on GitHubWebsite
Stars
342
Forks
63
Open issues
33
Last commit
7 hours ago

What is it?

Gokart is a Python framework for building machine learning pipelines as chains of tasks. It wraps luigi and focuses on reproducibility: each task's output, parameters, imported module versions, random seed, and logs are saved together with a hash, and the pipeline reruns automatically when a task's parameters change.

Task outputs are passed between steps as intermediate files rather than held in memory. Storage can be local disk, S3, or GCS, and pandas DataFrame I/O includes column checking. Seeds for numpy and random are fixed automatically, and tasks can be locked with Redis when running in parallel.

The project is aimed at ML batch jobs and team development. It is not an experiment tracker and does not provide task visualization; pipelines are written in Python rather than configuration files.

Key features

  • 01Saves each task's output, parameters, module versions, seed, and logs with a hash.
  • 02Reruns the pipeline automatically when task parameters change.
  • 03Stores intermediate results on local disk, S3, or GCS.
  • 04Checks pandas DataFrame types and columns during I/O.
  • 05Fixes numpy and random seeds automatically.
  • 06Locks parallel tasks through Redis.

Use cases

  • Building reproducible machine learning training and preprocessing pipelines.
  • Sharing ML batch workflows across a team with consistent intermediate outputs.
  • Running multi-step data jobs that store large intermediate results on S3 or GCS.

Quick start

  1. 1

    Install gokart with pip

    pip install gokart
  2. 2

    Optionally install S3 support

    pip install gokart[s3]
  3. 3

    Run the minimal example as a Python script

Things to know

  • Redis is needed for task locking when running tasks in parallel.
  • The README says gokart is not an experiment management tool and does not support task visualization.

Topics

#machine-learning#pipeline#luigi#reproducibility#task-scheduling#python#mlops#gokart#pipeline-framework

FAQ

What is gokart?
A Python framework built on luigi for writing machine learning pipelines with automatic caching, reproducibility, and dependency management. Gokart is a Python framework for building machine learning pipelines as chains of tasks. It wraps luigi and focuses on reproducibility: each task's output, parameters, imported module versions, random seed, and logs are saved together with a hash, and the pipeline reruns automatically when a task's parameters change.
Is gokart free to use? What license does it have?
gokart is open source under the MIT license. Check the license terms before using it in commercial products.
Is gokart still maintained?
Actively maintained. The last commit to gokart was on Nov 7, 2026.
What can I use gokart for?
Building reproducible machine learning training and preprocessing pipelines. Sharing ML batch workflows across a team with consistent intermediate outputs. Running multi-step data jobs that store large intermediate results on S3 or GCS.
How do I install gokart?
Install gokart with pip: pip install gokart Optionally install S3 support: pip install gokart[s3] Run the minimal example as a Python script
Where can I get gokart?
The source code and installation instructions are on GitHub: https://github.com/m3dev/gokart

Summary generated by AI from the project's README. GitHub stats refresh daily.

Related projects