Repolode
Submit
harbor-framework

harbor-framework/harbor

A framework from the Terminal-Bench team for evaluating and optimizing AI agents and language models in container environments.

View on GitHubWebsite
Stars
6,257
Forks
2,032
Open issues
1,094
Last commit
2 hours ago

What is it?

Harbor is a framework for evaluating and optimizing agents and language models. It runs agents such as Claude Code, OpenHands and Codex CLI against benchmark tasks, with each task executed inside a container environment.

Beyond running existing benchmarks, Harbor lets users build and share their own benchmarks and environments. Experiments can run in parallel across many environments, locally with Docker or on cloud sandbox providers such as Daytona and Modal, selected with a flag. It is the official harness for Terminal-Bench-2.0, and it can also run third-party datasets like SWE-Bench, which are listed through a CLI command.

Harbor can also generate rollouts for reinforcement learning optimization. It is installed as a CLI with uv or pip.

Key features

  • 01Evaluates arbitrary agents such as Claude Code, OpenHands and Codex CLI.
  • 02Runs experiments in parallel across many containerized environments.
  • 03Supports local Docker execution and cloud providers such as Daytona and Modal.
  • 04Lets users build and share custom benchmarks and environments.
  • 05Generates rollouts for RL optimization.
  • 06Official harness for Terminal-Bench-2.0.

Use cases

  • Comparing different coding agents and language models on terminal tasks.
  • Running SWE-Bench or other third-party benchmarks at scale.
  • Creating custom evaluation environments for an internal agent.
  • Collecting rollout data to train agents with reinforcement learning.

Quick start

  1. 1

    Install Harbor with uv

    uv tool install harbor
  2. 2

    Or install Harbor with pip

    pip install harbor
  3. 3

    Set your model provider API key

    export ANTHROPIC_API_KEY=<YOUR-KEY>
  4. 4

    Run Terminal-Bench-2.0 locally with Docker

    harbor run --dataset terminal-bench@2.0 --agent claude-code --model anthropic/claude-opus-5-5 --n-concurrent 4

Things to know

  • Running the example requires an Anthropic API key and Docker.
  • Cloud environments such as Daytona need a separate API key and account.

Topics

#evals#agents#llm-benchmark#reinforcement-learning#docker#terminal#rl-environments#terminal-bench

FAQ

What is harbor?
A framework from the Terminal-Bench team for evaluating and optimizing AI agents and language models in container environments. Harbor is a framework for evaluating and optimizing agents and language models. It runs agents such as Claude Code, OpenHands and Codex CLI against benchmark tasks, with each task executed inside a container environment.
Is harbor free to use? What license does it have?
harbor is open source under the Apache-2.0 license. Check the license terms before using it in commercial products.
Is harbor still maintained?
Actively maintained. The last commit to harbor was on Oct 19, 2026.
What can I use harbor for?
Comparing different coding agents and language models on terminal tasks. Running SWE-Bench or other third-party benchmarks at scale. Creating custom evaluation environments for an internal agent. Collecting rollout data to train agents with reinforcement learning.
How do I install harbor?
Install Harbor with uv: uv tool install harbor Or install Harbor with pip: pip install harbor Set your model provider API key: export ANTHROPIC_API_KEY=<YOUR-KEY> Run Terminal-Bench-2.0 locally with Docker: harbor run --dataset terminal-bench@2.0 --agent claude-code --model anthropic/claude-opus-5-5 --n-concurrent 4
Where can I get harbor?
The source code and installation instructions are on GitHub: https://github.com/harbor-framework/harbor

Summary generated by AI from the project's README. GitHub stats refresh daily.

Related projects