Repolode
Submit
suyoumo

suyoumo/ClawProBench

A live-first benchmark harness for evaluating LLM agents inside the OpenClaw runtime, with deterministic grading and repeated-trial reliability metrics.

View on GitHubWebsite
Stars
828
Forks
54
Open issues
0
Last commit
2 months ago

What is it?

ClawProBench runs LLM agents against real OpenClaw execution rather than simulated outputs. It ships a catalog of benchmark scenarios, grouped into profiles such as a default ranking suite (core), an extended capability suite (intelligence), and a native-only slice, and produces structured reports with scores, cost, and latency.

Grading is deterministic, using scenario-specific custom checkers. Repeated trials via --trials N let the harness report pass^3, pass@3, average score, and a combined FinalScore that weights consistent success most heavily. Interrupted runs can resume from checkpoints, and execution failures can be re-queued.

The repository includes a CLI (run.py) with inventory, dry-run validation, run, and compare commands, plus scenario YAML files, seeded datasets, and a public leaderboard site.

Key features

  • 01Runs agent tasks in a live local OpenClaw runtime
  • 02Deterministic grading via scenario-specific custom checkers
  • 03Repeated-trial metrics including pass^3, pass@3 and average score
  • 04Benchmark profiles from a 26-scenario core suite to the 102-scenario full suite
  • 05Checkpoint resume and re-queueing of execution failures
  • 06Report comparison across models via the compare command

Use cases

  • Ranking LLM models on agent tasks run inside OpenClaw
  • Measuring whether an agent completes the same task reliably across repeated runs
  • Comparing model versions or providers on a shared, reproducible task set
  • Adding custom scenarios and graders for an OpenClaw-based workflow

Quick start

  1. 1

    Install uv and create a Python 3.11 virtual environment

    pip install uv
  2. 2

    Create the virtual environment and install requirements

    uv venv --python 3.11
  3. 3

    Inspect the benchmark catalog and validate the scenario set

    python3 run.py inventory
  4. 4

    Run a one-trial smoke test on the default ranking benchmark (requires a local OpenClaw runtime and a model)

    python3 run.py run --model '<MODEL>' --execution-mode live --benchmark-profile core --trials 1 --cleanup-agents

Things to know

  • Live runs require a working local OpenClaw CLI plus model auth and config.
  • Full benchmark runs need model access and may be costly, since each trial runs live tasks.

Topics

#agent#benchmark#evaluation#llm#openclaw#harness#leaderboard

FAQ

What is ClawProBench?
A live-first benchmark harness for evaluating LLM agents inside the OpenClaw runtime, with deterministic grading and repeated-trial reliability metrics. ClawProBench runs LLM agents against real OpenClaw execution rather than simulated outputs. It ships a catalog of benchmark scenarios, grouped into profiles such as a default ranking suite (core), an extended capability suite (intelligence), and a native-only slice, and produces structured reports with scores, cost, and latency.
Is ClawProBench free to use? What license does it have?
ClawProBench is open source under the Apache-2.0 license. Check the license terms before using it in commercial products.
Is ClawProBench still maintained?
Updated within 6 months. The last commit to ClawProBench was on Aug 25, 2026.
What can I use ClawProBench for?
Ranking LLM models on agent tasks run inside OpenClaw Measuring whether an agent completes the same task reliably across repeated runs Comparing model versions or providers on a shared, reproducible task set Adding custom scenarios and graders for an OpenClaw-based workflow
How do I install ClawProBench?
Install uv and create a Python 3.11 virtual environment: pip install uv Create the virtual environment and install requirements: uv venv --python 3.11 Inspect the benchmark catalog and validate the scenario set: python3 run.py inventory Run a one-trial smoke test on the default ranking benchmark (requires a local OpenClaw runtime and a model): python3 run.py run --model '<MODEL>' --execution-mode live --benchmark-profile core --trials 1 --cleanup-agents
Where can I get ClawProBench?
The source code and installation instructions are on GitHub: https://github.com/suyoumo/ClawProBench

Summary generated by AI from the project's README. GitHub stats refresh daily.

Related projects