Repolode
Submit
camel-ai

camel-ai/crab

A Python framework for building cross-environment benchmarks that evaluate multimodal LLM agents across in-memory, Docker, VM, and physical-machine environments.

View on GitHubWebsite
Stars
430
Forks
64
Open issues
39
Last commit
3 days ago

What is it?

CRAB is a Python framework for building benchmark environments that test LLM agents. Environments can be in-memory, hosted in Docker, run on virtual machines, or spread across physical machines, as long as they can be reached through Python functions. An agent can access several of these environments at once through a single interface.

Actions are defined by adding an @action decorator to a Python function, and an environment is assembled by combining several actions. Tasks and their evaluators are written in plain Python as well. The project also introduces a graph-based evaluator that produces fine-grained metrics.

The repository includes the CRAB-Benchmark-v0 datasets and experiment code, along with example scripts that run a single-environment or multi-environment template with an OpenAI agent. It requires Python 3.10 or newer and is installed with pip as crab-framework.

Key features

  • 01Supports in-memory, Docker, VM, and distributed physical-machine environments.
  • 02Exposes multiple environments to an agent through one unified interface.
  • 03Turns any Python function into an agent action with an @action decorator.
  • 04Defines tasks and evaluators in Python.
  • 05Provides a graph evaluator for fine-grained metrics.

Use cases

  • Benchmarking multimodal LLM agents across desktop, web, and mobile-style environments.
  • Building custom agent test environments from Python functions.
  • Reproducing the CRAB-Benchmark-v0 experiments.

Quick start

  1. 1

    Install prerequisites: Python 3.10 or newer

  2. 2

    Install the CRAB framework client package

    pip install crab-framework[client]
  3. 3

    Set your OpenAI API key

    export OPENAI_API_KEY=<your api key>
  4. 4

    Run the single-environment template example with an OpenAI agent

    python examples/single_env.py

Things to know

  • The README does not state a license, so usage rights are unclear.
  • Running the examples requires an OpenAI API key.
  • Some environments (Docker, VMs, physical machines) need significant self-hosting setup.

In collections

Open-source AI agent frameworks & appsTesting frameworks & test automation tools

Topics

#ai-agent#benchmark#llm-evaluation#multimodal#python#docker#gui-automation#language-model-agent#large-language-models#multi-agent-systems

FAQ

What is crab?
A Python framework for building cross-environment benchmarks that evaluate multimodal LLM agents across in-memory, Docker, VM, and physical-machine environments. CRAB is a Python framework for building benchmark environments that test LLM agents. Environments can be in-memory, hosted in Docker, run on virtual machines, or spread across physical machines, as long as they can be reached through Python functions. An agent can access several of these environments at once through a single interface.
Is crab free to use? What license does it have?
crab has no declared license. Its code is public, but without a license you have no explicit permission to reuse it; ask the author before relying on it.
Is crab still maintained?
Actively maintained. The last commit to crab was on Oct 19, 2026.
What can I use crab for?
Benchmarking multimodal LLM agents across desktop, web, and mobile-style environments. Building custom agent test environments from Python functions. Reproducing the CRAB-Benchmark-v0 experiments.
How do I install crab?
Install prerequisites: Python 3.10 or newer Install the CRAB framework client package: pip install crab-framework[client] Set your OpenAI API key: export OPENAI_API_KEY=<your api key> Run the single-environment template example with an OpenAI agent: python examples/single_env.py
Where can I get crab?
The source code and installation instructions are on GitHub: https://github.com/camel-ai/crab

Summary generated by AI from the project's README. GitHub stats refresh daily.

Related projects