Is ToolSandbox free to use? What license does it have?

ToolSandbox is open source under the Other license. Check the license terms before using it in commercial products.

Is ToolSandbox still maintained?

Updated within 6 months. The last commit to ToolSandbox was on Sep 11, 2026.

What can I use ToolSandbox for?

Comparing how different LLMs handle multi-step tool use with state dependencies. Testing whether an agent recognizes when it lacks the tools or information to finish a task. Building regression tests for an agent's tool-calling behavior in conversational settings.

How do I install ToolSandbox?

Create a conda environment with Python 3.9: conda create -n ToolSandbox python=3.9 Activate the environment: conda activate ToolSandbox Install the project and its dev dependencies from the repository root: pip install '.[dev]' Run a single scenario with GPT-4o as user simulator and agent (replace the API key placeholders): env OPENAI_API_KEY= RAPID_API_KEY= tool_sandbox --user GPT_4_o_2024_05_13 --agent GPT_4_o_2024_05_13

Where can I get ToolSandbox?

The source code and installation instructions are on GitHub: https://github.com/apple-aiml-research/ToolSandbox

apple-aiml-research/ToolSandbox

A stateful, conversational benchmark for evaluating how LLM agents use tools, with a built-in user simulator and milestone-based scoring.

View on GitHub

Stars: 288
Forks: 33
Open issues: 3
Last commit: last month

What is it?

ToolSandbox is an evaluation benchmark for large language model tool use, released alongside an Apple research paper. Unlike benchmarks that test stateless REST APIs with a single prompt, it runs tools against a persistent world state (settings, contacts, messages and reminders), so a tool call can change what later calls return.

The framework includes a user simulator that holds a multi-turn conversation with the agent, and a set of Python tools with implicit dependencies between them, such as needing to enable cellular service before sending a message. Scenarios are defined in Python as extensions of a base device state, and tools can be augmented with distractors, renamed or stripped of descriptions to test robustness.

Evaluation is milestone-based: each scenario defines a DAG of required intermediate and final states, and scoring checks whether a trajectory reaches them in a valid order rather than matching one exact sequence of calls. The project provides a `tool_sandbox` CLI and supports models from OpenAI, Anthropic, Google Gemini, Cohere, and open-source models served through vLLM.

Key features

01Stateful tool execution against a persistent world state of settings, contacts, messages and reminders.
02Built-in user simulator for on-policy, multi-turn conversational evaluation.
03Milestone DAG scoring that accepts any valid ordering of intermediate steps.
04Scenario augmentations such as distractor tools, renamed tools and removed type hints.
05Supports OpenAI, Anthropic, Gemini, Cohere and vLLM-hosted open-source models.

Use cases

Comparing how different LLMs handle multi-step tool use with state dependencies.
Testing whether an agent recognizes when it lacks the tools or information to finish a task.
Building regression tests for an agent's tool-calling behavior in conversational settings.

Quick start

1
Create a conda environment with Python 3.9
conda create -n ToolSandbox python=3.9
2
Activate the environment
conda activate ToolSandbox
3
Install the project and its dev dependencies from the repository root
pip install '.[dev]'
4
Run a single scenario with GPT-4o as user simulator and agent (replace the API key placeholders)
env OPENAI_API_KEY=<YOUR_OPENAI_API_KEY> RAPID_API_KEY=<YOUR_RAPID_API_KEY> tool_sandbox --user GPT_4_o_2024_05_13 --agent GPT_4_o_2024_05_13

Things to know

The license is not clearly specified in the metadata, so reuse terms are unclear.
Running scenarios with hosted models requires paid API keys, and search scenarios also need a RapidAPI key.
Tool calls execute directly on the host machine without a sandbox.

Topics

#llm-evaluation #benchmark #tool-use #function-calling #agents #python

FAQ

What is ToolSandbox?: A stateful, conversational benchmark for evaluating how LLM agents use tools, with a built-in user simulator and milestone-based scoring. ToolSandbox is an evaluation benchmark for large language model tool use, released alongside an Apple research paper. Unlike benchmarks that test stateless REST APIs with a single prompt, it runs tools against a persistent world state (settings, contacts, messages and reminders), so a tool call can change what later calls return.
Is ToolSandbox free to use? What license does it have?: ToolSandbox is open source under the Other license. Check the license terms before using it in commercial products.
Is ToolSandbox still maintained?: Updated within 6 months. The last commit to ToolSandbox was on Sep 11, 2026.
What can I use ToolSandbox for?: Comparing how different LLMs handle multi-step tool use with state dependencies. Testing whether an agent recognizes when it lacks the tools or information to finish a task. Building regression tests for an agent's tool-calling behavior in conversational settings.
How do I install ToolSandbox?: Create a conda environment with Python 3.9: conda create -n ToolSandbox python=3.9 Activate the environment: conda activate ToolSandbox Install the project and its dev dependencies from the repository root: pip install '.[dev]' Run a single scenario with GPT-4o as user simulator and agent (replace the API key placeholders): env OPENAI_API_KEY=<YOUR_OPENAI_API_KEY> RAPID_API_KEY=<YOUR_RAPID_API_KEY> tool_sandbox --user GPT_4_o_2024_05_13 --agent GPT_4_o_2024_05_13
Where can I get ToolSandbox?: The source code and installation instructions are on GitHub: https://github.com/apple-aiml-research/ToolSandbox

Summary generated by AI from the project's README. GitHub stats refresh daily.