OpenViSTAI · Agents · Blockchain · Crypto
Submit a project
Menu
Agent Evaluation & Observability

Agentbench

AgentBench is an open-source benchmark for evaluating language models as agents across interactive task environments. Its current version, AgentBench FC, tests function-calling behavior and is integrated with AgentRL, a multitask and multi-turn agent reinforcement-learning framework.

Explore the practical guide ↓

Independent OpenViST project profile, based on linked sources.

Agentbench — OpenViST project profile in Agent Evaluation & Observability.
OpenViST project illustration. See the linked sources for product details.
Understand the project

Practical overview

When it fits

Use AgentBench FC when you need a documented benchmark spanning multiple agent task environments and can accommodate its Docker-based setup. The older v0.1 and v0.2 releases remain available for experiments based on the original benchmark.

Who it is for

  • Researchers comparing language models on agent tasks.
  • Developers evaluating function-calling agents across multiple task environments.
  • Teams exploring benchmark results through the published leaderboard.

Documented uses

  • Compare models on function-calling agent tasks.
  • Evaluate agent behavior across multitask and multi-turn environments.
  • Review reported test-set results on the benchmark leaderboard.
Before you choose

Requirements & limits

The repository is public and identified as Apache-2.0 licensed.

  • The current main repository describes AgentBench FC, while v0.1 and v0.2 contain the older benchmark design.
  • The documented setup requires task-specific Docker images and Freebase data for Knowledge Graph.
  • WebShop needs about 16 GB RAM to start; ALFWorld workers may need restarting because of memory and disk growth.
  • The README links to a leaderboard containing reported test-set results.

Check current requirements in the official documentation. Source-based guidance does not mean OpenViST has run or independently tested this project.

Source-based practical guide

Prepare the original AgentBench environment for dbbench-std and os-std

Set up the documented Python environment, agent configuration, and task images for the original `dbbench-std` and `os-std` quick start.

Before you start

  • A cloned AgentBench repository.
  • Conda and Python 3.9; the README recommends Python 3.9 for its pinned scientific Python dependencies.
  • Docker installed and working.
  • An OpenAI API key to configure the documented OpenAI agent.

Steps

  1. Create and activate the Python environment, then install dependencies: conda create -n agent-bench python=3.9, conda activate agent-bench, and the command below.

    Documented command
    pip install -r requirements.txt

  2. Check Docker with the command below, then pull the MySQL and Ubuntu images and build the three local-os images using the commands in the README.

    Documented command
    docker ps

  3. Add the OpenAI API key in configs/agents/openai-chat.yaml.

  4. Optionally check agent configuration with the command below, or use the documented config and agent arguments to select an agent.

    Documented command
    python -m src.client.agent_test

Expected outcome

The Python dependencies, Docker images, and OpenAI agent configuration needed for the documented original-version `dbbench-std` and `os-std` quick start are prepared.

Guide limitations

  • This covers preparation and agent configuration, not launching or completing benchmark tasks.
  • The README notes that older scientific Python dependencies are pinned and recommends Python 3.9.
  • The documented task-server startup assumes ports 5000 through 5015 are available.
Documentation 1 ↗
Evidence and transparency

Sources & review

Official repository ↗
Source record details
Source availability
Active
Sources checked
7 October 2026
Original website title
THUDM/AgentBench · Official homepage title · retained in its source language
How we review sourcesSuggest a correction