OpenViSTAI · Agents · Blockchain · Crypto
Submit a project
Menu
Agent Evaluation & Observability

Osworld V2

Osworld V2 is a Python benchmark project for evaluating computer-use agents on long-horizon desktop tasks. It provides task definitions and assets, environment-provider setup, and links to a trajectory viewer and downloadable execution data.

Explore the practical guide ↓

Independent OpenViST project profile, based on linked sources.

Osworld V2 — OpenViST project profile in Agent Evaluation & Observability.
OpenViST project illustration. See the linked sources for product details.
Understand the project

Practical overview

When it fits

Use OSWorld V2 when you can provision its evaluation environment and need to assess GUI agents against consistent, extended desktop tasks. Keep code, task files, assets, websites and provider images aligned to one release for reproducible evaluation.

Who it is for

  • Researchers comparing computer-use agents on long-horizon tasks
  • Agent developers who need a structured desktop-task evaluation environment
  • Teams reviewing agent trajectories or building task collections from benchmark data

Documented uses

  • Evaluate an agent against desktop tasks that span multiple steps.
  • Compare evaluations using matching benchmark releases.
  • Inspect execution trajectories through the linked viewer or downloaded data.
  • Build task collections from the linked benchmark dataset.
Before you choose

Requirements & limits

  • The documented v2.1 setup requires Python 3.12 or later and uses uv.
  • Docker is documented for Linux servers, especially those with KVM support; AWS is documented for large-scale parallel evaluation.
  • The project supplies provider images for Docker and AWS; other provider code paths may require image migration.
  • Some v2.1 tasks require self-hosting mocked websites, and GitLab-backed tasks require a self-hosted GitLab instance.
  • The repository recommends the tagged osworld-v2.1 release and matching task, asset, website and image versions.

Check current requirements in the official documentation. Source-based guidance does not mean OpenViST has run or independently tested this project.

Source-based practical guide

Prepare an OSWorld V2.1 evaluation environment

Set up the benchmark code and download the release-matched task files and assets.

Before you start

  • Python 3.12 or later
  • The uv package manager
  • A Linux host with Docker, especially one with KVM support, or an AWS environment
  • The OSWorld-V2 repository and matching v2.1 release components

Steps

  1. Clone the repository at the osworld-v2.1 branch and enter the checkout.

  2. Install the host dependencies with the command below.

    Documented command
    uv sync --frozen

  3. Download the task files with the command below.

    Documented command
    uv run scripts/tools/download_osworld_v2_tasks.py --benchmark-release osworld-v2.1

  4. Download the gated assets to a fresh directory with the documented asset download script, then set OSWORLD_FILE_BASE_URL to that directory and OSWORLD_BENCHMARK_RELEASE to osworld-v2.1.

Expected outcome

The code, task files and gated assets are set to the same benchmark release, ready for the remaining provider and runner configuration.

Guide limitations

  • This covers package and benchmark-data setup, not a completed evaluation.
  • A provider image and runner configuration are still needed; the project provides images for Docker and AWS.
  • For reproducible results, keep all release-controlled components on the same release.
Documentation 1 ↗
Evidence and transparency

Sources & review

Official repository ↗
Source record details
Source availability
Active
Sources checked
8 October 2026
Original website title
xlang-ai/OSWorld-V2 · Official homepage title · retained in its source language
How we review sourcesSuggest a correction