Practical overview
When it fits
Use AutoPenBench when you need a container-based setting for evaluating an agent on penetration-testing tasks, or when you want to contribute new vulnerable machine scenarios.
Who it is for
- Researchers evaluating generative agents on penetration-testing tasks
- Developers creating or adapting vulnerable machine scenarios for a benchmark
Documented uses
- Evaluate an agent using the repository’s benchmark and its example agent notebook.
- Adapt a free-text agent’s responses to the JSON schemas expected by the benchmark tools.
- Create a machine scenario and define its task, flag, solution commands and milestones.
Requirements & limits
The project is a public GitHub repository identified as MIT-licensed.
- The example agent uses Instructor with structured JSON output based on Pydantic schemas.
- Free-text agents need an adapter to translate their output into the benchmark tools’ schemas.
- The installation instructions require CMake and use a Python virtual environment.
Check current requirements in the official documentation. Source-based guidance does not mean OpenViST has run or independently tested this project.
Set up the benchmark environment
Install the repository requirements and set up its machines.
Before you start
- A local machine with CMake installed
- Python 3
Steps
Create and activate a Python virtual environment with the command below and
source .venv/bin/activate.python3 -m venv .venvRun the command below to install requirements and set up the machines.
make install
Expected outcome
The README’s installation stage is completed, with requirements installed and machines set up.
Guide limitations
- These steps describe installation, not running or evaluating an agent.
Create a benchmark machine template
Initialize the files for a new machine scenario.
Before you start
- A terminal in the repository environment set up for the project
Steps
Run the command below, replacing the placeholders with the desired level, category and integer machine identifier.
make create [LEVEL] [CATEGORY] [MACHINE_ID]Customize the generated machine files and Docker Compose configuration.
Add the task, flag, target and vulnerability information to
data/games.json.Write the solution command sequence and command and stage milestone files.
Expected outcome
The repository provides template files for a machine scenario and specifies the task and milestone data to customize.
Guide limitations
- The README describes the creation and customization workflow but does not provide a complete scenario-specific configuration here.
Sources & review
Source record details
- Source availability
- Active
- Sources checked
- 17 October 2026
- Original website title
- lucagioacchini/auto-pen-bench · Official homepage title · retained in its source language