OpenViSTAI · Agents · Blockchain · Crypto
Submit a project
Menu
AI Evaluation & Observability

Audiobench

Audiobench is a Python research benchmark for evaluating audio large language models across more than 50 datasets covering speech, audio-scene and voice-understanding tasks. Its repository provides evaluation code and task-specific metrics, and links to a Hugging Face leaderboard and dataset organization.

Explore the practical guide ↓

Independent OpenViST project profile, based on linked sources.

Audiobench — OpenViST project profile in AI Evaluation & Observability.
OpenViST project illustration. See the linked sources for product details.
Understand the project

Practical overview

When it fits

Use Audiobench when evaluating an audio language model across task-specific datasets and metrics, or when you want to compare results with the project’s leaderboard.

Who it is for

  • Researchers comparing audio language models across multiple benchmark tasks.
  • Teams evaluating speech recognition, speech translation or spoken question-answering performance.

Documented uses

  • Measure speech-recognition performance using metrics such as word error rate.
  • Compare speech-translation results using BLEU on listed datasets.
  • Evaluate speech question-answering with model-as-judge metrics.
  • Review comparative results through the linked leaderboard.
Before you choose

Requirements & limits

  • The benchmark covers more than 50 datasets, with task and metric details listed in the repository.
  • The README documents a local Python setup using the repository requirements file.
  • The model-as-judge example uses vLLM and specifies one 80 GB GPU for the judge server plus a second GPU for inference.

Check current requirements in the official documentation. Source-based guidance does not mean OpenViST has run or independently tested this project.

Source-based practical guide

Run the documented model-as-judge example

Evaluate Qwen2-Audio-7B-Instruct on the cn_college_listen_mcq_test dataset using the Llama-3-70B judge metric.

Before you start

  • Install the repository requirements with pip.
  • Use vLLM to serve Llama-3-70B-Instruct; the example specifies an int4-quantized model and one 80 GB GPU for this service.
  • Provide a second GPU for model inference.

Steps

  1. Install dependencies with the command below.

    Documented command
    pip install -r requirements.txt

  2. Start the judge service with the command below.

    Documented command
    bash vllm_model_judge_llama_3_70b.sh

  3. Set the example evaluation values: GPU=2, BATCH_SIZE=1, OVERWRITE=True, NUMBER_OF_SAMPLES=-1, MODEL_NAME=Qwen2-Audio-7B-Instruct, DATASET=cn_college_listen_mcq_test, and METRICS=llama3_70b_judge.

  4. Run the command below.

    Documented command
    bash eval.sh $DATASET $MODEL_NAME $GPU $BATCH_SIZE $OVERWRITE $METRICS $NUMBER_OF_SAMPLES

Expected outcome

The example is configured to run model inference and obtain evaluation results for the selected dataset and judge metric.

Guide limitations

  • The README specifies one 80 GB GPU for hosting the judge and a second GPU for inference.
  • This recipe describes the README's example configuration; it does not establish that other hardware or configurations will work.
Documentation 1 ↗
Evidence and transparency

Sources & review

Official repository ↗
Source record details
Source availability
Active
Sources checked
15 October 2026
Original website title
AudioLLMs/AudioBench · Official homepage title · retained in its source language
How we review sourcesSuggest a correction