OpenViSTAI · Agents · Blockchain · Crypto
Submit a project
Menu
AI Evaluation & Observability

Mmmu

MMMU is a benchmark project and Python evaluation-code repository for assessing multimodal models on multidisciplinary questions that combine visual interpretation with subject knowledge and reasoning. It includes the standard MMMU benchmark and the more stringent MMMU-Pro variant.

Explore the practical guide ↓

Independent OpenViST project profile, based on linked sources.

Mmmu — OpenViST project profile in AI Evaluation & Observability.
OpenViST project illustration. See the linked sources for product details.
Understand the project

Practical overview

When it fits

Use MMMU when an evaluation needs multidisciplinary questions with varied visual material, or MMMU-Pro when the study calls for stricter testing of intrinsic multimodal understanding.

Who it is for

  • Researchers comparing multimodal model performance across subject areas.
  • Model developers investigating visual understanding and expert-domain reasoning.
  • Teams building research evaluation workflows around shared benchmark datasets.

Documented uses

  • Evaluate a multimodal model on college-level questions using images and domain knowledge.
  • Use development samples for few-shot or in-context learning experiments.
  • Use validation samples to debug models, select hyperparameters and run quick evaluations.
  • Compare results against a more demanding benchmark that reduces text-only shortcuts and tests vision-only input.
Before you choose

Requirements & limits

The public repository specifies an Apache-2.0 license. Evaluation code is provided in Python.

  • MMMU provides development, validation and test splits; the project associates them with in-context learning, debugging and quick evaluation, and local test evaluation respectively.
  • MMMU-Pro filters text-only answerable items, adds plausible answer options and includes a vision-only input setting.
  • The repository identifies its official evaluation pipeline and released test answers as the route for leaderboard-equivalent local test-set evaluation; EvalScope is a separate community-maintained workflow.

Check current requirements in the official documentation. Source-based guidance does not mean OpenViST has run or independently tested this project.

Source-based practical guide

Choose an MMMU split for evaluation work

Select the documented MMMU split appropriate to few-shot work, debugging, hyperparameter selection, or local test-set evaluation.

Before you start

  • Access to the MMMU evaluation code and dataset files.
  • For local test-set evaluation, the released answer_dict_test.json file.

Steps

  1. Use the development set for few-shot or in-context learning.

  2. Use the validation set to debug models, select hyperparameters, or run quick evaluations.

  3. For local test-set evaluation, use answer_dict_test.json with the same pipeline used for the validation set.

Expected outcome

An evaluation workflow matched to the intended development, validation, or released test-set use.

Guide limitations

  • The README identifies 150 development samples, 900 validation samples, and 10,500 test samples.
  • Leaderboard-equivalent test results are tied to the official MMMU evaluation pipeline and released test-set answer files.
Documentation 1 ↗
Evidence and transparency

Sources & review

Official repository ↗
Source record details
Source availability
Active
Sources checked
14 October 2026
Original website title
MMMU-Benchmark/MMMU · Official homepage title · retained in its source language
How we review sourcesSuggest a correction