OpenViSTAI · Agents · Blockchain · Crypto
Submit a project
Menu
AI Evaluation & Observability

Mlmm Evaluation

Mlmm Evaluation is a Python toolkit containing multilingual benchmark datasets and command-line evaluation scripts for assessing large language models on ARC, HellaSwag and MMLU across 26 languages.

Explore the practical guide ↓

Independent OpenViST project profile, based on linked sources.

Mlmm Evaluation — OpenViST project profile in AI Evaluation & Observability.
OpenViST project illustration. See the linked sources for product details.
Understand the project

Practical overview

When it fits

Use it when a research evaluation needs the three included benchmarks in one of the toolkit’s 26 supported languages and the model can be supplied by path to its evaluation script.

Who it is for

  • Researchers evaluating multilingual large language models.
  • Teams comparing model performance across supported languages on multiple-choice benchmarks.

Documented uses

  • Evaluate a model on ARC, HellaSwag and MMLU for a selected language.
  • Compare multilingual model performance across the 26 listed languages.
  • Prepare a research benchmarking workflow using the included dataset download and evaluation scripts.
Before you choose

Requirements & limits

The repository code is Apache 2.0 licensed. The datasets are CC BY-NC 4.0 and designated for research use only.

  • The 26-language set includes Russian, German, Chinese, French, Spanish, Vietnamese, Arabic, Hindi and several South Asian languages; check the listed set for a required language.
  • The benchmarks cover distinct multiple-choice tasks: science questions, grounded commonsense inference and broad subject knowledge.
  • The multilingual datasets were translated from English using ChatGPT, which matters when interpreting results.
  • The code and datasets have different licenses; dataset use is restricted to non-commercial research purposes.
  • The README says the framework largely inherits from lm-evaluation-harness and requests citation of that project.

Check current requirements in the official documentation. Source-based guidance does not mean OpenViST has run or independently tested this project.

Source-based practical guide

Install and run a multilingual benchmark evaluation

Prepare the repository datasets and invoke its evaluation script for a model and language.

Before you start

  • Python environment with pip
  • A model path accessible to the evaluation script
  • Repository cloned from https://github.com/nlp-uoregon/mlmm-evaluation

Steps

  1. Clone the repository and change into its directory.

  2. Install the project with the command below.

    Documented command
    pip install -e ".[multilingual]"

  3. Download the evaluation datasets with the command below.

    Documented command
    bash scripts/download.sh

  4. Run the command below, substituting a language code and model path.

    Documented command
    bash scripts/run.sh [LANG] [YOUR-MODEL-PATH]

Expected outcome

The repository’s script is invoked to evaluate the selected model on ARC, HellaSwag and MMLU for the specified language.

Guide limitations

  • The datasets and framework are intended for research use; datasets are licensed CC BY-NC 4.0 for non-commercial use.
  • The multilingual datasets are translated from the original English datasets using ChatGPT.
Documentation 1 ↗
Evidence and transparency

Sources & review

Official repository ↗
Source record details
Source availability
Active
Sources checked
15 October 2026
Original website title
nlp-uoregon/mlmm-evaluation · Official homepage title · retained in its source language
How we review sourcesSuggest a correction