OpenViSTAI · Agents · Blockchain · Crypto
Submit a project
Menu
AI Evaluation & Observability

Jury

Jury is an open-source Python toolkit for evaluating NLP experiments with automated metrics. It provides a unified input structure for predictions and references, can calculate multiple metrics concurrently, and supports custom metrics through class extension.

Explore the practical guide ↓

Independent OpenViST project profile, based on linked sources.

Jury — OpenViST project profile in AI Evaluation & Observability.
OpenViST project illustration. See the linked sources for product details.
Understand the project

Practical overview

When it fits

Use Jury when predictions and reference texts are available and you want automated NLP scores through Python or a command-line workflow, including runs with multiple metrics or multiple references.

Who it is for

  • NLP researchers comparing experiment outputs against reference texts.
  • Developers who want a Python API or CLI for applying evaluation metrics.
  • Teams that need to add project-specific metrics to an evaluation workflow.

Documented uses

  • Score generated text against one or more references.
  • Calculate multiple evaluation metrics in one workflow.
  • Evaluate multiple experiments from paired files or folders.
  • Load supported metrics from Jury or Hugging Face evaluate.
  • Extend the toolkit with a custom metric.
Before you choose

Requirements & limits

The public GitHub repository identifies the project as MIT-licensed open-source software and documents installation through pip or from source.

  • The toolkit accepts predictions and references through its Python API; its CLI pairs file entries by line or pairs files in folders by matching filenames.
  • Metric selection can be specified when creating a scorer, and the CLI can use a JSON configuration to define metrics.
  • The loader can fall back to Hugging Face evaluate for metrics not available in Jury.
  • The README warns that some metrics may malfunction on Windows due to sacrebleu and pywin32; it recommends conda installation of pywin32 if that package causes trouble.

Check current requirements in the official documentation. Source-based guidance does not mean OpenViST has run or independently tested this project.

Source-based practical guide

Evaluate paired prediction and reference files with the CLI

Calculate and export metric scores from paired text files.

Before you start

  • Install Jury.
  • Prepare prediction and reference files with corresponding entries on each line.

Steps

  1. Provide the predictions and references file paths to jury eval.

  2. Optionally specify a reduction function and an export path.

  3. Run the command with the desired options, such as jury eval --predictions /path/to/predictions.txt --references /path/to/references.txt --reduce_fn max --export /path/to/export.txt.

Expected outcome

Jury evaluates the paired files and can write the results to the specified export path.

Guide limitations

  • Prediction and reference entries must be paired line by line.
  • The supplied command-line example describes options but does not specify a default metric configuration.
Documentation 1 ↗
Source-based practical guide

Score predictions and references with the Python API

Calculate selected automated metrics from prediction and reference lists.

Before you start

  • Install Jury.
  • Prepare prediction and reference lists for evaluation.

Steps

  1. Import Jury from the jury package.

  2. Create a scorer, optionally selecting metrics such as bleu and meteor.

  3. Call the scorer with predictions and references.

Expected outcome

The scorer returns metric scores for the supplied predictions and references.

Guide limitations

  • The documented example supplies text predictions and references; it does not describe a hosted interface.
Documentation 1 ↗
Evidence and transparency

Sources & review

Official repository ↗
Source record details
Source availability
Active
Sources checked
15 October 2026
Original website title
obss/jury · Official homepage title · retained in its source language
How we review sourcesSuggest a correction