OpenViSTAI · Agents · Blockchain · Crypto
Submit a project
Menu
AI Evaluation & Observability

Nlg Eval

Nlg Eval is a Python library and command-line tool for calculating automated natural-language-generation metrics from generated text and aligned reference examples.

Explore the practical guide ↓

Independent OpenViST project profile, based on linked sources.

Nlg Eval — OpenViST project profile in AI Evaluation & Observability.
OpenViST project illustration. See the linked sources for product details.
Understand the project

Practical overview

When it fits

Use it when generated outputs can be paired row by row with reference examples and you want automated metric reports through a CLI or Python workflow.

Who it is for

  • Researchers evaluating natural-language-generation experiments
  • Developers comparing machine-translation or dialogue-generation outputs with reference text

Documented uses

  • Calculate automated metric scores for a corpus of generated sentences.
  • Compare an individual generated sentence with one or more reference strings through the Python API.
  • Reuse loaded models for repeated evaluations in a script.
Before you choose

Requirements & limits

  • Reference and hypothesis rows must correspond to the same examples.
  • Setup requires Java 1.8 or higher and downloads data such as models and embeddings.
  • The metric set includes BLEU, METEOR, ROUGE, CIDEr, SPICE, SkipThought cosine similarity, Embedding Average cosine similarity, Vector Extrema cosine similarity and Greedy Matching.

Check current requirements in the official documentation. Source-based guidance does not mean OpenViST has run or independently tested this project.

Source-based practical guide

Set up Nlg Eval for command-line evaluation

Install the library and download its required metric data and external code files.

Before you start

  • Java 1.8.0 or higher
  • Python and the project's Python dependencies
  • A Bash-compatible terminal for the documented command-line workflow

Steps

  1. Install the Python package from the project's GitHub repository using the documented pip command.

  2. Run nlg-eval --setup to download required data and external code files.

  3. Prepare hypothesis and reference files with corresponding examples on the same rows.

  4. Run the documented nlg-eval command with the hypothesis file and one or more reference files.

Expected outcome

The command-line tool reports metric values for the aligned generated and reference examples.

Guide limitations

  • The README requires Java 1.8.0 or higher.
  • On macOS High Sierra or later, the README specifies an environment setting to allow multithreading.
  • Windows users outside a Bash terminal may need to locate the nlg-eval script.
Documentation 1 ↗
Evidence and transparency

Sources & review

Official repository ↗
Source record details
Source availability
Active
Sources checked
18 October 2026
Original website title
Maluuba/nlg-eval · Official homepage title · retained in its source language
How we review sourcesSuggest a correction