OpenViSTAI · Agents · Blockchain · Crypto
Submit a project
Menu
AI Evaluation & Observability

Nervaluate

Nervaluate is a Python library for evaluating named-entity recognition predictions against labeled references using entity-level metrics based on SemEval 2013 Task 9.1.

Explore the practical guide ↓

Independent OpenViST project profile, based on linked sources.

Nervaluate — OpenViST project profile in AI Evaluation & Observability.
OpenViST project illustration. See the linked sources for product details.
Understand the project

Practical overview

When it fits

Use it after generating NER predictions when token-level scores do not show enough about complete entity boundaries, partial matches or entity-type errors.

Who it is for

  • Developers and researchers evaluating named-entity recognition models.
  • Teams comparing NER predictions with labeled reference sequences.

Documented uses

  • Assess whether predicted entities match reference spans and types.
  • Review precision, recall and F1 alongside partial, missed and spurious entity counts.
  • Break down evaluation results by entity type.
Before you choose

Requirements & limits

The library is open source under the MIT license. The README documents installation with pip.

  • The documented input example consists of sentence-level lists of token labels, with reference and predicted sequences.
  • Evaluation requires specifying the entity types through the tags argument in the example.
  • Reports distinguish scenarios and include entity counts as well as precision, recall and F1-score.

Check current requirements in the official documentation. Source-based guidance does not mean OpenViST has run or independently tested this project.

Source-based practical guide

Evaluate NER label sequences

Generate an entity-level summary report from reference and predicted NER labels.

Before you start

  • Python environment with nervaluate installed using pip.
  • Reference and predicted labels arranged as lists of token labels, with each list corresponding to a sentence.
  • The entity types to evaluate.

Steps

  1. Install the library with the command below.

    Documented command
    pip install nervaluate

  2. Import Evaluator from nervaluate.evaluator.

  3. Initialize Evaluator with the true labels, predicted labels, the entity types in tags, and loader="list".

  4. Call evaluator.summary_report() to display the report, or use evaluator.summary_report(mode='entities') to group results by entity type.

Expected outcome

A report showing evaluation scenarios and measures including precision, recall, F1-score, and counts for correct, incorrect, partial, missed and spurious entities.

Guide limitations

  • The documented example uses token-label sequences and requires both reference and prediction labels.
  • The guide covers the README's list-input example; it does not establish support for other input formats.
Documentation 1 ↗
Evidence and transparency

Sources & review

Official repository ↗
Source record details
Source availability
Active
Sources checked
14 October 2026
Original website title
MantisAI/nervaluate · Official homepage title · retained in its source language
How we review sourcesSuggest a correction