OpenViSTAI · Agents · Blockchain · Crypto
Submit a project
Menu
AI Evaluation & Observability

Evret

Evret is an open-source Python framework for evaluating information-retrieval systems, especially retrievers used in search, recommendation and RAG pipelines. It calculates standard ranking metrics and uses configurable judges to match retrieved text against expected answers.

Explore the practical guide ↓

Independent OpenViST project profile, based on linked sources.

Evret — OpenViST project profile in AI Evaluation & Observability.
OpenViST project illustration. See the linked sources for product details.
Understand the project

Practical overview

When it fits

Use Evret during retrieval-system development when you have a retriever and representative evaluation queries with expected answers, and need metric-based comparisons.

Who it is for

  • Developers building search, recommendation or RAG systems who need to evaluate retrieval quality.
  • Teams comparing retriever configurations against queries and expected answers.
  • Python practitioners integrating retrieval evaluation with Qdrant or LangChain.

Documented uses

  • Measure whether relevant documents appear in a retriever’s top results.
  • Compare retrieval configurations using selected ranking metrics.
  • Judge retrieved text against expected answers using token overlap, semantic similarity or an LLM judge.
  • Summarize or export evaluation results as JSON or CSV.
Before you choose

Requirements & limits

The README identifies Evret as an MIT-licensed open-source Python project and specifies Python 3.10 or newer.

  • Choose metrics according to the retrieval behavior you want to assess: for example, MRR for the rank of the first relevant result or Recall for coverage of relevant documents.
  • Evaluation depends on the quality of the query set and expected answers supplied to the dataset.
  • TokenOverlapJudge is the default; semantic and LLM judges are alternatives, and the README shows an LLM judge configured with an OpenAI provider and model.
  • The basic package is installed with pip; integrations such as Qdrant and LangChain are shown as optional installation extras.

Check current requirements in the official documentation. Source-based guidance does not mean OpenViST has run or independently tested this project.

Source-based practical guide

Run a retrieval evaluation from JSON data

Evaluate a retriever using selected metrics and export the resulting scores.

Before you start

  • Python 3.10 or newer
  • Evret installed with pip
  • A retriever callable by Evret
  • An evaluation dataset in Evret’s JSON format

Steps

  1. Install Evret with the command below.

    Documented command
    pip install evret

  2. Load the evaluation set with EvaluationDataset.from_json("eval_data.json").

  3. Create an Evaluator with your retriever, chosen metrics and a judge; the README example uses HitRate, MRR, NDCG and TokenOverlapJudge.

  4. Call evaluator.evaluate(dataset), summarize the results, and optionally export them with to_json or to_csv.

Expected outcome

A summary of the selected retrieval metrics, with optional JSON or CSV results files.

Guide limitations

  • The example requires an existing retriever and evaluation data; the guide does not configure either one.
  • The README lists optional integrations separately; install the relevant extra when needed.
Documentation 1 ↗
COMPARE OPTIONS

Compare options for Retrieval-augmented applications

These projects document a common task and share a project role. This is a source-based comparison, not an execution benchmark or a guarantee of interchangeability.

ProjectDocumented purposePlatformAccess and requirements
EvretMeasure whether relevant documents appear in a retriever’s top results.CLIThe README identifies Evret as an MIT-licensed open-source Python project and specifies Python 3.10 or newer.
SentrevCompare retrieval performance across embedding models on a document collection.CLIThe project is an open-source Python package under the MIT license. Its README documents installation with pip or from source.
DiragnosisGenerate synthetic evaluation questions from supplied documents.Docker, Linux, macOS, Self-hostedThe repository identifies the software as open source under the MIT license. The README documents a local Docker application and a pip-installable Python package.
Evidence and transparency

Sources & review

Official repository ↗
Source record details
Source availability
Active
Sources checked
15 October 2026
Original website title
kaivid-labs/evret · Official homepage title · retained in its source language
How we review sourcesSuggest a correction