OpenViSTAI · Agents · Blockchain · Crypto
Submit a project
Menu
AI Evaluation & Observability

Open Rag Eval

Open Rag Eval is an open-source Python toolkit and command-line tool for evaluating retrieval-augmented generation pipelines. It includes metrics that do not require golden answers, optional reference-answer evaluation, and connectors for Vectara, LlamaIndex and LangChain.

Explore the practical guide ↓

Independent OpenViST project profile, based on linked sources.

Open Rag Eval — OpenViST project profile in AI Evaluation & Observability.
OpenViST project illustration. See the linked sources for product details.
Understand the project

Practical overview

When it fits

Use it during RAG development when you need structured evaluation, especially if creating a golden-answer dataset for every test query is impractical.

Who it is for

  • Developers evaluating retrieval-augmented generation systems.
  • Teams comparing RAG configurations or runs.
  • Practitioners who want to measure RAG quality without preparing golden answers for every query.

Documented uses

  • Evaluate RAG results using metrics that do not require golden answers or golden chunks.
  • Run optional evaluation against reference answers when they are available.
  • Review per-query scores and intermediate outputs to investigate results.
  • Compare evaluation results across configurations or runs.
Before you choose

Requirements & limits

The project is open source under the Apache-2.0 license. The documented installation is through pip or from source; using the Vectara connector requires a Vectara account and API credentials.

  • Core UMBRELA and AutoNuggetizer metrics do not require golden answers or golden chunks; reference-answer evaluation is also available.
  • Some metrics use an OpenAI API key, while factual-consistency evaluation has separate HHEM and Vectara API options.
  • The documented Vectara workflow needs a corpus, querying-permission API key, Customer ID and corpus key.
  • The toolkit provides per-query scoring, intermediate outputs and plotting utilities for comparing runs.

Check current requirements in the official documentation. Source-based guidance does not mean OpenViST has run or independently tested this project.

Source-based practical guide

Configure a Vectara RAG evaluation

Prepare a query set and configure the documented Vectara evaluation workflow.

Before you start

  • Python 3.9 or later.
  • A Vectara account, corpus containing indexed data, querying-permission API key, Customer ID and corpus key.
  • An OpenAI API key for the default LLM judge used by some metrics.
  • For factual-consistency evaluation, either a Hugging Face token and access to the Vectara HHEM model, or a Vectara API key for its Factual Consistency API.

Steps

  1. Install the toolkit with the command below.

    Documented command
    pip install open-rag-eval

  2. Create a CSV of evaluation queries with a single column named query.

  3. Edit eval_config_vectara.yaml: set the input query file, generated-answer and result-file values, results folder, and Vectara corpus key in the connector configuration.

  4. Choose evaluator and metric settings, then use the installed command-line tool to run the evaluation.

Expected outcome

The configured workflow runs an evaluation and can produce per-query scores and result artifacts for analysis.

Guide limitations

  • The guide requires credentials and configuration for the selected connector and metrics.
  • The default LLM judge used in some metrics requires an OpenAI API key.
  • HHEM factual-consistency evaluation requires a Hugging Face token and model access; the alternative uses Vectara's Factual Consistency API.
Documentation 1 ↗
COMPARE OPTIONS

Compare options for Retrieval-augmented applications

These projects document a common task and share a project role. This is a source-based comparison, not an execution benchmark or a guarantee of interchangeability.

ProjectDocumented purposePlatformAccess and requirements
Open Rag EvalEvaluate RAG results using metrics that do not require golden answers or golden chunks.ServerThe project is open source under the Apache-2.0 license. The documented installation is through pip or from source; using the Vectara connector requires a Vectara account and API credentials.
SentrevCompare retrieval performance across embedding models on a document collection.CLIThe project is an open-source Python package under the MIT license. Its README documents installation with pip or from source.
DiragnosisGenerate synthetic evaluation questions from supplied documents.Docker, Linux, macOS, Self-hostedThe repository identifies the software as open source under the MIT license. The README documents a local Docker application and a pip-installable Python package.
Evidence and transparency

Sources & review

Official repository ↗
Source record details
Source availability
Active
Sources checked
13 October 2026
Original website title
vectara/open-rag-eval · Official homepage title · retained in its source language
How we review sourcesSuggest a correction