OpenViSTAI · Agents · Blockchain · Crypto
Submit a project
Menu
AI Evaluation & Observability

Bergen

BERGEN is a library for benchmarking question-answering retrieval-augmented generation (RAG) systems. It configures retrievers, rerankers, datasets and language-model generators, runs experiments, and calculates evaluation metrics, with support for multilingual experiments and pairwise LLM-as-judge comparisons.

Explore the practical guide ↓

Independent OpenViST project profile, based on linked sources.

Bergen — OpenViST project profile in AI Evaluation & Observability.
OpenViST project illustration. See the linked sources for product details.
Understand the project

Practical overview

When it fits

Use BERGEN when you need a configurable benchmark for RAG question-answering pipelines and want to compare component choices, datasets or languages.

Who it is for

  • Researchers comparing RAG methods on question-answering datasets.
  • Developers evaluating how retrieval, reranking and generation choices affect a RAG pipeline.
  • Teams running multilingual RAG experiments.

Documented uses

  • Run reproducible experiments with configurable RAG pipeline components.
  • Compare retriever, reranker and generator combinations over datasets.
  • Calculate Match, EM or LLMEval metrics for experiment results.
  • Make pairwise experiment comparisons using an LLM as judge.
Before you choose

Requirements & limits

BERGEN is released under the Creative Commons Attribution-NonCommercial-ShareAlike 4.0 license.

  • Experiments are configured through YAML files or command-line parameters.
  • The README reports support for more than 20 retrievers, four rerankers and more than 20 language models.
  • Evaluation options include Match, EM, LLMEval and LLM-judged pairwise comparisons.
  • The repository provides separate installation, configuration, evaluation and multilingual guides.
  • The stated license is CC BY-NC-SA 4.0.

Check current requirements in the official documentation. Source-based guidance does not mean OpenViST has run or independently tested this project.

Source-based practical guide

Run and evaluate a BERGEN RAG benchmark

Configure a question-answering RAG experiment, run it, and calculate LLMEval metrics from the experiment folder.

Before you start

  • BERGEN repository and Python 3, as reflected by the documented python3 commands.
  • A configured retriever, reranker, generator, and dataset; the README provides example names.
  • An experiments/ folder containing experiment results for evaluation.

Steps

  1. Run an experiment with the documented example configuration: the command below.

    Documented command
    python3 bergen.py retriever="bm25" reranker="minilm6" generator='tinyllama-chat' dataset='kilt_nq'

  2. Evaluate experiment results with the command below.

    Documented command
    python3 evaluate.py --experiments_folder experiments/ --llm_batch_size 16 --split 'dev' --llm vllm_SOLAR-107B

  3. Optionally parse experiment files into a dataframe with the command below.

    Documented command
    python print_results.py --folder experiments/ --format=tiny

Expected outcome

A BERGEN RAG experiment is run, LLMEval metrics are calculated and printed, and the optional command parses experiment files into a dataframe.

Guide limitations

  • The README points to a separate installation guide for detailed installation steps.
  • Component and dataset names in the example are examples; the README directs users to the configuration guide for full configuration.
  • The evaluation command assumes experiment results are available in the specified folder and names an evaluation LLM.
Documentation 1 ↗Documentation 2 ↗
COMPARE OPTIONS

Compare options for Retrieval-augmented applications

These projects document a common task and share a project role. This is a source-based comparison, not an execution benchmark or a guarantee of interchangeability.

ProjectDocumented purposePlatformAccess and requirements
BergenRun reproducible experiments with configurable RAG pipeline components.CLIBERGEN is released under the Creative Commons Attribution-NonCommercial-ShareAlike 4.0 license.
SentrevCompare retrieval performance across embedding models on a document collection.CLIThe project is an open-source Python package under the MIT license. Its README documents installation with pip or from source.
DiragnosisGenerate synthetic evaluation questions from supplied documents.Docker, Linux, macOS, Self-hostedThe repository identifies the software as open source under the MIT license. The README documents a local Docker application and a pip-installable Python package.
Evidence and transparency

Sources & review

Official repository ↗Guide documentation ↗
Source record details
Source availability
Active
Sources checked
16 October 2026
Original website title
naver/bergen · Official homepage title · retained in its source language
How we review sourcesSuggest a correction