OpenViSTAI · Agents · Blockchain · Crypto
Submit a project
Menu
AI Evaluation & Observability

Continuous Eval

Continuous Eval is an open-source Python package for data-driven evaluation of applications powered by large language models. It supplies metrics for retrieval-augmented generation, code generation, agent tool use, classification and other LLM use cases, and supports evaluating individual pipeline modules.

Explore the practical guide ↓

Independent OpenViST project profile, based on linked sources.

Continuous Eval — OpenViST project profile in AI Evaluation & Observability.
OpenViST project illustration. See the linked sources for product details.
Understand the project

Practical overview

When it fits

Use it when you can provide evaluation data and want repeatable measurements for an LLM application, especially when separate pipeline stages need their own metrics.

Who it is for

  • Developers building or maintaining LLM-powered applications.
  • Teams evaluating retrieval-augmented generation systems and other multi-stage pipelines.

Documented uses

  • Measure retrieval quality against ground-truth context.
  • Compare application outputs using deterministic, semantic, LLM-based or probabilistic metrics.
  • Attach different evaluation metrics and tests to individual modules in a pipeline.
  • Check evaluation results against configured metric thresholds.
Before you choose

Requirements & limits

The repository describes the package as open source and identifies its license as Apache-2.0.

  • The library is Python-based and distributed as a PyPI package.
  • LLM-based metrics require at least one LLM API key in a .env file.
  • The repository also documents installation from source with Poetry and all extras.
  • The examples use datasets, metric configuration and optional threshold tests; application-specific data and pipeline inputs must be supplied or adapted.

Check current requirements in the official documentation. Source-based guidance does not mean OpenViST has run or independently tested this project.

Source-based practical guide

Set up a retrieval evaluation pipeline

Configure a dataset-based evaluation with retrieval metrics and a context-recall threshold test.

Before you start

  • Continuous Eval installed.
  • A retrieval dataset with retrieved contexts and ground-truth contexts.

Steps

  1. Load a retrieval dataset, using the documented example data downloader or a suitable dataset.

  2. Create a SingleModulePipeline and attach PrecisionRecallF1 and RankedRetrievalMetrics to the retrieved and ground-truth contexts.

  3. Add a GreaterOrEqualThan test for the context_recall metric with a chosen minimum value.

  4. Pass the pipeline to EvaluationRunner, call evaluate(), and then call test() on the results.

Expected outcome

The example workflow produces aggregated evaluation results and test results for the retrieval metrics.

Guide limitations

  • The example notes that the script should run in a new process to avoid multiprocessing issues.
  • The example uses a minimum context-recall value of 0.8; it is a configured test threshold, not a general quality guarantee.
Documentation 1 ↗
COMPARE OPTIONS

Compare options for AI coding assistance

These projects document a common task and share a project role. This is a source-based comparison, not an execution benchmark or a guarantee of interchangeability.

ProjectDocumented purposePlatformAccess and requirements
Continuous EvalMeasure retrieval quality against ground-truth context.CloudThe repository describes the package as open source and identifies its license as Apache-2.0.
FactoolAssess claims and response-level factuality in knowledge-based answers.CLIThe repository identifies the project as open source under the Apache-2.0 license.
Evidence and transparency

Sources & review

Official repository ↗
Source record details
Source availability
Active
Sources checked
9 October 2026
Original website title
relari-ai/continuous-eval · Official homepage title · retained in its source language
How we review sourcesSuggest a correction