Practical overview
When it fits
Use EvalForge when an AI release process needs policy-based checks and portable evaluation reports from existing metrics or supported evaluator results.
Who it is for
- Developers maintaining RAG applications or AI assistants.
- Teams that want baseline and policy checks for AI evaluation results in CI.
- Maintainers who need to normalize results from supported evaluation frameworks into portable reports.
Documented uses
- Compare candidate evaluation metrics with a baseline.
- Apply explicit thresholds to quality, security, latency or configured cost metrics.
- Import supported promptfoo, Ragas or DeepEval result formats.
- Produce JSON, JUnit, SARIF or Markdown evaluation evidence.
Requirements & limits
The repository describes EvalForge as MIT-licensed open-source software. It documents installation from PyPI and identifies Python 3.9 or later.
- EvalForge complements rather than replaces evaluation frameworks; it handles result normalization, baseline comparisons, policy checks and reporting.
- Supported imports have explicit schema and metadata requirements; check these against the evaluator output in use.
- The built-in RAG evaluator has an offline extractive baseline that does not require a model key or hosted service.
- The README labels the project v0.6.0 alpha.
Check current requirements in the official documentation. Source-based guidance does not mean OpenViST has run or independently tested this project.
Apply a policy gate to a candidate metric summary
Compare a flat JSON candidate summary with a baseline under a versioned policy and generate CI reports.
Before you start
- Python 3.9 or later.
- Install the evalforge-ci package from PyPI.
- A candidate flat numeric JSON summary, a policy file and a baseline summary.
Steps
Install the package with the command below.
pip install evalforge-ciRun
evalforge gateon the candidate summary, supplying the policy and baseline files.Specify the desired JSON, JUnit, SARIF and Markdown output paths.
Expected outcome
The gate evaluates the candidate against policy and baseline checks and writes the requested reports. Exit code 0 indicates pass, 1 a failed or error check, and 2 invalid input or policy configuration.
Guide limitations
- This workflow evaluates supplied metrics; it does not itself run an external evaluator.
- Thresholds and input files must be configured for the team's evaluation.
Import promptfoo results for a policy check
Convert promptfoo output into a portable EvalForge artifact and apply a policy.
Before you start
- A promptfoo JSON result file.
- Promptfoo results schema 3 and versioned producer metadata.
- EvalForge installed and a policy file.
Steps
Export promptfoo evaluation results to a JSON file.
Run
evalforge import promptfooon that file, specifying an artifact output path and, if available, the source revision.Run
evalforge gateon the artifact with the selected policy.
Expected outcome
EvalForge produces an artifact for the gate and evaluates it against the supplied policy.
Guide limitations
- The adapter requires schema 3 and versioned producer metadata.
- The README states verification against promptfoo 0.122.2; this does not establish compatibility with other versions.
Sources & review
Source record details
- Source availability
- Active
- Sources checked
- 13 October 2026
- Original website title
- jsdhwfmax/EvalForge · Official homepage title · retained in its source language