OpenViSTAI · Agents · Blockchain · Crypto
Submit a project
Menu
AI Evaluation & Observability

Diragnosis

Diragnosis, named diRAGnosis in its repository, is an MIT-licensed Python framework and locally runnable application for evaluating language models and retrieval models in RAG systems using user-provided documents. Built with LlamaIndex, it provides a FastAPI and Gradio application as well as a Python package. It can create synthetic question datasets, evaluate answer faithfulness and relevance, and measure retrieval

Explore the practical guide ↓

Independent OpenViST project profile, based on linked sources.

Diragnosis — OpenViST project profile in AI Evaluation & Observability.
OpenViST project illustration. See the linked sources for product details.
Understand the project

Practical overview

When it fits

Use it when you need document-based measurements of RAG answer quality or retrieval performance, particularly when comparing model or search configurations.

Who it is for

  • Developers evaluating document-grounded RAG applications.
  • Teams comparing language models or retrieval configurations against their own document collections.
  • Python users who want evaluation functions for question-dataset generation, LLM evaluation, and retrieval evaluation.

Documented uses

  • Generate synthetic evaluation questions from supplied documents.
  • Assess answer faithfulness and relevancy against retrieved context.
  • Measure whether retrieval returns the correct document first and how highly it ranks relevant documents.
  • Run a local dashboard or call evaluation functions from Python.
Before you choose

Requirements & limits

The repository identifies the software as open source under the MIT license. The README documents a local Docker application and a pip-installable Python package.

  • The Docker application requires Docker and Docker Compose; the documented source-code setup also requires Conda.
  • The README's package example uses Qdrant clients and an OpenAI API key, and demonstrates Hugging Face embeddings.
  • Search can be configured as semantic-only or hybrid, according to the README.
  • Evaluation outputs differ by task: answer evaluation reports binary passes and faithfulness/relevancy scores; retrieval evaluation reports hit rate and MRR.
  • The Docker setup may take up to 30 minutes the first time, depending on hardware and connection.

Check current requirements in the official documentation. Source-based guidance does not mean OpenViST has run or independently tested this project.

Source-based practical guide

Set up the local Docker dashboard

Launch the diRAGnosis application and open its locally hosted dashboard.

Before you start

  • A cloned diRAGnosis repository.
  • Docker and Docker Compose.
  • On Windows, use the documented PowerShell script; on Linux/macOS, use the documented Bash script.

Steps

  1. Clone the repository and change into its diRAGnosis directory.

  2. Run bash run_services.sh on Linux/macOS, or .\run_services.ps1 on Windows. Alternatively, run docker compose up db -d followed by docker compose up dashboard -d.

  3. Open http://localhost:8000/dashboard.

Expected outcome

The README says the application will be running at the local dashboard address.

Guide limitations

  • First-time setup can take up to 30 minutes depending on connection and hardware.
  • This setup guide does not cover conducting an evaluation.
Documentation 1 ↗
COMPARE OPTIONS

Compare options for Retrieval-augmented applications

These projects document a common task and share a project role. This is a source-based comparison, not an execution benchmark or a guarantee of interchangeability.

ProjectDocumented purposePlatformAccess and requirements
DiragnosisGenerate synthetic evaluation questions from supplied documents.Docker, Linux, macOS, Self-hostedThe repository identifies the software as open source under the MIT license. The README documents a local Docker application and a pip-installable Python package.
SentrevCompare retrieval performance across embedding models on a document collection.CLIThe project is an open-source Python package under the MIT license. Its README documents installation with pip or from source.
BergenRun reproducible experiments with configurable RAG pipeline components.CLIBERGEN is released under the Creative Commons Attribution-NonCommercial-ShareAlike 4.0 license.
Evidence and transparency

Sources & review

Official repository ↗
Source record details
Source availability
Active
Sources checked
17 October 2026
Original website title
AstraBert/diRAGnosis · Official homepage title · retained in its source language
How we review sourcesSuggest a correction