OpenViSTAI · Agents · Blockchain · Crypto
Submit a project
Menu
Agent Evaluation & Observability

Judgeval

Judgeval is an open-source Python SDK and CLI for tracing, evaluating and monitoring LLM-powered agents. It captures application activity through OpenTelemetry-based tracing, applies prompt-based judges to agent behavior, and helps teams investigate failures and validate fixes against production traces.

Explore the practical guide ↓

Independent OpenViST project profile, based on linked sources.

Judgeval — OpenViST project profile in Agent Evaluation & Observability.
OpenViST project illustration. See the linked sources for product details.
Understand the project

Practical overview

When it fits

Use Judgeval during agent development and operation when you need to inspect execution traces, evaluate behavior, monitor production traffic or investigate a regression against historical production cases.

Who it is for

  • Developers building LLM-powered agents who need trace-level visibility into tools and model calls.
  • Teams that want to score agent behavior on live traffic or historical traces and use the results to investigate failures.
  • Engineering groups managing traces, judges and evaluations from a terminal.

Documented uses

  • Instrument agent and tool functions to capture inputs, outputs and LLM token usage.
  • Score agent behavior with prompt-based evaluators and keep structured behavior records.
  • Monitor live production traffic and surface detected behaviors as signals, with configurable Slack alerts.
  • Replay judges on historical traces to check proposed fixes against production cases.
  • Use the CLI to query trace history, inspect detected behaviors and run evaluations against production data.
Before you choose

Requirements & limits

The repository is public and specifies an Apache-2.0 license. The documented quickstart uses an API key and organization ID.

  • The documented instrumentation example uses Python and an OpenAI client; integrations also list Anthropic, Google GenAI, Together AI, LangGraph, OpenLit and Claude Agent SDK.
  • Prompt-based judges produce scored, labeled behavior records that can be searched over time.
  • Live monitoring is described as server-side scoring, with configurable Slack alerts for detected regressions and recurrences.
  • The CLI is a separate terminal interface for managing agents, traces, judges, behaviors and evaluations.
  • The quickstart requires Judgment API credentials and organization identification.

Check current requirements in the official documentation. Source-based guidance does not mean OpenViST has run or independently tested this project.

Source-based practical guide

Set up tracing for an OpenAI-based agent

Add Judgeval tracing to an agent and an observed tool function, following the repository’s quickstart example.

Before you start

  • Python environment with judgeval and the OpenAI package installed.
  • JUDGMENT_API_KEY and JUDGMENT_ORG_ID environment variables.
  • An OpenAI client and application functions to instrument.

Steps

  1. Install the SDK with the command below.

    Documented command
    pip install judgeval

  2. Set JUDGMENT_API_KEY and JUDGMENT_ORG_ID in the environment.

  3. Initialize tracing with Tracer.init(project_name="my-project") and wrap an OpenAI client using wrap(OpenAI()).

  4. Decorate relevant tool and agent functions with @Tracer.observe() and call the agent function.

Expected outcome

The example instruments agent and tool functions for observability while wrapping the OpenAI client.

Guide limitations

  • The guide follows the documented OpenAI quickstart; it does not establish support for every model, framework or application configuration.
  • The repository example does not provide a complete production deployment procedure.
Documentation 1 ↗
Evidence and transparency

Sources & review

Official repository ↗
Source record details
Source availability
Active
Sources checked
13 October 2026
Original website title
JudgmentLabs/judgeval · Official homepage title · retained in its source language
How we review sourcesSuggest a correction