OpenViSTAI · Agents · Blockchain · Crypto
Submit a project
Menu
AI Evaluation & Observability

Sample Gen AI Evaluations Workshop

Sample Gen AI Evaluations Workshop is a public GitHub repository of Jupyter Notebook learning materials on evaluating generative AI workloads for production, including quality, reliability, cost, performance and failure analysis.

Explore the practical guide ↓

Independent OpenViST project profile, based on linked sources.

Sample Gen AI Evaluations Workshop — OpenViST project profile in AI Evaluation & Observability.
OpenViST project illustration. See the linked sources for product details.
Understand the project

Practical overview

When it fits

Use it when you want a guided learning path for generative AI evaluation, beginning with core concepts and continuing into modules relevant to a particular workload or framework.

Who it is for

  • Developers and practitioners learning to evaluate generative AI systems for production use.
  • Learners with basic Python and machine-learning familiarity who want a structured introduction to AI evaluation.

Documented uses

  • Assess workload operational metrics such as cost and performance.
  • Evaluate and tune the quality of generative AI results.
  • Find failure patterns by examining agent traces.
  • Explore evaluation methods for particular workloads, frameworks or industries.
Before you choose

Requirements & limits

The repository is public on GitHub and carries an MIT-0 license. The workshop prerequisites include an AWS account with Amazon Bedrock enabled and configured AWS credentials.

  • The recommended foundational path covers operational metrics, quality metrics, failure analysis and agentic metrics.
  • Optional modules can be chosen by workload, framework or industry rather than followed as one fixed sequence.
  • The interactive-learning option is a separate way to study the notebooks using a compatible AI coding assistant; the repository names Kiro and Claude as examples.

Check current requirements in the official documentation. Source-based guidance does not mean OpenViST has run or independently tested this project.

Source-based practical guide

Start with the foundational evaluation modules

Set up the workshop and follow its recommended sequence for core generative AI evaluation topics.

Before you start

  • An AWS account with Amazon Bedrock enabled
  • Basic Python and machine-learning familiarity
  • AWS credentials

Steps

  1. Clone the workshop repository.

  2. Configure AWS credentials.

  3. Work through the Foundational Evaluations modules in order: operational metrics, quality metrics, understanding failures, and agentic metrics.

Expected outcome

A structured introduction to operational and quality evaluation, agent-trace failure analysis, and agentic metrics.

Guide limitations

  • The workshop lists AWS account access with Amazon Bedrock enabled as a prerequisite.
  • Workload-, framework- and industry-specific modules are separate optional material.
Documentation 1 ↗
Evidence and transparency

Sources & review

Official repository ↗
Source record details
Source availability
Active
Sources checked
13 October 2026
Original website title
aws-samples/sample-gen-ai-evaluations-workshop · Official homepage title · retained in its source language
How we review sourcesSuggest a correction