Practical overview
When it fits
Use it during development or evaluation when an LLM system needs repeatable scenario checks or adversarial probing, especially where outputs are non-deterministic or conversations span multiple turns.
Who it is for
- Developers integrating evaluation into LLM or agent workflows
- Teams checking RAG answer grounding or behavior across multi-turn conversations
- Practitioners generating adversarial tests for agent vulnerabilities
Documented uses
- Check agent outputs with assertions or LLM-as-judge evaluations.
- Run scenario suites to catch behavioral regressions after changes.
- Assess whether RAG responses are grounded in retrieved context.
- Generate adversarial tests for areas such as prompt injection and harmful content.
Requirements & limits
The project is open source under the Apache-2.0 license. The documented library requires Python 3.12 or later.
- Giskard Checks supports sync or async callable targets and checks over traces.
- Giskard Scan requires its scan package extra and an LLM provider configuration for generators.
- The v3 README separates agent and RAG scanning from legacy v2-only automatic tabular-model scanning.
- Optional aggregated telemetry can be disabled using DO_NOT_TRACK=1 or GISKARD_TELEMETRY_DISABLED=1.
Check current requirements in the official documentation. Source-based guidance does not mean OpenViST has run or independently tested this project.
Add a groundedness evaluation scenario
Define and run one evaluation that checks whether an agent answer is grounded in supplied context.
Before you start
- Python 3.12 or later
- Install the Giskard checks package
- A callable system under test; the documented example uses a synchronous callable
- An LLM provider extra and matching API key for the Groundedness judge
Steps
Create a Scenario and provide an interaction input and the system callable as its output function.
Attach a Groundedness check with the context against which the answer should be assessed.
Run the scenario asynchronously and print its report.
Expected outcome
A scenario result report for the groundedness evaluation.
Guide limitations
- The Groundedness check is an LLM judge and needs a configured provider.
- The example demonstrates one interaction; broader suites and multi-turn scenarios are covered in the project documentation.
Run an agent vulnerability scan
Generate and run adversarial tests against an agent callable.
Before you start
- Python 3.12 or later
- Install Giskard with the scan extra
- An agent callable
- An LLM provider extra and matching API key for scan generators
Steps
Define the agent callable that accepts input and returns an output.
Call vulnerability_scan with the target callable, a plain-language description of the agent and the chosen language.
Await the scan.
Expected outcome
A vulnerability scan run using generated adversarial tests.
Guide limitations
- The scan generators require an LLM provider extra and API key.
- The README example does not specify a result-reporting or export step.
Sources & review
Source record details
- Source availability
- Active
- Sources checked
- 7 October 2026
- Original website title
- Giskard-AI/giskard-oss · Official homepage title · retained in its source language