Practical overview
When it fits
Use it when you have a SKILL.md and task evaluations and want to compare model outputs with the skill loaded against a baseline without it.
Who it is for
- Developers creating or revising Agent Skills.
- Teams comparing model performance with and without a skill.
- Engineers adding repeatable skill evaluations to CI, custom pipelines or dashboards.
Documented uses
- Measure whether a skill changes a model’s performance on defined tasks.
- Compare outputs and judge grades between skill-enabled and baseline runs.
- Check tool-using agents with tool-call assertions.
- Review or integrate evaluation artifacts through JSON, JSONL and static HTML reports.
Requirements & limits
The repository identifies the project as open source under the MIT license. It can be installed with npm or invoked through npx.
- Baseline comparison must be enabled to get with-skill versus without-skill results.
- Both a target model and judge model are configured; the judge grades outputs against expected output and assertions.
- The README describes an OpenAI-compatible chat API route, including compatible hosted services and local Llama servers.
- The CLI is suited to direct runs, while the TypeScript SDK supports custom pipelines, providers and dashboards.
- Evaluation definitions and skill files provide the task criteria; the tool also supports deterministic tool-call assertions.
Check current requirements in the official documentation. Source-based guidance does not mean OpenViST has run or independently tested this project.
Run a baseline comparison from a skills folder
Run configured evaluations with and without a skill and generate a report for review.
Before you start
- Node.js, as indicated by the project’s Node version badge.
- A folder containing skills with SKILL.md files and evaluations.
- A target model and judge model available through an OpenAI-compatible chat API.
- An API key exposed through the environment variable configured for the provider; the example uses OPENAI_API_KEY.
Steps
Run npx agent-skills-eval ./skills --target gpt-4o-mini --judge gpt-4o-mini --baseline --strict, substituting the skills folder and model names as needed.
Alternatively, configure root, target, judge, baseline, baseUrl and apiKeyEnv in agent-skills-eval.yaml, then run OPENAI_API_KEY=... npx agent-skills-eval --config agent-skills-eval.yaml.
Review the generated workspace artifacts, including the with_skill and without_skill results and the report at iteration-1/report/index.html when using the documented iteration layout.
Expected outcome
A workspace containing evaluation outputs and grading, with a static HTML report for comparing results.
Guide limitations
- The baseline comparison requires the --baseline flag or baseline: true in configuration; otherwise only the with_skill run is produced.
- Results depend on the configured target and judge models and the defined expected outputs and assertions; the tool reports evaluations rather than guaranteeing skill improvement.
Sources & review
Source record details
- Source availability
- Active
- Sources checked
- 13 October 2026
- Original website title
- darkrishabh/agent-skills-eval · Official homepage title · retained in its source language