Practical overview
When it fits
Use TeaLeaves when output-only comparisons do not explain how prompt changes affect attention patterns, and you can run analysis on a Hugging Face model with sufficient GPU memory.
Who it is for
- Prompt developers investigating how revisions affect model attention.
- Researchers studying prompt processing and model interpretability.
- Teams evaluating prompt variants across test cases, including structured-output tasks.
Documented uses
- Mark named regions in system prompts and user messages for analysis.
- Capture attention weights across layers and examine token- or region-level patterns.
- Compare prompt variants using delta tables, multi-seed stability analysis and reports.
- Inspect token-rank trajectories with the logit lens.
Requirements & limits
- Rendering and analysis can be installed locally; running the model engine uses a separate GPU-dependency installation.
- GPU memory requirements rise with model size: the repository estimates about 21 GB VRAM for 8B parameters and 69 GB for 32B, plus its stated memory rule of thumb.
- The workflow requires structured prompt regions and conversation test cases before analysis.
- TeaLeaves is a pipeline that produces analysis outputs, rather than a low-level hook library requiring users to assemble their own analysis.
Check current requirements in the official documentation. Source-based guidance does not mean OpenViST has run or independently tested this project.
Prepare prompt inputs and define analysis regions
Create the structured input files TeaLeaves uses before running model analysis.
Before you start
- A system prompt text file.
- A regions JSON file defining named spans with markers, regular expressions or character ranges.
- A conversations JSON file containing test cases.
Steps
Create a regions JSON file that names prompt spans, such as rules, examples, context and current message.
Save the full system prompt as system_prompt.txt.
Save conversation test cases in conversations.json.
Run the documented input-preparation module with the prompt, regions and conversations files to produce test_cases.json.
Expected outcome
A test_cases.json input file prepared for the documented analysis stage.
Guide limitations
- This guide covers input preparation only; it does not run model analysis or verify that the inputs are valid.
Render and compare existing TeaLeaves results
Generate documented visualizations and comparison reports from result files already produced by the pipeline.
Before you start
- TeaLeaves installed locally with rendering and analysis dependencies.
- Result files in a results directory, including baseline and modified variants for comparison.
Steps
Render a per-token heatmap from a result JSON file.
Render per-region attention trajectories as cooking curves, or render an animated layer sweep.
Optionally generate a multi-sample aggregate from a results directory.
Run the comparison module with named baseline and modified variants, then generate a markdown report from the experiment directories.
Expected outcome
Rendered attention views and, when comparison inputs are provided, variant comparison output and a markdown report.
Guide limitations
- This guide starts from existing result files and does not cover model setup or capture.
- The documented commands assume the shown module names, file layout and variant identifiers.
Sources & review
Source record details
- Source availability
- Active
- Sources checked
- 18 October 2026
- Original website title
- taylorsatula/TeaLeaves · Official homepage title · retained in its source language