OpenViSTAI · Agents · Blockchain · Crypto
Submit a project
Menu
AI Evaluation & Observability

Trustllm

Trustllm is an open-source Python research toolkit and command-line tool for benchmarking large language models across truthfulness, safety, fairness, robustness, privacy and ethics. It generates benchmark responses using compatible model APIs or local weights, then scores completed responses with task-specific evaluation pipelines.

Explore the practical guide ↓

Independent OpenViST project profile, based on linked sources.

Trustllm — OpenViST project profile in AI Evaluation & Observability.
OpenViST project illustration. See the linked sources for product details.
Understand the project

Practical overview

When it fits

Use Trustllm when you need a dataset-backed evaluation workflow for model responses and can provide a compatible API endpoint or supported local model weights.

Who it is for

  • Researchers comparing language models on trustworthiness benchmarks.
  • Model teams evaluating safety, fairness, robustness, privacy or other documented dimensions.
  • Developers who want to run benchmark generation and scoring through Python or a CLI.

Documented uses

  • Generate benchmark responses from an API-accessible or locally loaded model.
  • Inspect whether a generation run completed using its HTML report.
  • Score complete response sets and review JSON or HTML results.
  • Compare model behavior across six trustworthiness dimensions.
Before you choose

Requirements & limits

The repository is public and licensed under MIT. The documented base package supports API generation and data downloads; local generation and evaluation use optional dependencies.

  • API generation uses text-only Chat Completions; the provider endpoint, model access and any credentials must be configured.
  • Local generation supports Transformers causal language models, with model access, hardware capacity and architecture compatibility affecting use.
  • Generation reports show completion counts, not benchmark scores; scoring is a separate step.
  • Scoring methods vary by task, and API judge calls can incur charges.
  • The current generation engine and integrations differ from the original paper settings, so new runs are not automatically equivalent to those experiments.

Check current requirements in the official documentation. Source-based guidance does not mean OpenViST has run or independently tested this project.

Source-based practical guide

Generate API responses for a safety task

Download the benchmark data and create an API-backed generation run with an HTML completion report.

Before you start

  • Python and Git
  • A compatible model API endpoint and model ID
  • API credentials if required by the provider

Steps

  1. Install the base package from the TrustLLM GitHub repository.

  2. Download the benchmark data with the command below.

    Documented command
    python -m trustllm download --output data

  3. Set OPENAI_API_KEY and OPENAI_BASE_URL for the provider, then run the command below.

    Documented command
    python -m trustllm generate --backend api --model your-model-id --task safety --data data/dataset --limit 5 --concurrency 4 --output runs/api-safety

  4. Open runs/api-safety/report.html to inspect completion counts.

Expected outcome

An HTML report showing generation completion counts for the run; it does not contain benchmark scores.

Guide limitations

  • The API route uses text-only Chat Completions.
  • --limit 5 selects five samples per dataset file and is intended as a smoke test.
  • Use the provider's actual API root; model access and endpoint requirements depend on that service.
Documentation 1 ↗
Source-based practical guide

Score a completed safety generation run

Evaluate a full directory of generated responses and save benchmark scores.

Before you start

  • TrustLLM installed with its eval dependencies
  • A full-run output directory with complete responses
  • An available judge model and OPENAI_JUDGE_MODEL when the selected scoring task uses an API judge

Steps

  1. Install the evaluation dependencies from the TrustLLM repository.

  2. Run the command below, replacing the path with the full-run output directory.

    Documented command
    python -m trustllm evaluate --task safety --data runs/api-safety-full

  3. Review scores.json and scores.html in the output directory.

Expected outcome

JSON and HTML benchmark score reports for the selected task.

Guide limitations

  • Incomplete responses are rejected.
  • The scoring method varies by task and can involve rules, a downloaded classifier, embeddings or an API judge.
  • API judge calls can incur charges; consult the scoring guide before a full run.
Documentation 1 ↗
Evidence and transparency

Sources & review

Official repository ↗
Source record details
Source availability
Active
Sources checked
14 October 2026
Original website title
HowieHwong/TrustLLM · Official homepage title · retained in its source language
How we review sourcesSuggest a correction