AI · Agents · Blockchain · Crypto

Submit a project Sign in

Menu

Directory category

AI Evaluation & Observability

Evaluate and monitor AI applications through tests, traces and performance diagnostics. Compare official resources, requirements and limitations.

128 projects

Directory results

AI Evaluation & Observability projects

T Transformer Uncertainty Transformer Uncertainty is a Python research codebase for evaluating predictive uncertainty methods on Transformer-based natural-language-understanding tasks.

P Powerful Benchmarker Powerful Benchmarker is a Python-based machine-learning benchmarking library and script collection for hyperparameter searches, model-training experiments, and evaluation of validation methods.

D Deepmark Deepmark is an open-source LLM assessment tool for comparing language models on user-provided data with task-specific measures such as accuracy, latency, cost and failure rate.

D Daam DAAM is a Python library and command-line tool that interprets Stable Diffusion generations by tracing cross-attention and producing heat maps attributing image regions to prompt words.

F Factool Factool is a Python framework that evaluates factual errors in generative AI outputs across knowledge-based question answering, code generation, mathematical reasoning and scientific literature review.

P Pythonrouge Pythonrouge is a Python wrapper for the ROUGE summarization evaluation toolkit. It calculates configurable ROUGE scores by comparing system-generated summaries with reference summaries supplied as text or stored in directories.

P Parea SDK for Python Parea SDK for Python is a Python package for evaluating and testing LLM applications, instrumenting logs and traces for monitoring, and supporting collaborative prompt deployment.

F FMBench FMBench is a Python package and command-line tool for benchmarking foundation models on Amazon SageMaker, Bedrock, EKS, EC2, or a user-provided endpoint.