OpenViSTAI · Agents · Blockchain · Crypto
Submit a project
Menu
AI Evaluation & Observability

Smallevals

Smallevals is a Python framework for locally evaluating RAG retrieval. It uses a small question-generation model to create evaluation questions from document chunks, connects to a vector database and embedding model, then tests retrieval and calculates scores. Its README also documents a command-line dashboard for viewing results.

Explore the practical guide ↓

Independent OpenViST project profile, based on linked sources.

Smallevals — OpenViST project profile in AI Evaluation & Observability.
OpenViST project illustration. See the linked sources for product details.
Understand the project

Practical overview

When it fits

Use Smallevals when you have an indexed vector store and want to evaluate its retrieval stage locally with generated questions and scores.

Who it is for

  • RAG developers assessing retrieval quality against their own document chunks.
  • Teams comparing retrieval configurations across connected vector databases.
  • Python users who want a local question-generation and retrieval-scoring workflow.

Documented uses

  • Generate synthetic questions from chunks or documents for retrieval evaluation.
  • Check whether a vector database retrieves relevant documents for generated questions.
  • Calculate retrieval scores and view evaluation results in a dashboard.
Before you choose

Requirements & limits

Smallevals is presented as a public Python repository and documents installation with pip. The repository information supplied does not identify a license.

  • The documented workflow requires a connected vector database and an embedding model.
  • The README describes CPU, GPU and MPS execution; it does not specify operating-system support.
  • Current documented model focus is retrieval evaluation; other listed RAG evaluation models are marked incoming.
  • The README shows connections for Chroma, Elasticsearch, Milvus, pgvector, Qdrant and Weaviate.

Check current requirements in the official documentation. Source-based guidance does not mean OpenViST has run or independently tested this project.

Source-based practical guide

Run a retrieval evaluation with an existing Chroma collection

Generate questions from selected chunks and evaluate whether the connected Chroma collection retrieves relevant documents.

Before you start

  • The Smallevals Python package, installed with pip.
  • An existing populated Chroma collection and a Chroma client.
  • An embedding model; the README example uses Sentence Transformers with intfloat/e5-small-v2.

Steps

  1. Install the package with pip install smallevals.

  2. Create a Chroma persistent client for the existing store and load the embedding model.

  3. Wrap the client, collection name and embedding model in SmallEvalsVDBConnection.

  4. Call evaluate_retrievals with the connection, top_k and n_chunks parameters.

Expected outcome

Smallevals generates questions for the selected chunks, tests retrieval and returns an evaluation result.

Guide limitations

  • The guide evaluates retrieval; answer-generation evaluation models are listed as incoming.
  • The Chroma collection must already contain chunks.
Documentation 1 ↗
COMPARE OPTIONS

Compare options for Vector search

These projects document a common task and share a project role. This is a source-based comparison, not an execution benchmark or a guarantee of interchangeability.

ProjectDocumented purposePlatformAccess and requirements
SmallevalsGenerate synthetic questions from chunks or documents for retrieval evaluation.Linux, macOS, Self-hosted, WindowsSmallevals is presented as a public Python repository and documents installation with pip. The repository information supplied does not identify a license.
MyScale Vector Database BenchmarkMeasure queries per second for vector search.CLI
Evidence and transparency

Sources & review

Official repository ↗
Source record details
Source availability
Active
Sources checked
18 October 2026
Original website title
mburaksayici/smallevals · Official homepage title · retained in its source language
How we review sourcesSuggest a correction