Practical overview
When it fits
Use it when you need a shared evaluation framework for an end-to-end visual document retrieval pipeline on ViDoRe v3. For single-model retriever evaluation on ViDoRe v1–v3, the repository points to MTEB.
Who it is for
- Researchers comparing visual document retrieval systems
- Practitioners evaluating custom document-search pipelines
- Teams preparing pipeline results and architecture descriptions for community comparison
Documented uses
- Evaluate a multi-stage retrieval workflow, such as retrieval followed by reranking.
- Compare hybrid dense-and-sparse retrieval or custom OCR-to-embedding processing.
- Measure retrieval scores and indexing and search computing times on ViDoRe v3 datasets.
- Prepare pipeline results and architecture details for a community results contribution.
Requirements & limits
The project is public, open source under the MIT license, and provides a Python package and CLI.
- The active repository focus is pipeline evaluation; older vision-retriever evaluation functionality is deprecated and retained for reproducibility.
- The available ViDoRe v3 collections span several subject areas, with separate English and French finance datasets.
- The CLI supports language filtering for datasets containing multilingual queries.
- Pipeline result submissions use JSON results plus an architecture description; single-model retriever submissions follow the MTEB process.
Check current requirements in the official documentation. Source-based guidance does not mean OpenViST has run or independently tested this project.
Evaluate a custom pipeline on a ViDoRe v3 dataset
Run the documented CLI evaluation for a custom pipeline and select its dataset and language.
Before you start
- Python environment with the vidore-benchmark package installed
- A custom pipeline class inheriting from BasePipeline, with index and search methods
- A ViDoRe v3 dataset name, such as vidore/vidore_v3_hr
Steps
Install the package with pip install vidore-benchmark.
Implement the custom pipeline in a Python file and provide its module path and class name.
Run vidore-benchmark pipeline evaluate with --dataset-name, --module-path, --class-name, --language, and --pipeline-args.
Expected outcome
The framework evaluates the custom pipeline on the selected dataset and tracks retrieval scores and indexing and search computing times.
Guide limitations
- The pipeline must implement the BasePipeline interface.
- The supplied CLI example uses the English language filter; evaluation results are specific to the selected dataset and configuration.
Submit pipeline results for community comparison
Prepare the documented files for a ViDoRe v3 pipeline-results contribution.
Before you start
- Evaluation result JSON files from the framework for ViDoRe v3 English splits
- A description.json file describing the pipeline architecture
Steps
Run evaluations on the ViDoRe v3 English splits using --language english.
Organize the result JSON files under results/metrics/your_pipeline_name/.
Add description.json in the same contribution, describing pipeline modules and their connections.
Open a pull request with the results and description.
Expected outcome
The contribution presents pipeline results and architecture details in the community results repository for comparison.
Guide limitations
- The README encourages hardware details for latency context but does not make them a required file.
- This submission route is for complex pipeline results; single-model retriever leaderboard submissions use MTEB.
Sources & review
Source record details
- Source availability
- Active
- Sources checked
- 12 October 2026
- Original website title
- illuin-tech/vidore-benchmark · Official homepage title · retained in its source language