Practical overview
When it fits
Use it when building or exploring a cross-language ONNX embedding workflow and the required output is a dense embedding.
Who it is for
- Developers integrating embedding models into .NET, Java, or Python applications.
- Teams that need an ONNX tokenizer alongside an ONNX embedding model.
- Developers exploring a tokenizer conversion workflow with ONNX Runtime Extensions.
Documented uses
- Convert a Hugging Face tokenizer into tokenizer.onnx.
- Pair the exported tokenizer with an ONNX embedding model to generate dense embeddings.
- Compare .NET and Java embeddings with Hugging Face-generated reference embeddings using the supplied scripts.
Requirements & limits
The repository is public and open source under the Apache-2.0 license.
- The README describes dense embeddings only; it directs users needing BGE-M3 dense, sparse, and ColBERT outputs to the separate BGE-M3 ONNX repository.
- The documented tokenizer conversion uses ONNX Runtime Extensions.
- The README identifies C#, Java, C++, and Python as supported by ONNX Runtime Extensions; repository samples are listed for .NET and Java, with Python example material.
- The model files are obtained separately from Hugging Face or as an onnx.zip release archive.
Check current requirements in the official documentation. Source-based guidance does not mean OpenViST has run or independently tested this project.
Set up the tokenizer conversion and embedding samples
Prepare the repository’s ONNX files and generate a tokenizer model for the documented samples.
Before you start
- Clone the public repository.
- Download BGE-M3 model.onnx and model.onnx_data from the linked Hugging Face location.
- Use the repository’s Jupyter notebook to generate tokenizer.onnx, or obtain onnx.zip from the repository releases.
Steps
Create an onnx folder at the repository root and place model.onnx and model.onnx_data there.
Run tokenizer_to_onnx_model.ipynb to create tokenizer.onnx, or use the release archive containing the model and tokenizer.
With all three ONNX files in the folder, follow the repository’s .NET sample in /samples/dotnet or Java sample in /samples/java.
Expected outcome
The repository’s documented sample setup has the tokenizer and embedding model files required for the .NET or Java example.
Guide limitations
- This repository demonstrates dense embeddings only.
- The README does not provide a single shared sample-run command for the .NET and Java implementations.
Run the optional cross-language embedding comparison
Use the repository’s scripts to compare .NET and Java embeddings with Hugging Face-generated reference embeddings.
Before you start
- Python, .NET, Java, and Maven installed.
- The ONNX model and tokenizer files prepared as described in the repository.
Steps
On Linux or macOS, make run_tests.sh executable with chmod +x run_tests.sh.
Run ./run_tests.sh on Linux or macOS, or run ./run_tests.ps1 on Windows.
Expected outcome
The repository’s scripts perform the documented cross-language embedding comparison.
Guide limitations
- The README says these scripts are primarily used for CI.
- The comparison covers the documented .NET and Java embeddings.
Sources & review
Source record details
- Source availability
- Active
- Sources checked
- 17 October 2026
- Original website title
- yuniko-software/tokenizer-to-onnx-model · Official homepage title · retained in its source language
Related options
BGE-M3 ONNX — The repository points to this separate project for BGE-M3 dense, sparse, and ColBERT embedding outputs.