OpenViSTAI · Agents · Blockchain · Crypto
Submit a project
Menu
AI Models

Smaller Transformers

Smaller Transformers is Geotrend’s open-source collection of reduced multilingual BERT and DistilBERT models, accompanied by a Python script that generates a smaller transformer from a source model and selected vocabulary. Published variants are available through Hugging Face and can be loaded with the Transformers library.

Explore the practical guide ↓

Independent OpenViST project profile, based on linked sources.

Smaller Transformers — OpenViST project profile in AI Models.
OpenViST project illustration. See the linked sources for product details.
Understand the project

Practical overview

When it fits

Use it when a multilingual NLP project needs a model for fewer languages than a broad multilingual model supports, or when you want to create a reduced variant from a source transformer and vocabulary file.

Who it is for

  • NLP practitioners selecting multilingual models for a defined set of languages.
  • Developers seeking to reduce model size or memory footprint for inference or fine-tuning.
  • Researchers exploring multilingual and cross-lingual transfer tasks.

Documented uses

  • Find a reduced BERT or DistilBERT variant for a chosen language set.
  • Load a published model with Hugging Face Transformers for downstream inference or fine-tuning.
  • Generate a smaller transformer using a selected vocabulary subset.
  • Compare repository-reported model size and memory measurements when evaluating candidate variants.
Before you choose

Requirements & limits

  • The repository reports variants extracted from mBERT and distil-mBERT and states that models are hosted on Hugging Face.
  • The README reports reductions of up to 64% and provides size and memory comparisons for selected models; these figures are not task-independent guarantees.
  • The reduction script requires a source model and vocabulary file, and exposes an option for TensorFlow conversion.
  • The repository is licensed under Apache-2.0.
  • The README describes the Transformers library loading path using AutoTokenizer and AutoModel.

Check current requirements in the official documentation. Source-based guidance does not mean OpenViST has run or independently tested this project.

Source-based practical guide

Generate a reduced transformer model

Prepare a smaller transformer from a source model and selected vocabulary using the repository’s script.

Before you start

  • A source transformer model identifier
  • A vocabulary file path
  • Python environment with the repository requirements installed

Steps

  1. Install the dependencies with the command below.

    Documented command
    pip install -r requirements.txt

  2. Run reduce_model.py with --source_model, --vocab_file, and --output_model set to the chosen source, vocabulary path, and output name.

  3. Set --convert_to_tf to choose whether to generate a TensorFlow version.

Expected outcome

A reduced transformer model is generated using the selected vocabulary, with optional TensorFlow conversion.

Guide limitations

  • The README does not establish performance for arbitrary source models or vocabularies.
  • The documented command describes generation; it does not provide a task-specific evaluation procedure.
Documentation 1 ↗
Evidence and transparency

Sources & review

Official repository ↗
Source record details
Source availability
Active
Sources checked
18 October 2026
Original website title
Geotrend-research/smaller-transformers · Official homepage title · retained in its source language
How we review sourcesSuggest a correction