Practical overview
When it fits
Use it when building a routed MoE variant from Gemma-3-270m and you need a documented path from checkpoint conversion through optional training and routing analysis to llama.cpp CPU serving.
Who it is for
- Researchers and developers experimenting with sparse-upcycled Gemma models.
- Model builders who want to train or fine-tune routed experts and inspect their routing behavior.
- Teams preparing a converted model for local CPU inference through llama.cpp.
Documented uses
- Convert a dense Gemma-3-270m checkpoint into a model with a router and configurable expert count and top-k selection.
- Initialize experts from one checkpoint or from different Gemma-3-270m variants.
- Continue-train or fine-tune the resulting MoE with Transformers Trainer.
- Inspect load balance, dead experts, routing entropy and task specialization.
- Convert and quantize a model for llama.cpp CPU inference.
Requirements & limits
The repository documents local installation as a Python package and requires Python 3.10 or later. Training and routing tools require the optional `datasets` dependency.
- The supported base family documented here is Gemma-3-270m; both full-vocabulary and trimmed checkpoints are accepted.
- Expert count and top-k determine the active expert path; the project describes CPU latency and memory as scaling with active rather than total parameters.
- Vocabulary trimming is optional and requires a separate trimmer plus an extra converter patch for the llama.cpp path.
- CPU serving requires work in a fresh llama.cpp checkout: patch, build, convert to GGUF and quantize.
- The repository describes a published 16-expert, top-4 full-vocabulary starter checkpoint as untrained and intended for further training or fine-tuning.
Check current requirements in the official documentation. Source-based guidance does not mean OpenViST has run or independently tested this project.
Convert a Gemma-3-270m checkpoint into an MoE
Create a copy-initialized routed MoE checkpoint from a Gemma-3-270m source.
Before you start
- Python 3.10 or later.
- A Gemma-3-270m checkpoint accessible as the source.
- The gemma3moe-builder repository and its Python package installed.
Steps
Install the package from the repository with the command below.
pip install -e .Run the command below.
python -m gemma3moe.upcycle --src google/gemma-3-270m --out ./my-moe --num-experts 8 --top-k 2
Expected outcome
A converted MoE checkpoint is written to `./my-moe` with eight experts and top-2 routing.
Guide limitations
- This covers conversion only, not training or CPU serving.
- The repository documents a separate correctness-check command, but this recipe does not run it.
Prepare the training and routing-inspection stage
Install the optional training dependency and use the documented commands to train and inspect a converted checkpoint.
Before you start
- Python 3.10 or later.
- An initial MoE checkpoint.
- Training data in JSONL files and validation data in JSONL format.
- A prompts file for the routing-diversity inspection.
Steps
Install the package with training dependencies using the command below.
pip install -e .[train]Run the documented training command with the checkpoint, train and validation files, and output path.
Run the command below on the resulting checkpoint and validation file.
python -m gemma3moe.eval.router_healthRun the command below with the checkpoint and prompts file.
python -m gemma3moe.eval.router_diversity
Expected outcome
The training command writes a fine-tuned checkpoint; the inspection commands report routing health and diversity measures.
Guide limitations
- The README’s example uses three epochs and an auxiliary coefficient of 0.003; those are example arguments, not stated requirements.
- This recipe does not cover llama.cpp conversion or serving.
Sources & review
Source record details
- Source availability
- Active
- Sources checked
- 18 October 2026
- Original website title
- tardellirs/gemma3moe-builder · Official homepage title · retained in its source language