Practical overview
When it fits
Use Mergoo when you have expert model or adapter identifiers and want to configure a merge, save the composed checkpoint and train the result.
Who it is for
- Developers and researchers working with multiple full-model or LoRA fine-tuned LLM experts
- Teams seeking to combine generic or domain-specific models into a specialized model
Documented uses
- Compose full-model experts into a Mixture-of-Experts model.
- Combine LoRA fine-tunes using a routing layer as a Mixture-of-Adapters.
- Save a merged model checkpoint and fine-tune it using a supported trainer.
Requirements & limits
The project is a public GitHub repository under the LGPL-3.0 license. The README documents installation through pip or from source.
- The README names Llama (including Llama 3), Mistral, Phi-3 and BERT as supported base-model families; its quick-start configuration lists Mistral, Llama and BERT.
- Documented merging approaches are Mixture-of-Experts, Mixture-of-Adapters and layer-wise merging.
- Trainer integrations listed are Hugging Face Trainer, SFTrainer and PEFT.
- The repository lists CPU, MPS and GPU device support.
- The quick start documents saving a composed checkpoint before loading and fine-tuning it.
Check current requirements in the official documentation. Source-based guidance does not mean OpenViST has run or independently tested this project.
Compose and save a merged expert checkpoint
Configure expert model identifiers, compose a merged LLM and save its checkpoint.
Before you start
- Mergoo installed from pip or source
- A supported base-model family and expert model identifiers
- A Python configuration specifying model type and experts; MoE configuration also specifies experts per token and router layers
Steps
Define the merge configuration with the base-model type and expert identifiers.
Import torch and ComposeExperts, then initialize ComposeExperts with the configuration and torch_dtype.
Call compose() to create the merged model.
Call save_checkpoint() with a checkpoint path.
Expected outcome
A composed LLM is saved as a checkpoint at the chosen path.
Guide limitations
- The repository's quick-start configuration lists Mistral, Llama and BERT; its feature list also names Phi-3.
- The repository notes that router layers are untrained when loading a merged checkpoint and may produce a weight-loading warning.
- The guide covers composition and checkpoint saving, not model execution or deployment.
Load a merged checkpoint for fine-tuning
Load a saved merged checkpoint and begin training it with Hugging Face Trainer.
Before you start
- A checkpoint composed and saved with Mergoo
- Hugging Face Transformers and a training setup to provide to Trainer
Steps
Import Trainer and the documented Mergoo Mistral model class.
Load the saved checkpoint with from_pretrained().
Create a Trainer with the training setup.
Call trainer.train().
Expected outcome
The documented workflow starts fine-tuning the loaded merged model.
Guide limitations
- The README leaves the Trainer arguments as an ellipsis, so dataset and training-argument setup must be supplied separately.
- The example is specific to Mistral.
- The README warns that router weights are untrained at load time and may trigger a weight-loading warning.
Sources & review
Source record details
- Source availability
- Active
- Sources checked
- 14 October 2026
- Original website title
- Leeroo-AI/mergoo · Official homepage title · retained in its source language