Practical overview
When it fits
Use it when developing a non-autoregressive encoder and you want to integrate a Soft MoE layer or compare its documented slot variants.
Who it is for
- PyTorch developers building non-autoregressive encoder models
- Researchers comparing fixed-slot and dynamic-slot Soft MoE
- Developers exploring MoE layers in text-to-image model architectures
Documented uses
- Add a Soft MoE layer in place of a transformer feed-forward layer.
- Choose between a configured fixed-slot module and a dynamic-slot module.
- Explore the effect of slot approach across sequence lengths.
Requirements & limits
- SoftMoE uses a configured sequence length to calculate slots by default, while DynamicSlotsSoftMoE adjusts slots with sequence length.
- Routing cost grows quadratically with sequence length.
- The documentation does not provide an autoregressive implementation.
- Distributed handling of variable sequence lengths is listed as unfinished.
Check current requirements in the official documentation. Source-based guidance does not mean OpenViST has run or independently tested this project.
Set up a Soft MoE layer in a PyTorch model
Install the package and configure either documented Soft MoE variant for sequence representations.
Before you start
- A Python environment with pip
- PyTorch, imported as torch
- Model sequence representations and their dimension
- For SoftMoE, a maximum sequence length and number of experts, or an explicitly selected number of slots
Steps
Install the package with the command below.
pip install soft-moe-pytorchImport
SoftMoEfromsoft_moe_pytorchfor fixed-slot use, orDynamicSlotsSoftMoEfor dynamic slots.Configure the selected module with the model dimension and number of experts; for
SoftMoE, also set sequence length or the slot count.Pass a sequence tensor through the module and, following the documented example, add its output to the input as a residual.
Expected outcome
A Soft MoE layer output that can be used in place of a transformer feed-forward layer in a non-autoregressive encoder.
Guide limitations
- The implementation is described as intended for non-autoregressive encoders, not autoregressive models.
- Routing cost grows quadratically as sequence length increases.
- The documentation lists distributed variable-sequence-length handling as unfinished.
Sources & review
Source record details
- Source availability
- Active
- Sources checked
- 13 October 2026
- Original website title
- lucidrains/soft-moe-pytorch · Official homepage title · retained in its source language