St Moe Pytorch: Add sparse MoE layers to PyTorch models

Understand the project

Practical overview

When it fits

Use it when prototyping or adapting an autoregressive transformer in PyTorch and you want a sparse expert layer with configurable routing and auxiliary losses.

Who it is for

PyTorch developers incorporating sparse expert layers into neural networks.
Researchers experimenting with conditional computation in autoregressive transformer models.

Documented uses

Create a mixture-of-experts layer with configurable expert count and token-routing settings.
Obtain auxiliary, balance and router z-loss values from a MoE layer call.
Add a sparse MoE block to a transformer-oriented model, with optional feed-forward layers around it.

Before you choose

Requirements & limits

The repository is public and MIT licensed. Its README documents installation with pip.

The README describes the implementation as largely transcribed from Google’s Mesh TensorFlow implementation.
The included example returns three loss values and directs users to add the total auxiliary loss to the main loss.
The README recommends the separate Soft MoE project for non-autoregressive use.

Check current requirements in the official documentation. Source-based guidance does not mean OpenViST has run or independently tested this project.

Source-based practical guide

Install the package and inspect a MoE layer

Set up the library and follow its documented example for creating and calling a mixture-of-experts layer.

Before you start

Python environment with PyTorch available; the example imports torch.
Install with pip.

Steps

Run the command below.
Documented command
```
pip install st-moe-pytorch
```
Import torch and MoE from st_moe_pytorch.
Construct MoE with a model dimension, expert count and routing and loss settings, following the README example.
Pass an input tensor to the layer and inspect the output and returned auxiliary, balance and router z-loss values.

Expected outcome

A configured MoE layer call returns an output tensor and three loss values, as shown in the repository example.

Guide limitations

The example does not document a complete model training workflow or data pipeline.
The README says the total auxiliary loss must be summed and added to the main loss; balance and router z-loss are unweighted breakdowns for logging.

Documentation 1 ↗

Evidence and transparency

Sources & review

Official repository ↗

Source record details

Source availability: Active
Sources checked: 14 October 2026
Original website title: lucidrains/st-moe-pytorch · Official homepage title · retained in its source language

How we review sources Suggest a correction

Related options

Soft MoE — The README recommends this separate project by the same repository owner for non-autoregressive models.

St Moe Pytorch