OpenViSTAI · Agents · Blockchain · Crypto
Submit a project
Menu
AI Models

Routing Transformer

Routing Transformer is a Python and PyTorch implementation of transformer models using k-means to route similar queries and keys into clusters for attention.

Explore the practical guide ↓

Independent OpenViST project profile, based on linked sources.

Routing Transformer — OpenViST project profile in AI Models.
OpenViST project illustration. See the linked sources for product details.
Understand the project

Practical overview

When it fits

Use it when you want implementation code for experimenting with k-means-routed transformer attention in a Python/PyTorch model, rather than a hosted model service.

Who it is for

  • Researchers exploring clustered attention mechanisms.
  • Developers prototyping or training transformer language models in PyTorch.
  • Practitioners building encoder–decoder or continuous-input transformer experiments.

Documented uses

  • Configure a causal language model with routed attention.
  • Train a transformer using token sequences and an input mask, with an auxiliary loss.
  • Experiment with an encoder–decoder model and source-conditioned token generation.
  • Compare model configurations such as cluster window size and local-attention heads.
Before you choose

Requirements & limits

The repository identifies an MIT license and gives a pip installation command.

  • The library returns an auxiliary loss in the examples; the language-model example adds it to the main loss before backpropagation.
  • The README shows both automatic k-means updates during backward passes and a manual update option.
  • The README cautions that the architecture has trouble generalizing to shorter sequence lengths.

Check current requirements in the official documentation. Source-based guidance does not mean OpenViST has run or independently tested this project.

Source-based practical guide

Install the package and construct a language model

Set up the documented package and instantiate its example language-model class.

Before you start

  • Python environment with PyTorch available

Steps

  1. Install the package with the command below.

    Documented command
    pip install routing_transformer

  2. Import RoutingTransformerLM from routing_transformer.

  3. Construct the model with configuration values such as vocabulary size, dimension, heads, depth, maximum sequence length, and causal mode.

Expected outcome

A configured RoutingTransformerLM instance, as shown in the repository example.

Guide limitations

  • The README notes trouble generalizing to shorter sequence lengths.
  • The example does not establish a pretrained model or hosted inference service.
Documentation 1 ↗
Evidence and transparency

Sources & review

Official repository ↗
Source record details
Source availability
Active
Sources checked
16 October 2026
Original website title
lucidrains/routing-transformer · Official homepage title · retained in its source language
How we review sourcesSuggest a correction