OpenViSTAI · Agents · Blockchain · Crypto
Submit a project
Menu
AI Models

Long Short Transformer

Long Short Transformer is a Python/PyTorch library implementing a sequence model that combines local windowed attention with a projected global sequence representation. It accepts token IDs and a mask and returns per-position vocabulary logits, with a documented causal mode using sequence segmentation.

Explore the practical guide ↓

Independent OpenViST project profile, based on linked sources.

Long Short Transformer — OpenViST project profile in AI Models.
OpenViST project illustration. See the linked sources for product details.
Understand the project

Practical overview

When it fits

Use it when you want a code-level PyTorch implementation to experiment with long-sequence attention or configure an autoregressive sequence model, rather than a hosted model or end-user application.

Who it is for

  • Python and PyTorch developers prototyping long-sequence models.
  • Researchers or practitioners exploring autoregressive sequence modeling with configurable local windows and sequence projection.

Documented uses

  • Configure a model to process token sequences using local and projected global attention.
  • Obtain per-position vocabulary logits from token IDs and a mask.
  • Set up a causal, segmented configuration for autoregressive experiments.
Before you choose

Requirements & limits

The implementation is available as a public GitHub repository under the MIT license. The README documents installation with pip.

  • The API returns vocabulary logits, so users need to provide token IDs and a mask and build their own surrounding workflow.
  • The non-causal and autoregressive examples use different configurations; causal use requires causal=True and segment_size.
  • The README's enwik8 reference identifies train.py as an autoregressive test entry point.

Check current requirements in the official documentation. Source-based guidance does not mean OpenViST has run or independently tested this project.

Source-based practical guide

Configure an autoregressive model

Set up the documented causal Long Short Transformer configuration and obtain per-position vocabulary logits.

Before you start

  • Python and PyTorch
  • The long-short-transformer package, installable with pip
  • Token IDs and a boolean mask
  • Model choices for vocabulary size, dimensions, depth, heads, sequence length, window size, segment size and projection value

Steps

  1. Install the package with the command below.

    Documented command
    pip install long-short-transformer

  2. Import LongShortTransformer and configure it with causal=True and a segment_size alongside the other model dimensions.

  3. Pass token IDs and a boolean mask to the model.

  4. Use the returned per-position vocabulary logits in your sequence-modeling workflow.

Expected outcome

A causal model configuration that returns vocabulary logits for each input position.

Guide limitations

  • The README provides a configuration example, not a complete training workflow.
  • The example uses synthetic token IDs and an all-true mask.
Documentation 1 ↗
Evidence and transparency

Sources & review

Official repository ↗
Source record details
Source availability
Active
Sources checked
18 October 2026
Original website title
lucidrains/long-short-transformer · Official homepage title · retained in its source language
How we review sourcesSuggest a correction