OpenViSTAI · Agents · Blockchain · Crypto
Submit a project
Menu
AI Models

Linformer Pytorch

Linformer Pytorch is a Python/PyTorch implementation of Linformer-style linear self-attention, including multi-head attention, transformer encoder and decoder modules, and a language-model example. It is intended for building or experimenting with transformer models that process long sequences using attention with linear sequence-length complexity.

Explore the practical guide ↓

Independent OpenViST project profile, based on linked sources.

Linformer Pytorch — OpenViST project profile in AI Models.
OpenViST project illustration. See the linked sources for product details.
Understand the project

Practical overview

When it fits

Use it when building or prototyping PyTorch transformer models where linear-complexity attention is relevant, and when you want examples of language-model, multi-head attention and encoder-decoder components.

Who it is for

  • PyTorch developers exploring attention mechanisms for long-sequence models.
  • Researchers prototyping transformer language models or encoder-decoder architectures.
  • Model builders who want configurable Linformer attention components.

Documented uses

  • Build a token-based language-model prototype with Linformer attention.
  • Add Linformer-style attention to a transformer architecture.
  • Experiment with long-sequence attention settings, including projection methods and causal masking.
Before you choose

Requirements & limits

The public GitHub repository identifies the project as MIT-licensed. The README provides pip-install and Git clone options.

  • The repository describes linear sequence-length complexity as the central distinction from full attention, which it includes as an O(n²) comparison option.
  • The README lists learnable, convolution and no-parameters projection methods, along with choices for causal masking and parameter sharing.
  • The project README says WikiText-2 validation was in progress and that optimal hyperparameters remained to be found.
  • The examples use PyTorch tensors and show CUDA calls in language-model and long-sequence examples; the multi-head attention example uses CPU tensors.

Check current requirements in the official documentation. Source-based guidance does not mean OpenViST has run or independently tested this project.

Source-based practical guide

Install and try the Linformer language-model example

Install linformer-pytorch and run the documented example that maps a token sequence to language-model outputs.

Before you start

  • Python environment with PyTorch available.
  • CUDA-enabled device, because the example moves the model and input to CUDA.

Steps

  1. Install the package with the command below.

    Documented command
    pip install linformer-pytorch

  2. Create a LinformerLM with the documented example configuration: num_tokens=10000, input_size=512, channels=64, dim_k=128, dim_ff=128, nhead=4, and depth=2, along with the other shown settings.

  3. Move the model to CUDA, create a random token tensor of shape (1, 512) with values from 1 to 9999, and move it to CUDA.

  4. Pass the tensor to the model and inspect the output; the example reports an output shape of (1, 512, 10000).

Expected outcome

A Linformer language-model example producing one vocabulary-sized output per input sequence position.

Guide limitations

  • The documented code requires CUDA; CPU execution is not shown.
  • The README says validation on WikiText-2 is in progress and that the best hyperparameters still need to be found.
Documentation 1 ↗
Source-based practical guide

Install and try the Linformer sequence-model example

Install linformer-pytorch and run the documented Linformer model example on a long sequence.

Before you start

  • Python environment with PyTorch available.
  • CUDA-enabled device, because the example moves the model and input to CUDA.

Steps

  1. Install the package with the command below.

    Documented command
    pip install linformer-pytorch

  2. Create a Linformer with the documented example configuration: input_size=262144, channels=64, dim_k=128, dim_ff=128, nhead=4, and depth=2, along with the other shown settings.

  3. Move the model to CUDA and create a random input tensor of shape (1, 262144, 64) on CUDA.

  4. Pass the tensor to the model and inspect the output; the example reports an output shape of (1, 262144, 64).

Expected outcome

A Linformer model example returning a sequence tensor with the same shape as its input.

Guide limitations

  • The documented code requires CUDA; CPU execution is not shown.
  • The README describes linear-complexity attention but does not establish performance for this particular configuration or hardware.
  • The README says validation on WikiText-2 is in progress and that the best hyperparameters still need to be found.
Documentation 1 ↗
Evidence and transparency

Sources & review

Official repository ↗
Source record details
Source availability
Active
Sources checked
15 October 2026
Original website title
tatp22/linformer-pytorch · Official homepage title · retained in its source language
How we review sourcesSuggest a correction