OpenViSTAI · Agents · Blockchain · Crypto
Submit a project
Menu
AI Models

Linear Attention Transformer

Linear Attention Transformer is an open-source Python/PyTorch library that implements Transformer components combining local conventional attention with global linear-complexity attention. It provides interfaces for language models and general Transformers, along with documented encoder–decoder, image-attention, and Linformer configurations.

Explore the practical guide ↓

Independent OpenViST project profile, based on linked sources.

Linear Attention Transformer — OpenViST project profile in AI Models.
OpenViST project illustration. See the linked sources for product details.
Understand the project

Practical overview

When it fits

Use it when building or experimenting with Transformer models in Python and a linear-complexity global-attention component is relevant to the sequence-length design.

Who it is for

  • PyTorch developers prototyping language models for longer token sequences
  • Researchers comparing local attention with linear-complexity global attention
  • Developers building encoder–decoder models that pass contextual representations

Documented uses

  • Configure a causal or non-causal language-model component.
  • Build a Transformer with chosen sequence length, depth, head count, and local-attention settings.
  • Connect encoder-produced context to a decoder.
  • Try the documented image-attention component or fixed-length Linformer settings.
Before you choose

Requirements & limits

The repository is public and licensed under MIT. The documented installation command is `pip install linear-attention-transformer`.

  • The architecture combines local (QKᵀ)V attention with global Q(KᵀV) attention.
  • Causal configurations expose a blindspot setting; the README recommends pairing it with a local window at least as large.
  • The documented Linformer option is limited to non-autoregressive fixed-length models.
  • Examples use PyTorch tensors, and the language-model and Transformer snippets place models and inputs on CUDA.

Check current requirements in the official documentation. Source-based guidance does not mean OpenViST has run or independently tested this project.

Source-based practical guide

Install the library and prepare a language-model example

Install the package and configure the documented language-model interface for a causal sequence.

Before you start

  • Python environment with PyTorch available
  • A CUDA-enabled setup for the README's shown model and tensor placement

Steps

  1. Install the package with the command below.

    Documented command
    pip install linear-attention-transformer

  2. Import LinearAttentionTransformerLM from linear_attention_transformer.

  3. Configure the model with a vocabulary size, dimensions, depth, maximum sequence length, and attention settings; the README example uses causal=True.

  4. Prepare token-ID input with the configured vocabulary and sequence length, then pass it to the model.

Expected outcome

A configured language-model component that accepts token IDs; the README example shows an output shaped `(1, 8192, 512)`.

Guide limitations

  • The documented example uses .cuda() for both model and input.
  • For causal use, the README says blindspot size should be paired with a local-attention window at least as large.
Documentation 1 ↗
Source-based practical guide

Configure fixed-length Linformer attention

Set up the library's documented Linformer option for a non-autoregressive, fixed-length model.

Before you start

  • Python environment with PyTorch available
  • A fixed sequence length and a non-autoregressive use case

Steps

  1. Import LinearAttentionTransformerLM and LinformerSettings.

  2. Create settings with a chosen k value, as in the README example using k=256.

  3. Pass the settings to LinearAttentionTransformerLM through linformer_settings, alongside the model's sequence length and other parameters.

Expected outcome

A language-model component configured to use the documented Linformer attention option.

Guide limitations

  • The README says this Linformer option is for non-autoregressive models with fixed sequence length.
Documentation 1 ↗
Evidence and transparency

Sources & review

Official repository ↗
Source record details
Source availability
Active
Sources checked
14 October 2026
Original website title
lucidrains/linear-attention-transformer · Official homepage title · retained in its source language
How we review sourcesSuggest a correction