OpenViSTAI · Agents · Blockchain · Crypto
Submit a project
Menu
AI Models

Infini Transformer Pytorch

Infini Transformer Pytorch is a Python/PyTorch implementation of Infini-Transformer with linear attention for compressing past memories and recurrent memory inputs and outputs for processing sequences in segments. It includes a transformer model and a wrapper for segmented training, evaluation and autoregressive generation.

Explore the practical guide ↓

Independent OpenViST project profile, based on linked sources.

Infini Transformer Pytorch — OpenViST project profile in AI Models.
OpenViST project illustration. See the linked sources for product details.
Understand the project

Practical overview

When it fits

Use it when you want a Python/PyTorch implementation for experiments that carry compressed memory across sequence segments, rather than a hosted model or ready-made chat application.

Who it is for

  • Researchers exploring recurrent-memory and linear-attention transformer designs.
  • Developers building PyTorch experiments for long-sequence training or generation.

Documented uses

  • Pass memory from one token-sequence segment into the next.
  • Train an autoregressive transformer using a wrapper that segments long input sequences.
  • Calculate evaluation loss and generate a continuation from a prompt.
Before you choose

Requirements & limits

  • The package installation command is pip install infini-transformer-pytorch.
  • The training example uses .cuda() for the wrapper and sequence, while the direct model example does not show that step.
  • The wrapper's memory-detachment interval is identified in the README as a confusing hyperparameter.
  • The code accepts token IDs; the README's example does not specify a text-tokenization workflow.

Check current requirements in the official documentation. Source-based guidance does not mean OpenViST has run or independently tested this project.

Source-based practical guide

Install the package and prepare a model

Install the library and configure an InfiniTransformer with the documented constructor parameters.

Before you start

  • Python environment with PyTorch installed.
  • The README documents pip install infini-transformer-pytorch.
  • Choose token count, model dimension, depth, head dimension and number of heads.

Steps

  1. Install with the command below.

    Documented command
    pip install infini-transformer-pytorch

  2. Import InfiniTransformer from infini_transformer_pytorch.

  3. Instantiate it with the required model parameters, as shown in the README.

Expected outcome

A configured InfiniTransformer object ready to receive token sequences.

Guide limitations

  • This covers installation and model construction only; it does not establish that a particular environment or configuration has been tested.
  • The README's example uses random integer tokens and does not describe tokenization of text.
Documentation 1 ↗
Source-based practical guide

Configure the segmented-training wrapper

Set up the documented wrapper to segment a sequence and handle recurrent-memory detachment during training.

Before you start

  • An instantiated InfiniTransformer model.
  • PyTorch and a CUDA-capable environment for the README's .cuda() example.
  • A sequence tensor and chosen segment length and memory-detachment interval.

Steps

  1. Create an InfiniTransformer model using the documented constructor.

  2. Pass it to InfiniTransformerWrapper with segment_length and detach_mems_every_num_segments settings.

  3. Move the wrapper to CUDA as in the example.

  4. Pass a sequence to the wrapper with backward=True to invoke the documented segmented training path.

Expected outcome

The wrapper segments the input and accumulates gradients while detaching memories according to the configured interval.

Guide limitations

  • The README says the detach_mems_every_num_segments hyperparameter is confusing.
  • The steps describe the documented example and are not a compatibility or execution guarantee.
Documentation 1 ↗
Evidence and transparency

Sources & review

Official repository ↗
Source record details
Source availability
Active
Sources checked
16 October 2026
Original website title
lucidrains/infini-transformer-pytorch · Official homepage title · retained in its source language
How we review sourcesSuggest a correction