OpenViSTAI · Agents · Blockchain · Crypto
Submit a project
Menu
AI Models

Memformer

Memformer is an open-source Python/PyTorch implementation of a memory-augmented Transformer, with attention-updated memory slots that can be carried across input segments and a Memory-Replay Back-Propagation utility for segmented-sequence training.

Explore the practical guide ↓

Independent OpenViST project profile, based on linked sources.

Memformer — OpenViST project profile in AI Models.
OpenViST project illustration. See the linked sources for product details.
Understand the project

Practical overview

When it fits

Use Memformer when a PyTorch sequence-model experiment needs memory to persist across consecutive input segments, or when exploring the included replay-based training utility.

Who it is for

  • Researchers prototyping memory-augmented sequence models.
  • Python and PyTorch developers building encoder-only or encoder-decoder experiments.
  • Practitioners exploring training on long sequences split into segments.

Documented uses

  • Pass learned memory slots from one input segment to the next.
  • Build an encoder-only representation workflow over successive segments.
  • Prototype encoder-decoder processing with source segments and target inputs.
  • Use the provided memory-replay utility with source and target masks for segmented-sequence training.
Before you choose

Requirements & limits

The repository lists an MIT license and documents installation with `pip install memformer`.

  • The examples require configuring token counts, dimensions, sequence lengths, attention heads, depth and memory slots.
  • The encoder-only example explicitly returns encoded outputs and memory slots for reuse on the next segment.
  • The memory-replay example uses CUDA tensors and the .cuda() method.

Check current requirements in the official documentation. Source-based guidance does not mean OpenViST has run or independently tested this project.

Source-based practical guide

Set up Memformer for a PyTorch experiment

Install the package and configure a documented encoder-only or encoder-decoder model.

Before you start

  • Python environment with PyTorch available.

Steps

  1. Install the package with the command below.

    Documented command
    pip install memformer

  2. Import Memformer from memformer.

  3. Choose the documented encoder-only or full encoder-decoder configuration and set its model dimensions, token counts, sequence lengths and memory slots.

Expected outcome

A configured Memformer model ready for the documented segment-processing examples.

Guide limitations

  • The setup example does not document dependency versions or platform-specific installation instructions.
Documentation 1 ↗
Source-based practical guide

Pass memory through an encoder-decoder sequence

Use the documented model calls to carry memory across source segments and compute a target loss.

Before you start

  • An installed Memformer package and PyTorch.
  • A configured encoder-decoder Memformer model.

Steps

  1. Pass the first source segment to the model and retain its returned memory slots.

  2. Pass the next source segment with the retained slots using mems.

  3. Pass a later source segment together with the target and the prior memory slots.

  4. Use the returned loss in the documented loss.backward() call.

Expected outcome

The example produces encoder outputs and updated memory across segments, then computes a loss for a target-bearing call.

Guide limitations

  • The example uses synthetic token tensors; it does not document data preparation or a complete training loop.
Documentation 1 ↗
Evidence and transparency

Sources & review

Official repository ↗
Source record details
Source availability
Active
Sources checked
18 October 2026
Original website title
lucidrains/memformer · Official homepage title · retained in its source language
How we review sourcesSuggest a correction