Practical overview
When it fits
Use it when you want a code-level starting point for recurrent-memory transformer experiments in PyTorch, especially when investigating how state can pass between sequence segments.
Who it is for
- Researchers exploring recurrent-memory and long-context transformer architectures.
- PyTorch developers prototyping sequence models that carry information across segments.
Documented uses
- Configure a Recurrent Memory Transformer model for autoregressive or other documented causal settings.
- Carry returned memory state from one input segment into the next.
- Explore XL memories as an additional state passed across segments.
- Study a wrapper-based example for training on longer sequences with memory-replay backpropagation.
Requirements & limits
The repository is public and MIT-licensed. The README documents installation with pip; it does not state a price or account requirement.
- The implementation is a development library, not a hosted model or ready-made conversational application.
- Its examples make memory state explicit: callers pass returned memory into later segment calls.
- The README includes a CUDA-based long-sequence training example, while the platform classification lists Linux, Windows and macOS.
- The cited one-million-token result is attributed to follow-up research and is not a guarantee for this implementation or a particular configuration.
Check current requirements in the official documentation. Source-based guidance does not mean OpenViST has run or independently tested this project.
Pass recurrent memory across sequence segments
Configure the model and carry its returned memory into successive segment calls.
Before you start
- Install the recurrent-memory-transformer-pytorch package with pip.
- Have PyTorch available and prepare token sequences matching the configured vocabulary and segment length.
Steps
Import RecurrentMemoryTransformer and configure vocabulary size, memory-token count, model dimensions, depth, causal mode, attention heads and segment length.
Call the model with a token segment to obtain logits and memory.
Pass the returned memory to the model with the next segment, repeating for subsequent segments.
Expected outcome
The documented pattern returns predictions and updated memory for each segment, with memory state carried into later calls.
Guide limitations
- The example uses randomly generated token IDs; it does not document a complete dataset preparation or evaluation workflow.
- The README describes the memory-token count as a bottleneck for information passed forward.
Configure the model with XL memories
Use the documented XL-memory mode and pass its state alongside recurrent memory between segments.
Before you start
- Install the recurrent-memory-transformer-pytorch package with pip.
- Have PyTorch available and prepare token segments matching the configured vocabulary and segment length.
Steps
Configure RecurrentMemoryTransformer with use_xl_memories=True and set xl_mem_len.
Call the model with the first segment to obtain logits, recurrent memory and XL-memory state.
Pass both returned memory values into each subsequent segment call.
Expected outcome
The example produces logits and updated recurrent and XL-memory state across successive segments.
Guide limitations
- The README presents this as a usage example and does not specify evaluation criteria or performance guarantees.
- Its comment on choosing XL-memory length is tentative.
Sources & review
Source record details
- Source availability
- Active
- Sources checked
- 14 October 2026
- Original website title
- lucidrains/recurrent-memory-transformer-pytorch · Official homepage title · retained in its source language