Practical overview
When it fits
Use it when a PyTorch prototype needs transformer attention augmented with retrieved memories, or when exploring KNN memory alongside Transformer-XL memory.
Who it is for
- PyTorch developers prototyping memory-augmented transformer models.
- Researchers and engineers experimenting with approximate-nearest-neighbor retrieval in attention.
- Developers exploring KNN and Transformer-XL memory together across successive model inputs.
Documented uses
- Configure a transformer with selected layers that use ANN memories.
- Create and pass batch-specific KNN memories into model calls.
- Retrieve stored key-value memories and manage their updates or clearing.
- Experiment with local attention combined with retrieved distant context.
- Run the repository's Enwik8 training example.
Requirements & limits
The project is a public GitHub repository with an MIT license; its README documents installation through pip.
- The implementation's local/distant attention combination and cosine-similarity KNN attention differ slightly from the cited paper's setup.
- The README documents both KNN-only usage and an example combining KNN memories with Transformer-XL memories.
- The transformer example resets its ANN memory collection when its configured capacity is reached, citing a Faiss removal limitation.
- A forward-call option can make passed KNN memories read-only by disabling memory additions.
Check current requirements in the official documentation. Source-based guidance does not mean OpenViST has run or independently tested this project.
Install the library and initialize KNN memories
Set up a MemorizingTransformer and a batch-matched KNN memory collection as shown in the README.
Before you start
- Python environment with PyTorch available.
- Install the package with pip using the documented package name.
Steps
Run the command below.
pip install memorizing-transformers-pytorchImport
MemorizingTransformerand configure a model, including token count, dimensions, depth, memory layers, capacity, and number of retrieved memories.Create a KNN memory collection with
model.create_knn_memories(batch_size=...)using the intended batch size.Pass the collection to the model with
model(data, knn_memories=knn_memories). Setadd_knn_memory=Falseon the forward call when the example's read-only behavior is wanted.
Expected outcome
A configured model call returns logits while using the supplied KNN memory collection; the example also shows how to keep that collection from being updated during a call.
Guide limitations
- The README notes that the transformer example resets the ANN memories after its configured capacity because of a Faiss limitation on removing entries.
- The sample inputs and dimensions are illustrative values in the README, not a guarantee of suitability for a particular workload.
Run the documented Enwik8 training script
Start the repository's Enwik8 training example using its documented command.
Before you start
- A local checkout of the repository.
- Python dependencies and environment configured for the project.
Steps
Change to the repository directory.
Run the command below as shown under the README's Enwik8 training section.
python train.py
Expected outcome
Invokes the repository's documented Enwik8 training script.
Guide limitations
- The README gives the command but does not specify training parameters or expected results in the cited section.
Sources & review
Source record details
- Source availability
- Active
- Sources checked
- 15 October 2026
- Original website title
- lucidrains/memorizing-transformers-pytorch · Official homepage title · retained in its source language