OpenViSTAI · Agents · Blockchain · Crypto
Submit a project
Menu
AI Developer Tools & SDKs

Product Key Memory

Product Key Memory is a Python/PyTorch library that supplies a PKM module for Transformer models and helper functions for separating PKM value parameters into optimizer groups.

Explore the practical guide ↓

Independent OpenViST project profile, based on linked sources.

Product Key Memory — OpenViST project profile in AI Developer Tools & SDKs.
OpenViST project illustration. See the linked sources for product details.
Understand the project

Practical overview

When it fits

Use it when building or adapting a PyTorch Transformer and you specifically want to integrate Product Key Memory in place of feed-forward layers, or to configure distinct optimizer learning rates for PKM values.

Who it is for

  • PyTorch developers modifying Transformer architectures.
  • Researchers and engineers configuring optimizer parameter groups for models that contain PKM layers.

Documented uses

  • Replace a Transformer feed-forward layer with a Product Key Memory module.
  • Pass sequence representations and an input mask through the PKM layer.
  • Assign PKM value parameters a learning rate distinct from other model parameters.
Before you choose

Requirements & limits

  • The API example requires choosing model dimension, head count, head dimension, number of subkeys and top-k.
  • The README says the value count is the square of the number of subkeys, so that setting affects the memory configuration.
  • The repository identifies an MIT license.

Check current requirements in the official documentation. Source-based guidance does not mean OpenViST has run or independently tested this project.

Source-based practical guide

Add a PKM layer to a Transformer

Install the package and call its PKM module with sequence representations and an input mask.

Before you start

  • Python environment with PyTorch available.
  • A Transformer integration point where a feed-forward layer can be replaced.

Steps

  1. Install the package with the command below.

    Documented command
    pip install product-key-memory

  2. Import PKM from product_key_memory and configure its dimension, heads, head dimension, number of subkeys and top-k setting.

  3. Pass sequence representations and a Boolean input mask to the PKM module.

Expected outcome

The module returns transformed values; the example output has the same batch, sequence and feature dimensions as its input.

Guide limitations

  • The example documents the module call but not a complete model-training or evaluation workflow.
  • The README recommends a head dimension of 128 for best results but does not provide comparative measurements.
Documentation 1 ↗
Source-based practical guide

Set a separate learning rate for PKM values

Group PKM value parameters separately from other model parameters when configuring an optimizer.

Before you start

  • A model containing PKM modules.
  • PyTorch and its Adam optimizer.

Steps

  1. Import fetch_pkm_value_parameters from product_key_memory.

  2. Call the helper with the root model to obtain PKM and other parameter groups.

  3. Pass the groups to torch.optim.Adam and assign the PKM group its own learning rate.

Expected outcome

The optimizer is configured with separate parameter groups, allowing the PKM value parameters to use a distinct learning rate.

Guide limitations

  • The example shows optimizer configuration, not a training loop or guidance for choosing learning rates.
  • The alternative fetch_optimizer_parameters helper is described only for the case where PKM parameters are the only parameters requiring a different learning rate.
Documentation 1 ↗
Evidence and transparency

Sources & review

Official repository ↗
Source record details
Source availability
Active
Sources checked
12 October 2026
Original website title
lucidrains/product-key-memory · Official homepage title · retained in its source language
How we review sourcesSuggest a correction