OpenViSTAI · Agents · Blockchain · Crypto
Submit a project
Menu
AI Developer Tools & SDKs

Rotary Embedding Torch

Rotary Embedding Torch is a standalone Python/PyTorch library that applies rotary positional embeddings to transformer tensors. It can rotate information across tensor axes for fixed or learned embeddings and includes utilities for cached-key inference, axial positional layouts, XPos, interpolated sequence positions and fused attention with rotary processing.

Explore the practical guide ↓

Independent OpenViST project profile, based on linked sources.

Rotary Embedding Torch — OpenViST project profile in AI Developer Tools & SDKs.
OpenViST project illustration. See the linked sources for product details.
Understand the project

Practical overview

When it fits

Use it when a PyTorch transformer needs rotary positional processing as an integrated attention component, particularly when the model has cached keys, multidimensional positional axes or a need to compare positional-encoding approaches.

Who it is for

  • PyTorch researchers implementing transformer architectures
  • Engineers adding or comparing positional-encoding methods
  • Developers working with video or other multidimensional transformer inputs
  • Teams handling autoregressive inference with key-value caches

Documented uses

  • Add rotary positional encoding to a custom transformer.
  • Rotate queries and keys before attention calculation.
  • Generate axial frequencies for multidimensional inputs.
  • Handle query-position offsets during cached-key inference.
  • Experiment with XPos or interpolated sequence positions.
Before you choose

Requirements & limits

The repository is public, MIT-licensed and provides installation through pip. The documented examples use Python and PyTorch.

  • It is a library component that must be integrated into an existing transformer rather than a ready-to-use model or hosted application.
  • The standard example expects sequence length and feature dimension to be the final tensor dimensions.
  • XPos is documented for autoregressive transformers.
  • The fused Flash Attention path uses Triton when available and falls back to a reference PyTorch implementation.
  • The README notes reported community concerns about the effectiveness of interpolated sequence positions.

Check current requirements in the official documentation. Source-based guidance does not mean OpenViST has run or independently tested this project.

Source-based practical guide

Add rotary embeddings to PyTorch attention

Install the library and apply rotary positions to transformer queries and keys before attention.

Before you start

  • Python and PyTorch
  • A transformer attention implementation with query and key tensors whose final dimensions are sequence length and feature dimension
  • The rotary-embedding-torch package

Steps

  1. Install the package with pip install rotary-embedding-torch.

  2. Create RotaryEmbedding with the intended embedding dimension.

  3. Pass query and key tensors to rotate_queries_or_keys after splitting out the attention heads.

  4. Continue with the usual query-key dot product and softmax attention calculation.

Expected outcome

The transformer attention path receives rotated queries and keys from the library.

Guide limitations

  • The documented tensor layout requires sequence length and feature dimension at the end, with preceding dimensions such as batch and heads.
  • The source example describes the integration pattern but does not establish model-training results for a particular implementation.
Documentation 1 ↗
Evidence and transparency

Sources & review

Official repository ↗
Source record details
Source availability
Active
Sources checked
3 October 2026
Original website title
lucidrains/rotary-embedding-torch · Official homepage title · retained in its source language
How we review sourcesSuggest a correction