Practical overview
When it fits
Use it when researching or prototyping cosine-similarity attention and when the documented PyTorch examples match the model's attention inputs. Evaluate both model quality and hardware-specific performance before relying on it.
Who it is for
- Deep-learning developers prototyping attention mechanisms in Python models.
- Researchers comparing cosine-similarity attention behavior across sequence types and model tasks.
Documented uses
- Replace or compare an attention operation with a fused cosine-similarity implementation.
- Prepare self-attention or cross-attention calls, including causal and key/value-masked examples.
- Explore operations on normalized queries and keys before the attention step.
Requirements & limits
The project is a public GitHub repository and declares an MIT license. Its README documents installation with pip.
- The README reports worse evaluation FID scores in experiments, despite losses not reflecting the difference, and advises caution.
- The project says autoregressive and variable-length cases may be faster, while non-autoregressive unmasked attention remains slower on an A100 for F16.
- For sequences longer than 2,048, the README describes memory efficiency relative to regular attention; older graphics cards may involve a speed-versus-memory tradeoff.
- The documented examples use PyTorch tensors on CUDA.
Check current requirements in the official documentation. Source-based guidance does not mean OpenViST has run or independently tested this project.
Install the library and prepare a self-attention call
Install the package and prepare the documented PyTorch self-attention example with CUDA tensors.
Before you start
- Python package installation via pip
- PyTorch
- CUDA tensors, as used in the repository example
Steps
Install the package with the command below.
pip install flash-cosine-sim-attentionImport
torchandflash_cosine_sim_attentionfromflash_cosine_sim_attention.Create query, key and value tensors with the documented shape
(1, 8, 1024, 64)on CUDA.Pass the three tensors to
flash_cosine_sim_attentionas shown in the self-attention example.
Expected outcome
The documented example calls the attention function with query, key and value tensors and shows an output shape of `(1, 8, 1024, 64)`.
Guide limitations
- The guide reflects the repository example and does not establish hardware compatibility or performance.
- The project labels the implementation work in progress and advises caution because evaluation FID scores were reported as worse in experiments.
Prepare a causal cross-attention call
Use the documented function signature and tensor shapes for causal cross-attention.
Before you start
- PyTorch
- CUDA tensors, as used in the repository example
Steps
Import
torchandflash_cosine_sim_attentionfromflash_cosine_sim_attention.Create a query tensor with shape
(1, 8, 1024, 64)and key and value tensors with shape(1, 8, 2048, 64), on CUDA.Call
flash_cosine_sim_attention(q, k, v, causal=True).
Expected outcome
The README example shows an output shape of `(1, 8, 1024, 64)` for this causal cross-attention call.
Guide limitations
- This describes the documented example only and does not establish compatibility or benchmark results.
- The repository notes that performance varies by attention workload and hardware.
Sources & review
Source record details
- Source availability
- Active
- Sources checked
- 8 October 2026
- Original website title
- lucidrains/flash-cosine-sim-attention · Official homepage title · retained in its source language