Practical overview
When it fits
Use it when you want a code-level starting point for experimenting with Taylor-series attention modules and can integrate them into your own model workflow.
Who it is for
- Researchers investigating efficient language-model attention mechanisms.
- Engineers prototyping attention layers or comparing attention designs in Python.
Documented uses
- Add a Taylor-series attention layer to a Python experiment.
- Try self-attention, cross-attention or causal autoregressive configurations.
- Explore higher-order Taylor Series Poly Attention and consider its head-dimension compute cost.
Requirements & limits
The repository is public, specifies an MIT license and documents installation through pip.
- The autoregressive example requires the additional
pytorch-fast-transformerspackage and usescausal=True. - The README cautions that the second Taylor expansion has O(D^3) scaling in the attention-head dimension.
- The documentation provides usage examples but does not include benchmark results.
Check current requirements in the official documentation. Source-based guidance does not mean OpenViST has run or independently tested this project.
Configure an autoregressive attention example
Set up the repository's documented causal attention variant in Python.
Before you start
- Python environment with the
taylor-series-linear-attentionpackage installed. - Install
pytorch-fast-transformersfor the autoregressive variant. - PyTorch tensors, as shown in the example.
Steps
Install the package with the command below.
pip install taylor-series-linear-attentionInstall the additional dependency with the command below.
pip install pytorch-fast-transformersImport
TaylorSeriesLinearAttnand instantiate it withcausal=True; the example also setsrotary_emb=True.Pass an input tensor to the module, following the README's example.
Expected outcome
A configured causal attention module that returns an output tensor for the supplied input.
Guide limitations
- The documentation does not specify environment or hardware requirements beyond the named dependency and PyTorch-based example.
- The README warns that linear attention shifts quadratic computation to the head dimension and that a second Taylor expansion scales as O(D^3).
Set up a cross-attention example
Configure the documented module to attend from an input tensor to a separate context tensor.
Before you start
- Python environment with
taylor-series-linear-attentioninstalled. - PyTorch tensors for the input, context and context mask.
Steps
Install the package with the command below.
pip install taylor-series-linear-attentionImport
TaylorSeriesLinearAttnand instantiate it with the model dimension, head dimension and number of heads.Provide the input tensor and context tensor to the module, with the context mask as shown in the README.
Expected outcome
A cross-attention output tensor with the same shape as the input in the documented example.
Guide limitations
- The example uses specified tensor dimensions but does not establish performance for other workloads.
- The README's stated attention-head computational trade-off also applies when evaluating the approach.
Sources & review
Source record details
- Source availability
- Active
- Sources checked
- 12 October 2026
- Original website title
- lucidrains/taylor-series-linear-attention · Official homepage title · retained in its source language