Practical overview
When it fits
Use it as a research starting point when investigating reasoning tokens or gradient flow in PyTorch transformer attention, rather than as a validated training recipe.
Who it is for
- PyTorch developers prototyping transformer training methods.
- Researchers exploring reasoning-token access and attention gradient flow.
Documented uses
- Configure reasoning tokens per time step and restrict how far into the future tokens can use them.
- Experiment with separate stop-gradient masks for query, key, and value tensors.
- Compute a model loss and retrieve logits in a transformer example.
Requirements & limits
The repository is public and lists an MIT license. Its README gives installation instructions using pip and imports PyTorch.
- The repository explicitly labels the implementation work in progress.
- The Transformer example exposes both reasoning-token count and a future-use limit.
- The attention function accepts separate masks for queries, keys, and values.
- The README presents subset-of-attention-head experiments as a todo, not an implemented capability.
Check current requirements in the official documentation. Source-based guidance does not mean OpenViST has run or independently tested this project.
Run a self-reasoning-token Transformer example
Create a Transformer with reasoning tokens, calculate a loss, run backpropagation, and obtain logits.
Before you start
- PyTorch
- The self-reasoning-tokens-pytorch package
Steps
Install the package with the command below.
pip install self-reasoning-tokens-pytorchImport
torchandTransformer; create a model withdim=512,depth=4,num_tokens=256, andstop_grad_next_tokens_to_reason=True.Create an input with
x = torch.randint(0, 256, (1, 4)), then call the model withnum_reason_tokens=4,num_steps_future_can_use_reason=16, andreturn_loss=True.Call
loss.backward(), then callmodel(x, num_reason_tokens=4)to obtain logits.
Expected outcome
The example produces a loss for backpropagation and then obtains logits from the model.
Guide limitations
- The project describes itself as work in progress and an exploration; the README does not establish this example as a verified test.
- The README says the future-use setting controls which tokens may attend to reasoning tokens; it does not document additional training or evaluation procedures.
Call attention with separate query, key, and value stop-gradient masks
Pass stop-gradient masks for queries, keys, and values to the library's attention function.
Before you start
- PyTorch
- The self-reasoning-tokens-pytorch package
Steps
Import
torchandstop_graddable_attn.Create
q,k, andvtensors withtorch.randn(2, 8, 1024, 64)for each.Create a Boolean mask with
torch.randint(0, 2, (8, 1024, 1024)).bool().Call
stop_graddable_attn(q, k, v, causal=True, q_stop_grad_mask=stop_grad_mask, k_stop_grad_mask=stop_grad_mask, v_stop_grad_mask=stop_grad_mask).
Expected outcome
The example returns an attention output whose documented shape is `(2, 8, 1024, 64)`.
Guide limitations
- The README presents this as an exploration and does not establish the example as a verified test.
- The README illustrates one mask shape and uses the same mask for queries, keys, and values; it does not document other mask layouts or validation behavior.
Sources & review
Source record details
- Source availability
- Active
- Sources checked
- 16 October 2026
- Original website title
- lucidrains/self-reasoning-tokens-pytorch · Official homepage title · retained in its source language