Practical overview
When it fits
Use it when an existing PyTorch training workflow needs an optimizer alternative to AdamW and the team is prepared to tune hyperparameters for its workload.
Who it is for
- PyTorch developers training deep-learning models.
- Researchers and engineers benchmarking optimizers against AdamW.
- Teams building reusable components for PyTorch training stacks.
Documented uses
- Add Lion parameter updates to a PyTorch training workflow.
- Compare Lion and AdamW on a model-training workload.
- Tune learning rate and weight decay when changing optimizers.
- Optionally use a Triton fused kernel for parameter updates.
Requirements & limits
The repository is publicly visible, active in the supplied status data, and MIT-licensed. The documented installation methods are pip and conda. No account, pricing or service-availability details are provided.
- Lion is a PyTorch optimizer, not a replacement for the surrounding training framework.
- The repository recommends learning rates typically three to ten times smaller than AdamW and weight decay three to ten times larger, with schedule values changed together.
- Reported results are sensitive to workload details including learning rate, batch size, data or augmentation and architecture.
- The documented Triton mode requires a separate Triton installation and is intended for a CUDA kernel.
- The repository states that high batch sizes, specifically 64 or above, are the setting it would recommend for the optimizer.
Check current requirements in the official documentation. Source-based guidance does not mean OpenViST has run or independently tested this project.
Replace an AdamW optimizer with Lion in a PyTorch training loop
Use Lion-based parameter updates while retaining a PyTorch model-training workflow.
Before you start
- A PyTorch model and training loop.
- The lion-pytorch package installed with pip or conda.
- Learning-rate and weight-decay values to configure the optimizer.
Steps
Install lion-pytorch with pip or conda.
Import Lion and instantiate it with the model parameters.
Set the learning rate and weight decay, taking the repository’s AdamW comparison guidance into account.
Run the existing backward, optimizer-step and zero-gradient stages with the Lion optimizer.
Expected outcome
The training loop uses Lion for parameter updates while the surrounding PyTorch workflow remains in place.
Guide limitations
- The repository does not provide a complete AdamW-to-Lion migration example.
- The documented tuning guidance is not a guarantee of improved results; reported outcomes vary with learning rate and workload.
- The evidence does not document a successful run for a particular model.
Sources & review
Source record details
- Source availability
- Active
- Sources checked
- 30 September 2026
- Original website title
- lucidrains/lion-pytorch · Official homepage title · retained in its source language