Practical overview
When it fits
Use it when a PyTorch model has multiple losses and you want to incorporate GradNorm-based adaptive weighting into the training loop, or compare that approach with fixed weights.
Who it is for
- PyTorch developers training networks with multiple objectives or auxiliary losses.
- Researchers comparing adaptive gradient-based loss balancing with fixed loss weights.
- Developers using Hugging Face Accelerate in a training workflow with multiple losses.
Documented uses
- Adjust loss weights based on gradient information during a multi-loss training loop.
- Choose whether gradient norms are measured at a shared parameter or through an intermediate activation.
- Freeze loss weights or use fixed weights for an experiment.
- Select which losses are subject to GradNorm with a loss mask.
- Pass an Accelerate instance to the loss weighter in an Accelerate-based workflow.
Requirements & limits
The repository provides the package installation command pip install gradnorm-pytorch and declares an MIT license.
- The README’s parameter-based example needs a shared parameter through which the losses contribute gradients.
- Activation-based weighting requires the network to return the intermediate activations and pass them to backward.
- A loss mask can limit which losses are adjusted; fixed weights and freezing are also documented.
- The README documents an Accelerate integration pattern and marks gradient accumulation as handled.
Check current requirements in the official documentation. Source-based guidance does not mean OpenViST has run or independently tested this project.
Add GradNorm weighting to a multi-loss PyTorch training loop
Use the documented GradNormLossWeighter pattern to balance several network losses during training.
Before you start
- A PyTorch network that produces multiple losses.
- A shared backbone parameter whose gradients receive contributions from the losses.
- The gradnorm-pytorch package, installed with the documented command: pip install gradnorm-pytorch.
Steps
Import GradNormLossWeighter from gradnorm_pytorch and create the network and optimizer.
Initialize GradNormLossWeighter with the number of losses and the selected shared parameter as grad_norm_parameters.
Run the network to obtain its losses, then call loss_weighter.backward(losses).
Run the optimizer step and clear gradients with optim.zero_grad().
Expected outcome
The README’s example updates the loss weights through the GradNorm backward call before the optimizer step.
Guide limitations
- The README example uses a mock network; it does not establish behavior for every model or training configuration.
- The recipe requires a suitable shared parameter. The README describes an alternative using intermediate activations when gradients should be measured there.
Pass intermediate activations to GradNorm
Configure the documented activation-based option for measuring gradients through an intermediate network output.
Before you start
- A PyTorch network whose forward pass can return the intermediate activations.
- Multiple losses and the gradnorm-pytorch package.
Steps
Initialize GradNormLossWeighter with grad_norm_parameters=None.
Run the network with return_backbone_outputs=True to obtain losses and backbone output activations.
Call loss_weighter.backward(losses, backbone_output_activations).
Continue with the optimizer step and zeroing gradients as shown in the README.
Expected outcome
The weighter receives the activations through which the losses’ gradients flow, instead of a selected backbone parameter.
Guide limitations
- The network must return the intermediate activations; the README does not provide a general method for modifying arbitrary models to do so.
Sources & review
Source record details
- Source availability
- Active
- Sources checked
- 12 October 2026
- Original website title
- lucidrains/gradnorm-pytorch · Official homepage title · retained in its source language