Lora: Add LoRA adaptation to PyTorch language models

Understand the project

Practical overview

When it fits

Use Lora when integrating task-specific low-rank adaptation into a PyTorch model project and handling adaptation parameters separately from pretrained weights.

Who it is for

Developers building PyTorch language-model fine-tuning workflows.
Teams adapting PyTorch models from the Hugging Face ecosystem.

Documented uses

Add low-rank adaptation layers to selected parts of a PyTorch model.
Train only LoRA parameters while keeping other model parameters frozen.
Save and load LoRA-specific checkpoints alongside a pretrained model checkpoint.
Explore adaptation of Transformer projections, embeddings, or MLP layers.

Before you choose

Requirements & limits

The repository states that PyTorch is the only supported framework.
The quickstart supports nn.Linear, nn.Embedding, and nn.Conv2d, as well as MergedLinear for certain combined projections.
Loading an adaptation checkpoint requires the original pretrained checkpoint to be loaded first.
The repository documents installation through pip or directly from GitHub and is licensed under MIT.
The README notes that Hugging Face PEFT also supports LoRA.

Check current requirements in the official documentation. Source-based guidance does not mean OpenViST has run or independently tested this project.

Source-based practical guide

Set up LoRA layers and checkpoints in a PyTorch model

Integrate LoRA layers into a PyTorch model workflow and prepare LoRA-specific parameters for training and checkpoint handling.

Before you start

A Python environment with PyTorch and a model project.
A pretrained model checkpoint when loading an existing model or LoRA checkpoint.

Steps

Install loralib with the command below, or install from the repository using the documented Git URL.
Documented command
```
pip install loralib
```
Replace supported layers with their loralib counterparts, such as lora.Linear, and choose a rank for the adaptation matrices.
Before training, call lora.mark_only_lora_as_trainable(model) to mark LoRA parameters as trainable.
Save LoRA-only parameters with lora.lora_state_dict(model); when loading, load the pretrained checkpoint first and the LoRA checkpoint afterward with strict=False.

Expected outcome

The model workflow uses LoRA layers and can save or load a checkpoint containing LoRA parameters separately from the pretrained model checkpoint.

Guide limitations

The library states that it supports PyTorch only.
Documented replacement layers are nn.Linear, nn.Embedding, and nn.Conv2d, with MergedLinear for certain combined projections.
The steps describe integration and checkpoint handling; they do not specify model-specific training settings or guarantee a particular result.

Documentation 1 ↗

Evidence and transparency

Sources & review

Official repository ↗Guide documentation ↗

Source record details

Source availability: Active
Sources checked: 15 October 2026
Original website title: microsoft/LoRA · Official homepage title · retained in its source language

How we review sources Suggest a correction

Related options

Hugging Face PEFT — The repository README states that LoRA is supported by the PEFT library; this is a separate library route for using LoRA.

Lora