Lumiere Pytorch: Adapt image U-Nets for video diffusion

Understand the project

Practical overview

Use it when you have a compatible image denoising U-Net and want to explore video denoising in code by adding configurable temporal operations.

Extend a compatible image denoising U-Net to process video tensors.
Configure temporal convolution, attention, and sampling stages by model module name.
Run a documented denoising pass on a noised video tensor.
Experiment with learning temporal parameters while leaving base-model parameters frozen.

Before you choose

The repository is public and specifies the MIT license. The README documents installation with pip.

The demonstrated base model is KarrasUnet; arbitrary U-Net compatibility is not established.
You must identify the model's temporal module locations and keep downsampling and upsampling stages balanced.
The video time dimension must fit the configured temporal downsampling depth.
The README demonstrates a denoising-model call, not a full text-to-video application workflow.
Testing with imagen-pytorch is listed as future work.

Check current requirements in the official documentation. Source-based guidance does not mean OpenViST has run or independently tested this project.

Source-based practical guide

Wrap the README's Karras U-Net with Lumiere Pytorch and pass a noised video tensor through the denoising interface.

Python environment with PyTorch, lumiere-pytorch, and denoising-diffusion-pytorch installed.
A Karras U-Net configuration and temporal module names matching the model structure.

Install lumiere-pytorch with pip; install the PyTorch and denoising-diffusion-pytorch dependencies used by the example.
Create a KarrasUnet and pass it to MPLumiere, setting the time-conditioning argument and the convolution, attention, downsampling, and upsampling module names.
Provide a noised video tensor and a time tensor to the wrapped model; the README example uses a video tensor shaped (2, 3, 8, 256, 256).

The example returns a denoised video tensor with the same shape as its input.

The documented configuration is for KarrasUnet and does not establish compatibility with arbitrary U-Nets.
The temporal dimension must match the configured number of downsampling stages, and the number of downsampling and upsampling stages must be equal.
This example demonstrates a denoising pass, not a complete text-to-video generation workflow.

Evidence and transparency

Source record details

Source availability: Active
Sources checked: 14 October 2026
Original website title: lucidrains/lumiere-pytorch · Official homepage title · retained in its source language