OpenViSTAI · Agents · Blockchain · Crypto
Submit a project
Menu
Image & Video AI

Video Diffusion Pytorch

Video Diffusion Pytorch is a Python and PyTorch implementation of diffusion models for video synthesis, using a space-time factored 3D U-Net to extend image diffusion to video.

Explore the practical guide ↓

Independent OpenViST project profile, based on linked sources.

Video Diffusion Pytorch — OpenViST project profile in Image & Video AI.
OpenViST project illustration. See the linked sources for product details.
Understand the project

Practical overview

When it fits

Use it when you want to study or adapt a PyTorch video-diffusion implementation and are prepared to configure and train the model rather than use a hosted generator.

Who it is for

  • Researchers exploring diffusion-based video synthesis
  • Python developers adapting PyTorch video-generation models
  • Practitioners experimenting with text-conditioned video or GIF-based model training

Documented uses

  • Configure and sample a diffusion model that produces video tensors
  • Condition video generation on text embeddings or descriptions
  • Train a model using GIF files of configured dimensions and frame count
  • Explore a proposed image-and-video co-training attention option
Before you choose

Requirements & limits

The repository is public and lists an MIT license.

  • The README’s examples involve model training before sampling, so this is an implementation for development and experimentation rather than a turnkey generation service.
  • Text conditioning is shown through BERT-large embeddings or direct descriptions with the BERT text-conditioning option enabled.
  • Training data must match the configured image dimensions and frame count.
  • The README describes its present-focused co-training option as an implementation guess.

Check current requirements in the official documentation. Source-based guidance does not mean OpenViST has run or independently tested this project.

Source-based practical guide

Prepare a GIF training workflow

Configure the documented Trainer for training a diffusion model on a folder of GIFs.

Before you start

  • Python and PyTorch
  • The video-diffusion-pytorch package
  • A folder of GIF files matching the configured image dimensions and frame count

Steps

  1. Install the package with the command below.

    Documented command
    pip install video-diffusion-pytorch

  2. Create an Unet3D model and wrap it in GaussianDiffusion with an image size, frame count, timestep count and loss type.

  3. Initialize Trainer with the diffusion model and GIF folder, then set training parameters such as batch size, learning rate and total steps.

  4. Call trainer.train(); the example says GIF samples and model parameters are periodically saved in ./results.

Expected outcome

A configured training run that periodically saves sampled GIFs and diffusion model parameters.

Guide limitations

  • Training GIFs must have the configured image dimensions and number of frames.
  • The README’s example uses .cuda() and does not document hardware setup or requirements.
Documentation 1 ↗
Source-based practical guide

Configure text-conditioned sampling

Set up the documented model path for conditioning video diffusion on text descriptions.

Before you start

  • Python and PyTorch
  • The video-diffusion-pytorch package
  • Text descriptions and the BERT text-conditioning option, or text embeddings prepared with BERT-large

Steps

  1. Install the package with the command below.

    Documented command
    pip install video-diffusion-pytorch

  2. Create an Unet3D model with use_bert_text_cond=True for string descriptions, or configure cond_dim for supplied text embeddings.

  3. Initialize GaussianDiffusion with the model and the desired image size, number of frames, timesteps and loss type.

  4. Pass the descriptions or embeddings as cond when calling the documented sampling method.

Expected outcome

A configured sampling call that conditions generated video tensors on text.

Guide limitations

  • The README shows configuration and code examples but does not provide a pretrained model or promise immediate usable samples.
  • The examples describe training the model before sampling.
Documentation 1 ↗
Evidence and transparency

Sources & review

Official repository ↗
Source record details
Source availability
Active
Sources checked
15 October 2026
Original website title
lucidrains/video-diffusion-pytorch · Official homepage title · retained in its source language
How we review sourcesSuggest a correction

Related options

Imagen-pytorch — The repository README points to Imagen-pytorch as the place where further text-to-video synthesis developments will be centralized.