OpenViSTAI · Agents · Blockchain · Crypto
Submit a project
Menu
Image & Video AI

NÜWA – Pytorch

NÜWA - Pytorch is a Python implementation of the NÜWA attention network for text-to-video synthesis. It includes VQGanVAE and NUWA components, an example workflow for training on text and video data and generating frames, and a NUWASketch variant that uses a separately trained sketch VAE for conditioning. The README also describes a dual-decoder extension for video and audio generation.

Explore the practical guide ↓

Independent OpenViST project profile, based on linked sources.

NÜWA - Pytorch — OpenViST project profile in Image & Video AI.
OpenViST project illustration. See the linked sources for product details.
Understand the project

Practical overview

When it fits

Use it when you want to inspect or adapt a code-level PyTorch implementation and are prepared to train its models with your own data, rather than use a hosted video-generation interface.

Who it is for

  • Researchers exploring attention-based video generation.
  • Developers learning or adapting PyTorch generative-model implementations.
  • Teams investigating text- or sketch-conditioned video synthesis.

Documented uses

  • Train a VQGanVAE on images as preparation for video modeling.
  • Train a NUWA model with text and video tensors.
  • Generate video frames conditioned on text using the documented generation method.
  • Explore sketch-conditioned video experiments with a separately trained sketch VAE.
Before you choose

Requirements & limits

The repository is public, lists an MIT license, and documents installation with pip. The usage example uses CUDA tensors.

  • The README requires training the VQGanVAE before the text-conditioned NUWA stage.
  • The text-to-video example accepts tokenized text and video tensors; it does not document a prompt-to-video front end.
  • Sketch conditioning uses NUWASketch and a separate sketch VAE trained beforehand.
  • The dual-decoder video-and-audio extension is described, but the provided usage examples focus on video.
  • Repository status notes from March and April 2022 characterize the work as exploratory and discuss difficulty with Moving MNIST and changes in the research landscape.

Check current requirements in the official documentation. Source-based guidance does not mean OpenViST has run or independently tested this project.

Source-based practical guide

Train a VAE and prepare text-conditioned video modeling

Follow the README’s documented setup and model-preparation stages for text-to-video experiments.

Before you start

  • Python environment with PyTorch installed.
  • The nuwa-pytorch package, installed with the documented pip command.
  • Image data for VQGanVAE training and text/video tensors for the NUWA example.
  • CUDA is used in the README’s NUWA example.

Steps

  1. Install the package with the command below.

    Documented command
    pip install nuwa-pytorch

  2. Train a VQGanVAE on images, alternating the autoencoder and discriminator losses as shown in the README.

  3. Create a NUWA model using the learned VAE and configure text token, frame, image, encoder, and decoder parameters.

  4. Pass text and video tensors to NUWA with return_loss=True and backpropagate the returned loss.

Expected outcome

A trained VAE and a documented training path for a text-conditioned NUWA video model; the README also shows `nuwa.generate(text=text, num_frames=5)` as the subsequent generation call.

Guide limitations

  • The README says to train the VAE first and train NUWA with data; it does not provide pretrained weights or a ready-made dataset.
  • The example uses CUDA tensors and does not document hardware sizing or a complete environment specification.
  • The repository’s status notes describe difficulty with a Moving MNIST version.
Documentation 1 ↗
Evidence and transparency

Sources & review

Official repository ↗
Source record details
Source availability
Active
Sources checked
15 October 2026
Original website title
lucidrains/nuwa-pytorch · Official homepage title · retained in its source language
How we review sourcesSuggest a correction