OpenViSTAI · Agents · Blockchain · Crypto
Submit a project
Menu
AI Models

Tvlt

TVLT is an open-source PyTorch research model and codebase for learning compact vision-language representations from continuous video and audio signals without text-specific modules such as tokenization or automatic speech recognition. Its pretraining combines vision-audio matching with masked reconstruction of video frames and audio spectrograms.

Explore the practical guide ↓

Independent OpenViST project profile, based on linked sources.

Tvlt — OpenViST project profile in AI Models.
OpenViST project illustration. See the linked sources for product details.
Understand the project

Practical overview

When it fits

Use TVLT when a research or prototyping task calls for joint visual-audio representation learning without relying on written text or text tokens, and you can work with a local Python/PyTorch codebase.

Who it is for

  • Researchers studying textless multimodal representation learning.
  • Developers prototyping models that process video together with its audio channel.
  • Practitioners exploring visual-audio features for multimodal understanding or speech-related research.

Documented uses

  • Inspect a notebook demo of masked autoencoding on video frames and audio spectrograms.
  • Explore video-and-audio sentiment or emotional analysis demos.
  • Pretrain or fine-tune the model using the repository’s documented scripts and data guidance.
  • Use visual-acoustic representations as a basis for research tasks such as retrieval, audio-based visual question answering, speech translation or multimodal content understanding.
Before you choose

Requirements & limits

The code is available in a public GitHub repository under the MIT license. The documented workflow uses a local Python/PyTorch environment; model weights are listed on Hugging Face.

  • The model consumes audio and visual inputs and is designed to avoid text-specific modules, including tokenization and ASR.
  • Pretraining uses vision-audio matching and masked autoencoding to reconstruct continuous video-frame and audio-spectrogram inputs.
  • The repository provides demos for masked autoencoding, sentiment analysis and emotional analysis.
  • The README lists tested PyTorch, torchvision and torchaudio versions and flags CUDA/cuDNN compatibility for other PyTorch versions.
  • Pretraining and fine-tuning require following the repository’s separate data and script instructions.

Check current requirements in the official documentation. Source-based guidance does not mean OpenViST has run or independently tested this project.

Source-based practical guide

Set up the TVLT Python environment

Prepare the documented Python and dependency environment before using the repository’s demos or training scripts.

Before you start

  • The TVLT repository
  • Conda
  • PyTorch, torchvision and torchaudio compatible with the CUDA and cuDNN setup
  • The repository requirements.txt file

Steps

  1. Create a Python 3.8 environment named TVLT with Conda, or choose another Python environment as the README allows.

  2. Install PyTorch, torchvision and torchaudio; the README lists tested version combinations and advises checking compatibility with CUDA and cuDNN.

  3. Install the remaining dependencies with pip install -r requirements.txt.

Expected outcome

A configured Python environment with the repository’s listed dependencies, ready for the separately documented demos or training workflows.

Guide limitations

  • This setup guide does not cover running a demo, pretraining or fine-tuning.
  • The README’s tested versions are torch 1.10.0 or 1.12.1, torchvision 0.11.1 or 0.12.1, and torchaudio 0.10.0 or 0.13.0.
  • Other PyTorch versions require compatibility with the local CUDA and cuDNN setup.
Documentation 1 ↗
Evidence and transparency

Sources & review

Official repository ↗
Source record details
Source availability
Active
Sources checked
17 October 2026
Original website title
zinengtang/TVLT · Official homepage title · retained in its source language
How we review sourcesSuggest a correction