OpenViSTAI · Agents · Blockchain · Crypto
Submit a project
Menu
AI Models

Zorro

Zorro is an open-source Python library implementing the Zorro masked multimodal transformer in PyTorch. It accepts audio and video tensors and returns configurable audio, video, fusion, or global tokens for downstream learning workflows, including contrastive learning.

Explore the practical guide ↓

Independent OpenViST project profile, based on linked sources.

Zorro — OpenViST project profile in AI Models.
OpenViST project illustration. See the linked sources for product details.
Understand the project

Practical overview

When it fits

Use Zorro when a Python/PyTorch workflow needs configurable audio-video transformer representations and the surrounding data preparation and learning pipeline will be handled separately.

Who it is for

  • Researchers developing multimodal learning workflows with audio and video inputs.
  • Python and PyTorch developers who need selectable modality-specific or shared transformer representations.

Documented uses

  • Produce audio and video representations from paired tensor inputs.
  • Select audio and video tokens for a contrastive-learning pipeline.
  • Use fusion or global tokens in self-supervised or supervised downstream learning.
Before you choose

Requirements & limits

The public GitHub repository declares an MIT license and provides a pip installation command.

  • This is a developer library with a Python API, not a hosted media-processing interface.
  • The documented inputs are audio and video tensors; the README does not show media-file loading or preprocessing.
  • Returned token types and model settings are configured at model construction, and token indices can select outputs at call time.
  • The README shows a contrastive-learning use of selected audio and video tokens but leaves the training objective and data pipeline to the user.

Check current requirements in the official documentation. Source-based guidance does not mean OpenViST has run or independently tested this project.

Source-based practical guide

Install the package and configure a model

Install Zorro and set up a PyTorch model to accept audio and video tensors and return selected token types.

Before you start

  • Python environment with PyTorch available; the README imports torch.
  • Use pip to install the zorro-pytorch package.

Steps

  1. Install the package with the command below.

    Documented command
    pip install zorro-pytorch

  2. Import Zorro and TokenTypes from zorro_pytorch.

  3. Instantiate Zorro with model settings such as dimensions, depth, attention heads, fusion-token count, and audio and video patch sizes; set return_token_types to the representations needed.

Expected outcome

A configured model object with an explicit choice of audio, video, fusion, or global token types to return.

Guide limitations

  • The README setup example does not document dataset preparation, training, or deployment.
  • No Python or PyTorch version requirements are specified in the cited README.
Documentation 1 ↗
Source-based practical guide

Select audio and video tokens for a contrastive-learning workflow

Use the README's example to obtain one audio and one video token for downstream contrastive learning.

Before you start

  • An installed zorro-pytorch package and PyTorch.
  • A configured Zorro model whose returned tokens include the audio and video representations to select.
  • Audio and video tensors; the README example uses video shaped (batch, channels, time, height, width) and audio shaped (batch, time).

Steps

  1. Create audio and video tensors and pass them to the model as audio and video.

  2. Pass return_token_indices to select the desired token positions.

  3. Unbind the selected output along the token dimension to obtain separate audio and video representations.

Expected outcome

Separate audio and video token tensors that can be supplied to downstream contrastive-learning code.

Guide limitations

  • The example demonstrates tensor inputs rather than loading or preprocessing media files.
  • The README does not provide the downstream contrastive-learning objective or training loop.
Documentation 1 ↗
Evidence and transparency

Sources & review

Official repository ↗
Source record details
Source availability
Active
Sources checked
17 October 2026
Original website title
lucidrains/zorro-pytorch · Official homepage title · retained in its source language
How we review sourcesSuggest a correction