Practical overview
When it fits
Use Tempotokens when you want a research implementation and are prepared to configure a local model workflow, rather than a hosted interface for editing or producing video.
Who it is for
- Researchers exploring audio-to-video generation and audio-visual alignment.
- Developers who can work with Python, PyTorch, model checkpoints, and command-line training or inference.
Documented uses
- Generate video conditioned on a natural audio sample.
- Train the adaptor using the documented VGGSound, Landscape, or AudioSet-Drum configurations.
- Prototype generation conditioned on text and audio.
- Explore audio-video alignment using the paper's AV-Align evaluation approach.
Requirements & limits
The repository is public and licensed under MIT. The documented workflow uses local command-line setup and model files.
- Inference needs trained mapper weights; the README points to pre-trained weights for VGGSound, Landscape, and AudioSet-Drum.
- The README identifies a pre-trained BEATs audio model as a setup requirement.
- Training instructions are provided separately for three datasets, and the project recommends reviewing and customizing their configuration files.
- The method adapts an existing text-conditioned video model rather than describing a standalone hosted generation service.
Check current requirements in the official documentation. Source-based guidance does not mean OpenViST has run or independently tested this project.
Set up the TempoTokens environment
Install the repository dependencies, configure Accelerate, and place the required BEATs model file in the documented location.
Before you start
- A Linux environment with Python and pip.
- Git, wget, and the Accelerate package available to the environment.
Steps
Clone the TempoTokens repository and change into its directory.
Install dependencies with the command below.
pip install -r requirements.txtInitialize Accelerate with
accelerate config.Create
models/BEATs/and download the documented pre-trained BEATs checkpoint to that directory.
Expected outcome
The repository dependencies and Accelerate configuration are in place, with the BEATs checkpoint stored at the path specified in the README.
Guide limitations
- This is a setup procedure; it does not train a model or generate a video.
- Inference additionally requires trained mapper weights or the project's pre-trained weights.
Generate a video from an audio file
Use an existing VGGSound mapper checkpoint to invoke inference with a user-supplied audio path.
Before you start
- A TempoTokens environment with its dependencies and BEATs checkpoint in place.
- The VGGSound mapper weights at
models/vggsound/learned_embeds.pth, or a corresponding trained checkpoint. - An audio file accessible by its local path.
Steps
Obtain the project's pre-trained weights and save the relevant folder under
models/, as recommended in the README.Run
accelerate launch inference.py --mapper_weights models/vggsound/learned_embeds.pth --audio_path /audio/path, replacing/audio/pathwith the audio file's path.
Expected outcome
The inference script is invoked to generate video conditioned on the supplied audio.
Guide limitations
- The README does not specify the output location or guarantee a particular result for a given audio file.
- The example uses VGGSound mapper weights; the documented setup and model assets are prerequisites.
Sources & review
Source record details
- Source availability
- Active
- Sources checked
- 16 October 2026
- Original website title
- guyyariv/TempoTokens · Official homepage title · retained in its source language