Practical overview
When it fits
Use it for research experiments that connect sounds in audio to objects visible in video, especially when you need a documented preparation, training and evaluation workflow for categorized clips.
Who it is for
- Researchers studying audio-visual source separation and cross-modal learning.
- Developers assembling experimental pipelines for training and evaluating sound-separation models.
Documented uses
- Prepare paired video-frame and audio-waveform data and generate training and validation indexes.
- Train the default model using the MUSIC dataset workflow.
- Evaluate a trained model and review training visualizations.
Requirements & limits
The repository is public and identifies an MIT license. Its documented setup uses Ubuntu, CUDA, Python, PyTorch and one to four GPUs.
- Input preparation is part of the workflow: frames are extracted at 8 fps and waveforms at 11,025 Hz.
- The index-generation script expects paired audio and frame paths plus a frame count in each row.
- The README says alternative preprocessing is acceptable when the index files follow a similar structure.
- The project is research code and the README describes it as under construction, with core parts present.
Check current requirements in the official documentation. Source-based guidance does not mean OpenViST has run or independently tested this project.
Prepare data indexes for MUSIC model training
Create the training and validation CSV indexes that connect audio files with extracted video frames.
Before you start
- Ubuntu 16.04.3 LTS is the documented environment.
- CUDA 8.0 or newer, Python 3.5 or newer, PyTorch 0.4.0 or newer, and one to four GPUs are listed in the repository configuration.
- A prepared video dataset; the README names the MUSIC dataset and downloaded videos.
- Paired audio waveforms and video frames organized by category.
Steps
Prepare the video dataset and videos.
Extract frames at 8 fps and audio waveforms at 11,025 Hz, arranging them in the documented audio and frames directory structure.
Run the command below to create
train.csvandval.csventries containing audio path, frames path and frame count.python scripts/create_index_files.py
Expected outcome
Training and validation index files in the format expected by the repository's workflow.
Guide limitations
- The README allows other preprocessing methods only if their index files have a similar structure.
- This guide covers data preparation and indexing, not training or evaluation.
Train and evaluate the default MUSIC model
Run the repository's documented training and evaluation stages after preparing the data indexes.
Before you start
- Prepared dataset files and training/validation indexes from the repository workflow.
- The documented environment: Ubuntu 16.04.3 LTS, CUDA 8.0 or newer, Python 3.5 or newer, PyTorch 0.4.0 or newer, and one to four GPUs.
Steps
Run
./scripts/train_MUSIC.shto train the default model.Review training visualizations saved in HTML format under
ckpt/MODEL_ID/visualization/.Optionally run
./scripts/download_trained_model.shto download the project's trained weights for evaluation.Run
./scripts/eval_MUSIC.shto evaluate model performance.
Expected outcome
A trained default MUSIC model, training visualizations and an evaluation run using the repository's scripts.
Guide limitations
- The optional weight-download step is documented for evaluation; it does not replace the separate training workflow.
- The repository README describes the project as under construction, with core parts available.
Sources & review
Source record details
- Source availability
- Active
- Sources checked
- 17 October 2026
- Original website title
- hangzhaomit/Sound-of-Pixels · Official homepage title · retained in its source language