OpenViSTAI · Agents · Blockchain · Crypto
Submit a project
Menu
AI Models

Soundnet

SoundNet is Lua research code with pretrained neural networks and scripts for learning and using acoustic representations. Its method transfers visual-model knowledge to sound through naturally synchronized audio-video data. The repository supports audio category prediction, feature extraction for downstream work, fine-tuning on labeled categories, and training.

Explore the practical guide ↓

Independent OpenViST project profile, based on linked sources.

Soundnet — OpenViST project profile in AI Models.
OpenViST project illustration. See the linked sources for product details.
Understand the project

Practical overview

When it fits

Use SoundNet when a research workflow needs pretrained audio representations, category predictions, or a starting point for adapting a sound model. Its documented scripts operate on audio file lists; the training method uses the natural synchronization of audio and video to transfer visual-model supervision.

Who it is for

  • Audio researchers who want pretrained representations for downstream classification experiments.
  • Machine-learning practitioners studying sound recognition or student-teacher transfer from visual models to audio.
  • Researchers adapting the provided code to labeled audio datasets.

Documented uses

  • Produce object and scene category scores for audio files.
  • Extract time-varying SoundNet features for downstream machine-learning tasks.
  • Fine-tune a pretrained model using audio files paired with integer category labels.
  • Train the five- or eight-layer model using audio and visual-model class probabilities.
Before you choose

Requirements & limits

The code is in a public repository marked with an MIT license. The README lists downloadable five- and eight-layer pretrained models.

  • The implementation uses Torch7 and Lua rather than a hosted analysis interface.
  • Feature extraction requires Torch7 HDF5 in addition to Torch7 audio and SoX.
  • The README recommends the eight-layer pretrained model.
  • The supplied fine-tuning script may require changes for a particular dataset.
  • Training from scratch depends on audio plus class probabilities from visual models; the README describes the corresponding two-million-video data release as planned.

Check current requirements in the official documentation. Source-based guidance does not mean OpenViST has run or independently tested this project.

Source-based practical guide

Extract sound features from an audio list

Generate SoundNet feature files for audio inputs using the provided extraction script.

Before you start

  • Torch7, Torch7 audio, and SoX.
  • Torch7 HDF5 for feature extraction.
  • An audio file for each entry in a text list; the README uses MP3 and says most audio formats should be supported.
  • A pretrained SoundNet model.

Steps

  1. Create a text file with one audio file per line.

  2. Run list=data.txt th extract_feat.lua.

  3. To select another layer, set a layer value, as in list=data.txt layer=24 th extract_feat.lua.

  4. Use the resulting time-by-dimension features; average-pool or max-pool over time if a fixed-length vector is needed.

Expected outcome

Feature data is written to HDF5 files alongside the input files. The default extraction layer is conv7.

Guide limitations

  • The README says a GPU is probably needed.
  • The documented feature output varies over time; fixed-length vectors require pooling.
  • The README does not provide a general compatibility guarantee for audio formats.
Documentation 1 ↗
Source-based practical guide

Generate object and scene category predictions

Use the provided script to produce SoundNet category scores for a list of audio files.

Before you start

  • Torch7, Torch7 audio, and SoX.
  • A text file with one audio file per line.
  • A pretrained SoundNet model.

Steps

  1. Create a text file such as data.txt, listing the audio files to process.

  2. Run list=data.txt th extract_predictions.lua.

  3. Use categories/categories_places2.txt for scene labels and categories/categories_imagenet.txt for object labels.

Expected outcome

The script writes HDF5 files containing category scores alongside the input files and outputs the top-scoring object and scene category.

Guide limitations

  • The README says MP3 is used and most audio formats should be supported, but does not guarantee support for every format.
  • The documented requirements say a GPU is probably needed.
Documentation 1 ↗
Evidence and transparency

Sources & review

Official repository ↗
Source record details
Source availability
Active
Sources checked
17 October 2026
Original website title
cvondrick/soundnet · Official homepage title · retained in its source language
How we review sourcesSuggest a correction