OpenViSTAI · Agents · Blockchain · Crypto
Submit a project
Menu
Audio & Voice AI

Soundstorm Pytorch

Soundstorm Pytorch is a Python and PyTorch implementation of SoundStorm, a Conformer-based model for parallel audio generation from residual vector-quantized codes using MaskGIT-style iterative prediction.

Explore the practical guide ↓

Independent OpenViST project profile, based on linked sources.

Soundstorm Pytorch — OpenViST project profile in Audio & Voice AI.
OpenViST project illustration. See the linked sources for product details.
Understand the project

Practical overview

When it fits

Use it when building an audio-generation experiment around residual vector-quantized codes, or when connecting SoundStorm with SoundStream for raw-audio workflows.

Who it is for

  • Developers and researchers implementing or studying neural audio generation.
  • PyTorch users working with SoundStream audio codes or models.

Documented uses

  • Train SoundStorm on pre-encoded SoundStream codes.
  • Generate audio-code sequences with the SoundStorm model.
  • Train from raw audio and generate waveforms when a SoundStream model is supplied.
  • Explore text-to-speech by connecting a trained TextToSemantic model with SoundStream and SoundStorm.
Before you choose

Requirements & limits

The public GitHub repository lists an MIT license and documents installation with pip.

  • Pre-encoded-code training requires SoundStream code IDs as input.
  • Raw-audio training and waveform generation require passing a SoundStream model into SoundStorm.
  • Text-to-speech additionally requires trained TextToSemantic weights; the README describes that integration but notes the related project lacks pretraining and pseudo-labeling logic.
  • The README lists a command-line tool as a future task, so its documented usage is through Python.

Check current requirements in the official documentation. Source-based guidance does not mean OpenViST has run or independently tested this project.

Source-based practical guide

Generate audio from pre-encoded SoundStream codes

Configure SoundStorm and generate a batch of code sequences from a model trained on pre-encoded codes.

Before you start

  • SoundStorm-PyTorch installed.
  • Pre-encoded codebook IDs from SoundStream.

Steps

  1. Create a ConformerWrapper with codebook_size=1024, num_quantizers=12, and conformer=dict(dim=512, depth=2).

  2. Create SoundStorm with that wrapper, steps=18, and schedule='cosine'.

  3. Train on code IDs using loss, _ = model(codes) followed by loss.backward(); the README’s example uses a (2, 1024, 12) tensor.

  4. Call model.generate(1024, batch_size=2) to generate a batch of sequences.

Expected outcome

The generation call returns a tensor shaped `(2, 1024)` in the documented example.

Guide limitations

  • This path requires pre-encoded SoundStream code IDs; the README does not give an audio-decoding step for this example.
  • The example demonstrates a training step but does not document a complete training workflow or provide pretrained SoundStorm weights.
Documentation 1 ↗
Evidence and transparency

Sources & review

Official repository ↗
Source record details
Source availability
Active
Sources checked
13 October 2026
Original website title
lucidrains/soundstorm-pytorch · Official homepage title · retained in its source language
How we review sourcesSuggest a correction