OpenViSTAI · Agents · Blockchain · Crypto
Submit a project
Menu
Audio & Voice AI

Voicebox Pytorch

Voicebox Pytorch is an open-source Python implementation of the Voicebox text-to-speech model, with code examples for training and sampling through PyTorch.

Explore the practical guide ↓

Independent OpenViST project profile, based on linked sources.

Voicebox Pytorch — OpenViST project profile in Audio & Voice AI.
OpenViST project illustration. See the linked sources for product details.
Understand the project

Practical overview

When it fits

Use it when you want to inspect or build on a code-level Voicebox implementation and can provide the model assets required by the examples.

Who it is for

  • Developers exploring or adapting text-to-speech model implementations.
  • Researchers and practitioners working with PyTorch speech-generation training workflows.

Documented uses

  • Explore a Voicebox-based text-to-speech implementation.
  • Train and sample a text-conditioned speech model using a SpearTTS text-to-semantic component.
  • Set up an unconditional training and sampling path by disabling text conditioning.
Before you choose

Requirements & limits

The repository provides a pip installation example and identifies its license as MIT.

  • The text-conditioned example depends on Hubert checkpoint and k-means files, a trained SpearTTS model, and an audio encoder-decoder.
  • The README includes an unconditional path that sets condition_on_text=False.
  • The README recommends E2 TTS instead of this implementation.
  • The README does not specify hardware requirements or a serving interface.

Check current requirements in the official documentation. Source-based guidance does not mean OpenViST has run or independently tested this project.

Source-based practical guide

Train and sample with text conditioning

Set up the documented VoiceBox and TextToSemantic components to train on audio and sample conditioned on text.

Before you start

  • Install the package with pip install voicebox-pytorch.
  • Provide a HuBERT checkpoint and k-means file, plus a trained Spear-TTS model checkpoint.
  • Use PyTorch and the repository’s documented voicebox_pytorch components.

Steps

  1. Create HubertWithKmeans using the HuBERT checkpoint and k-means file paths.

  2. Create TextToSemantic with the HuBERT component and load the trained Spear-TTS model checkpoint.

  3. Create VoiceBox with audio_enc_dec=EncodecVoco(), then pass it and TextToSemantic to ConditionalFlowMatcherWrapper.

  4. Train by calling the wrapper on audio and backpropagating the returned loss; after training, call sample with conditioning audio and a list of texts.

Expected outcome

The README demonstrates a training loss and a sampling result shaped `(2, 1, )` after training.

Guide limitations

  • The example uses mock audio and conditioning tensors; it does not document data preparation or a complete training schedule.
  • Sampling is shown only after “much training”; no duration or quality guarantee is given.
  • The repository recommends using E2 TTS instead.
Documentation 1 ↗
Source-based practical guide

Train and sample without text conditioning

Configure VoiceBox for unconditional training and sampling using the documented tensor interface.

Before you start

  • Install the package with pip install voicebox-pytorch.
  • Use PyTorch and the repository’s documented VoiceBox and ConditionalFlowMatcherWrapper components.

Steps

  1. Create VoiceBox with the documented model parameters and set condition_on_text=False.

  2. Create ConditionalFlowMatcherWrapper with the model and no text-to-semantic component.

  3. Train by passing a tensor shaped (2, 1024, 512) to the wrapper and backpropagating the loss.

  4. After training, call sample with a conditioning tensor of the same documented shape.

Expected outcome

The README shows sampling output shaped `(2, 1024, 512)` after training.

Guide limitations

  • The example uses mock tensors rather than a documented real-data workflow.
  • Sampling is shown only after “much training”; the README does not specify training duration or guarantee results.
  • The repository recommends using E2 TTS instead.
Documentation 1 ↗
Evidence and transparency

Sources & review

Official repository ↗
Source record details
Source availability
Active
Sources checked
14 October 2026
Original website title
lucidrains/voicebox-pytorch · Official homepage title · retained in its source language
How we review sourcesSuggest a correction

Related options

E2 TTS — The repository README recommends using E2 TTS instead of this work.

SpearTTS — A separate project whose TextToSemantic module and trained model are used in the documented text-conditioned workflow.