OpenViSTAI · Agents · Blockchain · Crypto
Submit a project
Menu
AI Models

Ultravox

Ultravox is a multimodal language model that understands speech and text, accepts audio input and emits streaming text. Its architecture connects audio directly to a language model through a multimodal projector, without a separate speech-recognition stage.

Explore the practical guide ↓

Independent OpenViST project profile, based on linked sources.

Ultravox — OpenViST project profile in AI Models.
OpenViST project illustration. See the linked sources for product details.
Understand the project

Practical overview

When it fits

Use Ultravox when a project needs a language model that takes speech directly and returns streaming text, or when adapting its adapter to a different open-weight backbone. For live-agent development, the project points to its separate Realtime platform and managed APIs.

Who it is for

  • Developers building real-time voice applications or voice-enabled assistants.
  • Researchers adapting the model to another open-weight language model or audio encoder.
  • Teams preparing audio data for a model trained on additional languages or speech examples.

Documented uses

  • Process spoken input into a language-model interaction without a separate ASR stage.
  • Build a real-time voice agent using the separate Ultravox Realtime platform or managed APIs.
  • Try audio supplied as a WAV file through the documented BaseTen inference route.
  • Train an adapter for a different open-weight language-model or audio-encoder backbone.
  • Prepare audio and text-continuation data for training.
Before you choose

Requirements & limits

The repository is public and lists an MIT license.

  • The documented model output is streaming text; speech-token output converted to audio is described as future work.
  • The repository is model code and training guidance, while the demo, Realtime platform, managed APIs and BaseTen inference route are separate ways to explore or serve it.
  • The default model is described as based on Llama 3.3 70B, with an 8B variant available on Hugging Face.
  • Training a new adapter is a substantial compute task: the README reports 2–3 hours on eight H100 GPUs for an earlier Ultravox v0.4 run.
  • For custom audio data, the stated minimum fields are audio and text continuation.

Check current requirements in the official documentation. Source-based guidance does not mean OpenViST has run or independently tested this project.

Source-based practical guide

Set up the Ultravox training environment on macOS

Install the documented tools and Python environment needed to prepare for Ultravox training.

Before you start

  • macOS with Homebrew available; Debian or Ubuntu can alternatively use apt for the basic tools.
  • Use pyenv with Poetry; the README does not recommend conda with Poetry.

Steps

  1. Install Just with Homebrew: run brew update and brew install just.

  2. Install xz and pyenv with Homebrew, then initialize pyenv and install and select Python 3.11 using pyenv init, pyenv install 3.11, and pyenv global 3.11.

  3. Create a virtual environment and install the required packages by running just install.

  4. Optionally, install system packages needed for augmentations with just install-augs-system; inspect the Poetry environment with poetry env info.

Expected outcome

The documented Python environment and required packages are installed for Ultravox development; augmentation system packages can also be installed optionally.

Guide limitations

  • This covers environment setup only, not training or inference.
  • The README gives a Mac setup path and notes Debian or Ubuntu as alternatives for basic tools, but does not provide a complete alternative setup procedure for those systems.
  • The README does not specify environment resource requirements.
Documentation 1 ↗
Evidence and transparency

Sources & review

Official repository ↗
Source record details
Source availability
Active
Sources checked
7 October 2026
Original website title
fixie-ai/ultravox · Official homepage title · retained in its source language
How we review sourcesSuggest a correction