Practical overview
When it fits
Use MMAudio when you want to generate sound conditioned on a video, a prompt, or both, and can work with its documented Python and PyTorch setup or one of the linked demonstrations.
Who it is for
- Researchers exploring multimodal audio generation
- Creators adding generated sound effects or ambience to silent video
- Developers prototyping video-to-audio or text-to-audio workflows
Documented uses
- Generate synchronized audio for a video clip
- Create audio from a text prompt without supplying video
- Condition audio generation on both visual content and a text prompt
- Explore the model through its command-line or Gradio demo
Requirements & limits
- The repository is MIT-licensed and written in Python.
- The local installation instructions specify Ubuntu, Python 3.9+, PyTorch 2.5.1+ and matching torchvision and torchaudio.
- The project reports roughly 6 GB GPU memory use for inference in 16-bit mode in its experiments.
- Video resolution increases processing time but, according to the project, does not improve output quality.
- Image-to-audio through the Gradio interface is described as experimental and not a trained use case.
Check current requirements in the official documentation. Source-based guidance does not mean OpenViST has run or independently tested this project.
Install MMAudio and generate audio from a video
Set up the documented local command-line demo and produce audio conditioned on a video and prompt.
Before you start
- Ubuntu, the only operating system the project says it has tested
- Python 3.9 or later
- PyTorch 2.5.1 or later with corresponding torchvision and torchaudio
- A CUDA version supported by the GPU and driver
- A video file and a text prompt
Steps
Install PyTorch, torchvision and torchaudio for the CUDA version supported by the GPU and driver.
Clone the MMAudio repository and install it from its directory with pip install -e .
Run the documented demo command with a video path, prompt and duration; pretrained models download automatically when a demo script runs.
Expected outcome
The demo saves generated audio as a FLAC file and video as an MP4 file in ./output.
Guide limitations
- The documented default duration is eight seconds; durations far from the training duration may reduce quality.
- The README reports about 6 GB of GPU memory for inference in 16-bit mode in its experiments; this is a reported figure, not a hardware guarantee.
- The project notes known generation failures, including unintelligible speech-like sounds and difficulty with unfamiliar concepts.
Install MMAudio for text-to-audio generation
Set up the local demo and generate audio from a text prompt without providing a video.
Before you start
- Ubuntu, the only operating system the project says it has tested
- Python 3.9 or later
- PyTorch 2.5.1 or later with corresponding torchvision and torchaudio
- A CUDA version supported by the GPU and driver
- A text prompt
Steps
Install PyTorch, torchvision and torchaudio for the CUDA version supported by the GPU and driver.
Clone the MMAudio repository and install it from its directory with pip install -e .
Run demo.py with a prompt and omit the --video option; pretrained models download automatically when the demo script runs.
Expected outcome
The demo saves generated audio in FLAC format in ./output.
Guide limitations
- The default duration is eight seconds; substantially different durations may produce lower quality.
- The project lists unintelligible speech-like sounds and background music among possible failure modes.
Sources & review
Source record details
- Source availability
- Active
- Sources checked
- 14 October 2026
- Original website title
- hkchengrex/MMAudio · Official homepage title · retained in its source language