Practical overview
When it fits
Use Joyvasa when you can prepare the documented model environment and want to generate audio-driven portrait or animal animation from a reference image.
Who it is for
- Developers seeking a self-hosted project for generating talking portrait or animal animations from images and audio.
- Researchers or practitioners who want to inspect or adapt a diffusion-based facial-motion generation pipeline.
Documented uses
- Create a talking-head animation from a portrait image and speech audio.
- Generate facial motion and lip-synchronized movement driven by audio.
- Animate an animal-face image using the animal animation mode.
- Train a motion generator using human talking-face video data and the repository's documented data-preparation scripts.
Requirements & limits
The project is available as a public GitHub repository under the MIT license. The documented web demo is launched locally.
- The README lists specific tested GPU and CUDA configurations rather than a general hardware guarantee.
- The inference workflow requires multiple model checkpoints, Git LFS and FFmpeg.
- Animal image animation has an optional MultiScaleDeformableAttention dependency.
- Animation mode should match the subject type in the reference image.
- The documented wav2vec2-base motion-generation option is not yet supported; the README also lists Chinese HuBERT.
Check current requirements in the official documentation. Source-based guidance does not mean OpenViST has run or independently tested this project.
Run inference from the command line
Generate a human or animal animation from a reference image and audio using the documented inference script.
Before you start
- Python 3.10 environment with the repository requirements installed
- FFmpeg
- JoyVASA, audio-encoder and LivePortrait checkpoints in pretrained_weights
- Git LFS to obtain the model checkpoints
- A CUDA-capable system; the README lists Ubuntu 20.04/CUDA 12.1/A100 and Windows 11/CUDA 12.1/RTX 4060 Laptop 8 GB as tested configurations
Steps
Prepare the environment and install the required packages and FFmpeg.
Download the documented JoyVASA motion-generator, audio-encoder and LivePortrait checkpoints into pretrained_weights.
For an animal example, run the command below.
python inference.py -r assets/examples/imgs/joyvasa_001.png -a assets/examples/audios/joyvasa_001.wav --animation_mode animal --cfg_scale 2.0For a human example, run the command below.
python inference.py -r assets/examples/imgs/joyvasa_003.png -a assets/examples/audios/joyvasa_003.wav --animation_mode human --cfg_scale 2.0
Expected outcome
The inference script is invoked with a reference image, speech audio and selected animation mode to generate an output video.
Guide limitations
- The README cautions that mismatching the animation mode and reference image may produce incorrect results.
- Although wav2vec2-base is listed as an encoder option, the README says motion-generation support for it will come later.
Launch the local web demo
Start JoyVASA's documented web demo after preparing its environment and model checkpoints.
Before you start
- The installed JoyVASA environment and required model checkpoints
- Python and the repository's requirements
Steps
Complete the documented environment installation and checkpoint preparation.
Run the command below from the repository.
python app.pyOpen the local demo at
http://127.0.0.1:7862.
Expected outcome
The README identifies a local web demo at http://127.0.0.1:7862.
Guide limitations
- The documentation describes a local demo; it does not establish a hosted public service.
Sources & review
Source record details
- Source availability
- Active
- Sources checked
- 16 October 2026
- Original website title
- jdh-algo/JoyVASA · Official homepage title · retained in its source language