OpenViSTAI · Agents · Blockchain · Crypto
Submit a project
Menu
Image & Video AI

Joyhallo

Joyhallo is an MIT-licensed Python model project that generates talking-face videos from speech audio and a reference face image, with a focus on Mandarin and stated support for English and mixed-language audio.

Explore the practical guide ↓

Independent OpenViST project profile, based on linked sources.

Joyhallo — OpenViST project profile in Image & Video AI.
OpenViST project illustration. See the linked sources for product details.
Understand the project

Practical overview

When it fits

Use Joyhallo when you have a suitable face image and speech audio and want to generate a talking-face video through a local inference workflow.

Who it is for

  • Developers building or evaluating Mandarin digital-human video prototypes.
  • Researchers and practitioners who want to run audio-driven talking-face inference locally.
  • Teams exploring fine-tuning a talking-face model using prepared training data.

Documented uses

  • Generate a talking-face video from a reference portrait and WAV speech audio.
  • Run inference through a command-line script or a locally started web demo.
  • Train or fine-tune the model using the repository's staged training scripts.
Before you choose

Requirements & limits

The project repository is public and identifies the software license as MIT. The documented workflow is local and requires model checkpoints and a Python environment.

  • The image must be square and front-facing, with the face taking up 50–70% of the image.
  • Audio must be WAV, clear, and in Mandarin, English, or a mixture.
  • The README's tested environment is Ubuntu 20.04, CUDA 11.3, and an A100 GPU; this is not a compatibility guarantee.
  • The README provides a speed adjustment through inference steps and warns that fewer steps may reduce output quality.

Check current requirements in the official documentation. Source-based guidance does not mean OpenViST has run or independently tested this project.

Source-based practical guide

Run local audio-driven inference

Generate a talking-face video from a reference image and speech audio using the repository's command-line inference workflow.

Before you start

  • Python 3.10, the repository requirements, and FFmpeg.
  • The README lists Ubuntu 20.04, CUDA 11.3, and an A100 GPU as its tested setup.
  • Base, Chinese wav2vec2, and Joyhallo checkpoints.
  • A square, front-facing image with the face occupying 50–70% of the frame and clear WAV audio in Mandarin, English, or mixed speech.

Steps

  1. Create and activate a Python 3.10 Conda environment, install the repository requirements, and install FFmpeg.

  2. Download the base, Chinese wav2vec2, and Joyhallo checkpoints into the documented pretrained-model locations.

  3. Set the checkpoint, reference-image, audio, and output paths in configs/inference/inference.yaml.

  4. Run sh joyhallo-infer.sh.

Expected outcome

The README says inference results are saved in opts/joyhallo.

Guide limitations

  • The image and audio input requirements apply to inference as well as training.
  • Reducing inference steps can trade output quality for speed; the README cautions that going below 15 may worsen results.
Documentation 1 ↗
Evidence and transparency

Sources & review

Official repository ↗
Source record details
Source availability
Active
Sources checked
17 October 2026
Original website title
jdh-algo/JoyHallo · Official homepage title · retained in its source language
How we review sourcesSuggest a correction