OpenViSTAI · Agents · Blockchain · Crypto
Submit a project
Menu
Image & Video AI

Video Llama

Video Llama is an open-source Python project for an instruction-tuned audio-visual language model that supports video understanding and video-grounded conversation. It combines vision-language and audio-language branches with a language decoder, and provides research code, checkpoints and a demo.

Explore the practical guide ↓

Independent OpenViST project profile, based on linked sources.

Video Llama — OpenViST project profile in Image & Video AI.
OpenViST project illustration. See the linked sources for product details.
Understand the project

Practical overview

When it fits

Use Video Llama when exploring a research implementation for video understanding or building an experimental interface that asks questions about video and its audio.

Who it is for

  • Researchers studying video and audio-visual language models
  • Developers exploring video-question-answering interfaces
  • Teams evaluating model responses to video with audio and visual information

Documented uses

  • Ask questions about video using a language-model interface.
  • Explore video-grounded conversation with audio-visual inputs.
  • Study or adapt a model architecture that combines video frames and audio representations.
  • Launch the repository’s local demo after configuring its environment and checkpoint paths.
Before you choose

Requirements & limits

The repository is public and identifies its license as BSD-3-Clause. It links to hosted demos and checkpoints on Hugging Face and ModelScope, as well as instructions for running a demo locally.

  • The architecture has distinct vision-language and audio-language branches; the local demo configuration requires paths for both branches and the language decoder.
  • The repository’s current README describes Video-LLaMA 2, while earlier Video-LLaMA instructions are maintained separately.
  • The project is Python-based and the documented local environment uses Conda and ffmpeg.
  • The repository links to both hosted demos and checkpoints, offering routes to explore the project beyond its local demo instructions.

Check current requirements in the official documentation. Source-based guidance does not mean OpenViST has run or independently tested this project.

Source-based practical guide

Set up and launch the local audio-video demo

Prepare the documented local environment, configure model checkpoint paths and launch the demo script.

Before you start

  • The Video-LLaMA repository
  • ffmpeg
  • Conda
  • Paths to the language decoder, audio encoder, VL branch and AL branch checkpoints
  • A GPU ID for the launch command

Steps

  1. Install ffmpeg.

  2. Create and activate the Conda environment from environment.yml.

  3. Set the llama_model, imagebind_ckpt_path, ckpt and ckpt_2 paths in eval_configs/video_llama_eval_withaudio.yaml.

  4. Run demo_audiovideo.py with the evaluation configuration, model type and GPU ID.

Expected outcome

The repository’s local audio-video demo is launched with the selected model type and configured checkpoint paths.

Guide limitations

  • The instructions document setup and launch but do not specify hardware requirements beyond a GPU ID.
  • The README distinguishes Video-LLaMA 2 from the earlier Vicuna-based version; use the instructions for the intended version.
Documentation 1 ↗
Evidence and transparency

Sources & review

Official repository ↗
Source record details
Source availability
Active
Sources checked
3 November 2026
Original website title
DAMO-NLP-SG/Video-LLaMA · Official homepage title · retained in its source language
How we review sourcesSuggest a correction