OpenViSTAI · Agents · Blockchain · Crypto
Submit a project
Menu
AI Models

Macaw LLM

Macaw-LLM is an open-source Python research project that integrates image, video, audio and text inputs into a multimodal language-model workflow. Its stated components are CLIP for images and video frames, Whisper for audio, and a language model based on LLaMA, Vicuna or Bloom for instructions and responses. The repository documents dataset preprocessing, training and inference stages.

Explore the practical guide ↓

Independent OpenViST project profile, based on linked sources.

Macaw LLM — OpenViST project profile in AI Models.
OpenViST project illustration. See the linked sources for product details.
Understand the project

Practical overview

When it fits

Use the repository when you want to examine or adapt a research workflow that combines image, video, audio and text data, particularly for multimodal instruction tasks.

Who it is for

  • Researchers studying multimodal language models and instruction following.
  • Developers preparing multimodal datasets or exploring model training and inference workflows.

Documented uses

  • Prepare text, image and video data for multimodal model workflows.
  • Explore alignment of visual and audio features with language-model embeddings.
  • Train or run inference with a multimodal instruction model using the documented scripts.
Before you choose

Requirements & limits

The project is published in a public GitHub repository. Repository metadata lists Apache-2.0; the README also displays a CC BY 4.0 badge, so check the applicable terms for each component and associated materials.

  • The documented design names CLIP, Whisper and LLaMA, Vicuna or Bloom as components.
  • The README lists Python 3.8, requirements installation, ffmpeg and NVIDIA Apex in its setup instructions.
  • The usage workflow separates dataset preparation, training and inference, and requires choosing and organizing datasets.
  • The README describes its generated instruction data as single-turn dialogue examples.

Check current requirements in the official documentation. Source-based guidance does not mean OpenViST has run or independently tested this project.

Source-based practical guide

Prepare datasets for Macaw-LLM

Organize and preprocess text, image and video data for the project’s training or inference workflow.

Before you start

  • A local copy of the Macaw-LLM repository and its listed Python packages.
  • Text, image or video datasets; the README names Alpaca text data, COCO and VQA images, Charades and Video Dialog video data, and project image/video instruction datasets.
  • ffmpeg and NVIDIA Apex, both listed in the installation instructions.

Steps

  1. Install the listed Python packages with the command below, then follow the README’s ffmpeg and Apex installation instructions.

    Documented command
    pip install -r requirements.txt

  2. Place data in the documented data/text/, data/image/ and data/video/ folders.

  3. Run the command below to extract video frames and audio.

    Documented command
    python preprocess_data.py

  4. Run the command below or the command below to transform the corresponding data into datasets.

    Documented command
    python preprocess_data_supervised.py
    Documented command
    python preprocess_data_unsupervised.py

Expected outcome

The README’s preprocessing workflow organizes source data and transforms supervised or unsupervised examples into datasets for the later model stages.

Guide limitations

  • The README does not specify hardware requirements or a complete dataset-download procedure.
  • This guide covers preparation only; the repository documents training and inference as later stages without establishing that they will run successfully in a given environment.
Documentation 1 ↗
Evidence and transparency

Sources & review

Official repository ↗
Source record details
Source availability
Active
Sources checked
15 October 2026
Original website title
lyuchenyang/Macaw-LLM · Official homepage title · retained in its source language
How we review sourcesSuggest a correction