OpenViSTAI · Agents · Blockchain · Crypto
Submit a project
Menu
AI Models

Stream Omni

Stream Omni is an open-source Python language-vision-speech model for chatbot interactions using combinations of text, images and speech. The repository identifies an 8B checkpoint and describes text and speech responses, including intermediate ASR transcription and response text during speech interaction.

Explore the practical guide ↓

Independent OpenViST project profile, based on linked sources.

Stream Omni — OpenViST project profile in AI Models.
OpenViST project illustration. See the linked sources for product details.
Understand the project

Practical overview

When it fits

Use Stream Omni when you want to prototype or examine multimodal chatbot interactions locally, especially vision-grounded speech questions with intermediate text output.

Who it is for

  • Developers exploring multimodal chatbot prototypes.
  • Researchers or practitioners testing vision-grounded speech question-answering.
  • Python users able to set up model and speech-generation dependencies locally.

Documented uses

  • Ask a spoken question about an image and receive text and speech responses.
  • Inspect intermediate speech transcription and response text during an audio interaction.
  • Explore chatbot workflows that combine text, vision and speech modalities.
Before you choose

Requirements & limits

The repository is public and identifies its license as GPL-3.0. Running the documented examples requires local model and supporting component downloads and a configured Python environment.

  • The repository documents an 8B model checkpoint and a separate CosyVoice model dependency for speech output.
  • The stated setup uses Python 3.10, FlashAttention, repository and CosyVoice requirements, and CUDA configuration.
  • An 8-bit loading option is mentioned for VRAM below 32 GB; the repository does not frame this as a hardware compatibility guarantee.
  • The repository links the InstructOmni dataset, but the described example workflow uses local image and audio files.

Check current requirements in the official documentation. Source-based guidance does not mean OpenViST has run or independently tested this project.

Source-based practical guide

Run the documented image-and-speech example

Use the provided command-line example to submit an image and spoken question and inspect transcription, text, and speech outputs.

Before you start

  • Python 3.10 environment with the repository packages, FlashAttention, and CosyVoice requirements installed.
  • Downloaded Stream-Omni model checkpoint and CosyVoice-300M-25Hz files.
  • CUDA-capable setup configured as shown in the repository.

Steps

  1. Set CUDA_VISIBLE_DEVICES and PYTHONPATH to the values shown in the repository instructions.

  2. Set STREAMOMNI_CKPT to the downloaded model path and point the CosyVoice model reference in the script to its local checkpoint.

  3. Run stream_omni/eval/run_stream_omni.py with the documented cat image, cat_color.wav query, and model arguments.

  4. Review the printed ASR and model text and the saved speech output.

Expected outcome

The example shows a spoken question about an image, an ASR transcription, a text answer, and generated speech output.

Guide limitations

  • The repository notes that --load-8bit can be added for VRAM below 32 GB; this does not guarantee operation on a particular system.
  • The documented example uses the repository’s bundled cat image and audio query.
Documentation 1 ↗
Evidence and transparency

Sources & review

Official repository ↗
Source record details
Source availability
Active
Sources checked
14 October 2026
Original website title
ictnlp/Stream-Omni · Official homepage title · retained in its source language
How we review sourcesSuggest a correction