Fiftyone Multimodal Rag Plugin: Query image and text indexes

Understand the project

Practical overview

When it fits

Use it when prototyping a multimodal RAG workflow and examining how image indexing choices, retrieved context and model selection affect results.

Who it is for

Developers exploring multimodal retrieval-augmented generation
FiftyOne users who want to experiment with indexing and querying image-and-text datasets
Practitioners comparing image representation and retrieval choices in RAG workflows

Documented uses

Create a FiftyOne dataset from directories containing images, text files and PDFs
Add further directories of documents to an existing dataset
Create a multimodal index using CLIP image embeddings or captions
Inspect an index and query it with controls for image and text retrieval counts
Experiment with multimodal models for generating retrieval-augmented results

Before you choose

Requirements & limits

The workflow is built around FiftyOne, LlamaIndex and Milvus.
Image indexing can use CLIP embeddings or a selected caption field.
The query operator lets users set image and text retrieval counts and choose from the documented multimodal models.
The repository labels the project a work in progress and notes that some features may not work as expected.

Check current requirements in the official documentation. Source-based guidance does not mean OpenViST has run or independently tested this project.

Source-based practical guide

Create and query a multimodal RAG index

Build a FiftyOne dataset from local media and documents, index it, and query the resulting multimodal RAG index.

Before you start

FiftyOne installed
The plugin downloaded from its GitHub repository
Plugin requirements installed with the FiftyOne CLI
A directory containing images, text files, PDFs or other supported multimodal data

Steps

Install FiftyOne with the command below.
Documented command
```
pip install fiftyone
```
Download the plugin with fiftyone plugins download https://github.com/jacobmarks/fiftyone-multimodal-rag-plugin and install its requirements with fiftyone plugins requirements @jacobmarks/multimodal_rag --install.
Launch the FiftyOne App, open the operator menu with the backtick key, and run create_dataset_from_llama_documents on a directory of data.
Run create_multimodal_rag_index, name the index, and choose CLIP embeddings or a caption field for image indexing.
Run query_multimodal_rag_index, enter a query and index, then choose a supported multimodal model and image and text retrieval counts.

Expected outcome

A FiftyOne dataset and a queryable multimodal index whose retrieval-augmented results can include image and text context.

Guide limitations

The project is a work in progress and says some features may not work as expected.
If indexing images by captions, the dataset needs a text field to select as the caption.
LlamaIndex and associated package versions are pinned because its API changes frequently.

Documentation 1 ↗

COMPARE OPTIONS

Compare options for Retrieval-augmented applications

These projects document a common task and share a project role. This is a source-based comparison, not an execution benchmark or a guarantee of interchangeability.

Project	Documented purpose	Platform
Fiftyone Multimodal Rag Plugin	Create a FiftyOne dataset from directories containing images, text files and PDFs	Cloud
Rag Powered AI Assistant Transforming Data Retrieval And Analysis Across The Web And Pdfs	Retrieve relevant passages from web and PDF content to answer a question.	Web
LangChain in Supabase Edge Functions	Initialize and start a Supabase project locally.	Cloud, Server

Evidence and transparency

Sources & review

Official repository ↗

Source record details

Source availability: Active
Sources checked: 18 October 2026
Original website title: jacobmarks/fiftyone-multimodal-rag-plugin · Official homepage title · retained in its source language

How we review sources Suggest a correction

Fiftyone Multimodal Rag Plugin