OpenViSTAI · Agents · Blockchain · Crypto
Submit a project
Menu
AI Models

Flamingo Pytorch

Flamingo Pytorch is a Python and PyTorch library implementing Flamingo-style few-shot visual question-answering components, including a Perceiver resampler and gated cross-attention blocks that connect media embeddings to language-model text.

Explore the practical guide ↓

Independent OpenViST project profile, based on linked sources.

Flamingo Pytorch — OpenViST project profile in AI Models.
OpenViST project illustration. See the linked sources for product details.
Understand the project

Practical overview

When it fits

Use it when building or studying a PyTorch vision-language model and you need the documented resampling and cross-attention components to connect media embeddings with text processing.

Who it is for

  • Researchers prototyping few-shot visual question-answering systems.
  • Developers experimenting with Perceiver resampling and multimodal attention in PyTorch models.

Documented uses

  • Compress a sequence of media embeddings into a smaller set of latent representations.
  • Let text representations attend to media representations using gated cross-attention.
  • Explore a documented Flamingo-style integration with a ViT image encoder and PaLM-like model.
Before you choose

Requirements & limits

The repository is public and lists an MIT license. The README documents installation through pip.

  • The library supplies attention components rather than a ready-made end-user visual question-answering application.
  • The README’s full integration example uses a ViT image encoder and a PaLM-style model.
  • The documented component examples require media and text representations with compatible dimensions.

Check current requirements in the official documentation. Source-based guidance does not mean OpenViST has run or independently tested this project.

Source-based practical guide

Resample media embeddings and apply cross-attention

Use the documented components to compress media embeddings and let text representations attend to them.

Before you start

  • Python and PyTorch
  • Media embeddings shaped as batch, time, sequence length, and dimension
  • Text representations with matching model dimensions

Steps

  1. Install the package with the command below.

    Documented command
    pip install flamingo-pytorch

  2. Create a PerceiverResampler with dimensions, depth, attention heads, latent count, and time-embedding settings suited to the input.

  3. Pass media embeddings to the resampler to produce a shorter latent sequence.

  4. Create a GatedCrossAttentionBlock and pass it the text representations, resampled media, and a boolean media_locations tensor.

Expected outcome

The text representations attend to the resampled media through the gated cross-attention block.

Guide limitations

  • The example demonstrates components with tensors; it does not provide a complete trained visual question-answering application.
  • The README recommends deriving media locations from a dedicated media token in the text sequence.
Documentation 1 ↗
Source-based practical guide

Set up the documented FlamingoPaLM example

Prepare the example model components for a Flamingo-style PaLM integration.

Before you start

  • PyTorch
  • flamingo-pytorch
  • vit-pytorch for the example vision encoder
  • An image encoder that returns embeddings and a language-model configuration

Steps

  1. Install vit-pytorch with the command below.

    Documented command
    pip install vit-pytorch

  2. Create a ViT and wrap it with the documented Extractor to return embeddings.

  3. Configure FlamingoPaLM with model dimensions, the image encoder, a media token ID, cross-attention interval, and Perceiver settings.

Expected outcome

The documented example has a ViT embedding extractor and a configured FlamingoPaLM model ready for the subsequent training and image-input stages.

Guide limitations

  • The README describes later training and image-passing stages, but this guide covers setup only.
  • The example uses a PaLM-style model configuration and does not establish compatibility with other language models.
Documentation 1 ↗
Evidence and transparency

Sources & review

Official repository ↗
Source record details
Source availability
Active
Sources checked
15 October 2026
Original website title
lucidrains/flamingo-pytorch · Official homepage title · retained in its source language
How we review sourcesSuggest a correction