OpenViSTAI · Agents · Blockchain · Crypto
Submit a project
Menu
AI Models

Coformer

CoFormer is a PyTorch implementation of collaborative transformers for grounded situation recognition: a vision task that predicts an image’s main activity, entities assigned to semantic roles, and bounding-box locations for those entities. Its Glance transformer predicts the activity and its Gaze transformer estimates relevant entities and their groundings. The repository includes code, setup and dataset instruction

Explore the practical guide ↓

Independent OpenViST project profile, based on linked sources.

Coformer — OpenViST project profile in AI Models.
OpenViST project illustration. See the linked sources for product details.
Understand the project

Practical overview

When it fits

Use CoFormer when an image-understanding task needs an activity prediction linked to semantic-role entities and their locations, rather than an activity label alone.

Who it is for

  • Researchers reproducing or evaluating grounded situation-recognition experiments.
  • Developers prototyping scene understanding that connects activities with role-specific entities and image locations.

Documented uses

  • Predict an image’s main activity or verb.
  • Estimate the entities associated with semantic roles in that activity.
  • Ground role-linked entities in the image using bounding boxes.
  • Train on SWiG or run the documented inference workflow on a custom image.
Before you choose

Requirements & limits

The GitHub repository is public and lists an Apache-2.0 license. Its documented stack specifies Python 3.9, PyTorch 1.8.0, torchvision 0.9.0 and CUDA Toolkit 11.1.

  • Predictions combine activity classification with role-specific entity estimation and bounding-box grounding.
  • The repository provides a linked checkpoint as well as instructions and code.
  • SWiG annotations include verbs, semantic frames and groundings; some entities may lack a grounding box.
  • The README’s training example uses four GPUs on a single node for 40 epochs.

Check current requirements in the official documentation. Source-based guidance does not mean OpenViST has run or independently tested this project.

Source-based practical guide

Set up CoFormer for custom-image inference

Prepare the documented Python and PyTorch environment and identify the inputs used by the custom-image inference command.

Before you start

  • A Linux or self-hosted Python environment; the README specifies Python 3.9.
  • Conda, PyTorch 1.8.0, torchvision 0.9.0, CUDA Toolkit 11.1 and the repository's pip requirements.
  • A custom image and a CoFormer checkpoint; the repository links a model checkpoint.

Steps

  1. Clone the CoFormer repository and enter its directory.

  2. Create and activate a Conda environment with Python 3.9.

  3. Install the specified PyTorch, torchvision and CUDA Toolkit versions, then install requirements with pip.

  4. Use the documented inference command with the image path, saved checkpoint path and output directory.

Expected outcome

A configured environment and a documented command for producing grounded situation-recognition predictions for a custom image.

Guide limitations

  • The repository documents a four-GPU, single-node training example; this recipe does not establish hardware compatibility or inference performance.
  • The README provides command arguments but this guide does not claim that inference was executed.
Documentation 1 ↗
Evidence and transparency

Sources & review

Official repository ↗
Source record details
Source availability
Active
Sources checked
17 October 2026
Original website title
jhcho99/CoFormer · Official homepage title · retained in its source language
How we review sourcesSuggest a correction