M2Pt: Re-parameterize cross-modal transformer weights

Understand the project

Practical overview

Use M2Pt when investigating cross-modal transformer training where target and auxiliary modality datasets need not be paired.

Researchers studying multimodal learning and transformer methods.
Developers prototyping cross-modal training for image, video, point-cloud or audio recognition.

Explore using a transformer trained on one modality to improve a model for another.
Prototype re-parameterization of auxiliary transformer weights into a target model.
Review example checkpoints and modality-specific recognition documentation.

Before you choose

The public GitHub repository identifies the project as open source under the Apache-2.0 license.

The method is designed to use data from another modality, including data described as irrelevant to the target modality; it is distinguished from approaches based on paired or interleaved data.
The README example uses Python imports for PyTorch and timm, loads model checkpoints, and calls the re-parameterization function on named attention and MLP layers.
The model zoo lists ViT-B/16 checkpoints for image, audio, point and video, with MAE pretraining identified for each modality.

Check current requirements in the official documentation. Source-based guidance does not mean OpenViST has run or independently tested this project.

Source-based practical guide

Load target and auxiliary ViT-B/16 model weights and reparameterize the auxiliary model into the target model.

Python environment with PyTorch and timm available.
Image and audio ViT-B/16 checkpoint files named Image_ViT_B16.pth and Audio_ViT_B16.pth.
The M2PT multimodal_pathway module.

Create target and auxiliary ViT-B/16 models with timm.create_model("vit_base_patch16_224", pretrained=False).
Load the image checkpoint into the target model and the audio checkpoint into the auxiliary model using torch.load and load_state_dict with strict=True.
Import reparameterize_aux_into_target_model from multimodal_pathway.
Call it with the target and auxiliary models and layer_names=('attn.qkv', 'attn.proj', 'mlp.fc1', 'mlp.fc2').

The auxiliary model is reparameterized into the target model to construct a multimodal pathway.

The example shows model construction and pathway reparameterization only; it does not document running inference, training, or evaluating the resulting model.
The example assumes the named checkpoint files are available; it does not document how to download them or configure the environment.

Evidence and transparency

Source record details

Source availability: Active
Sources checked: 16 October 2026
Original website title: AILab-CVC/M2PT · Official homepage title · retained in its source language