OpenViSTAI · Agents · Blockchain · Crypto
Submit a project
Menu
AI Models

Clip Fine Tune Registers Gated

Clip Fine Tune Registers Gated is a public Python repository containing fine-tuned CLIP vision-language model variants and scripts for retrieval-focused experimentation and model visualization. Its central variant adds register tokens and gated MLPs, changes the project describes as reducing CLIP's modality gap.

Explore the practical guide ↓

Independent OpenViST project profile, based on linked sources.

Clip Fine Tune Registers Gated — OpenViST project profile in AI Models.
OpenViST project illustration. See the linked sources for product details.
Understand the project

Practical overview

When it fits

Use it when you want to study or experiment with register-gated CLIP retrieval models and their internal features, rather than access a documented hosted inference service.

Who it is for

  • Researchers exploring image-text retrieval with fine-tuned CLIP variants.
  • Developers examining transformer features, fusion gates, or model behavior with the repository's scripts.

Documented uses

  • Explore a fine-tuned CLIP model for image-text retrieval.
  • Visualize selected layer features and activation maxima.
  • Inspect fusion gating toward register or CLS tokens.
  • Use gradient ascent with a selected model and image.
Before you choose

Requirements & limits

The repository is public and lists an MIT license.

  • The README presents retrieval and a reduced modality gap as the model's focus; it points users seeking zero-shot accuracy or text encoders for generative AI to a separate classic CLIP fine-tuning repository.
  • The documented variant adds four register tokens, gated MLPs in vision-transformer residual blocks, and a final fusion MLP.
  • The README links full-model and text-encoder-only files, as well as a Long-CLIP variant described as supporting 248 tokens.
  • Model-specific scripts must match the model family: the README says to use OAI code for standard OpenAI CLIP models and REG code for register-gated models.
  • The README notes that selecting layer or feature indices beyond valid ranges can produce an IndexError.

Check current requirements in the official documentation. Source-based guidance does not mean OpenViST has run or independently tested this project.

Source-based practical guide

Visualize selected fusion-MLP features

Generate a feature-activation visualization for specified layers and features with the fusion-MLP visualization script.

Before you start

  • The fine-tuned model file at models/ViT-L-14-REG-GATED-balanced-ckpt12.safetensors (the script's documented default model).
  • A valid layer and feature range; the script reports valid ranges in its green output text.

Steps

  1. Run the command below.

    Documented command
    python REG-12-XGATED-featureviz-fusion-mlps.py --layer_range 8-11 --feature_range 42,1000,77

  2. Review the visualization for features 42, 1000, and 77 in layers 8 through 11.

Expected outcome

A visualization of the selected features across layers 8, 9, 10, and 11.

Guide limitations

  • Selecting layer or feature indices outside the valid range raises an IndexError.
  • The README does not specify installation or environment setup for running the script.
Documentation 1 ↗
Evidence and transparency

Sources & review

Official repository ↗
Source record details
Source availability
Active
Sources checked
15 October 2026
Original website title
zer0int/CLIP-fine-tune-registers-gated · Official homepage title · retained in its source language
How we review sourcesSuggest a correction