OpenViSTAI · Agents · Blockchain · Crypto
Submit a project
Menu
AI Models

Modernbert

ModernBERT is an open-source Python research repository for modernizing BERT-style encoder models through architecture changes and scaling. It introduces FlexBERT, a modular approach to encoder building blocks, and provides YAML-configured workflows for pre-training and evaluation.

Explore the practical guide ↓

Independent OpenViST project profile, based on linked sources.

Modernbert — OpenViST project profile in AI Models.
OpenViST project illustration. See the linked sources for product details.
Understand the project

Practical overview

When it fits

Use the repository when you need research and training code for BERT-style encoders, including configurable pre-training, evaluation or retrieval-model experiments. For integrating released checkpoints into a common pipeline, the repository points to the separate ModernBERT Hugging Face collection.

Who it is for

  • Researchers experimenting with BERT-style encoder architectures and training.
  • Engineers building encoder or embedding experiments from configurable model components.
  • Teams evaluating trained models on GLUE or exploring dense and ColBERT-style retrieval workflows.

Documented uses

  • Configure encoder model experiments using repository YAML files.
  • Train models with text or tokenized datasets in supported formats.
  • Evaluate ModernBERT models on GLUE using a checkpoint and training configuration.
  • Explore scripts for training and evaluating dense retrieval and ColBERT models.
Before you choose

Requirements & limits

The public GitHub repository specifies an Apache-2.0 license.

  • The documented training environment targets GPU-equipped machines and includes Flash Attention installation.
  • Training is configured through YAML and uses the Composer framework.
  • StreamingTextDataset supports MDS, CSV/TSV and JSONL; NoStreamingDataset requires decompressed MDS data.
  • The repository distinguishes research and training code from the separately linked Hugging Face checkpoint collection.
  • The README describes its documentation as under construction.

Check current requirements in the official documentation. Source-based guidance does not mean OpenViST has run or independently tested this project.

Source-based practical guide

Set up the documented GPU training environment

Prepare the repository’s Conda environment and install its documented Flash Attention dependency.

Before you start

  • A GPU-equipped machine.
  • The repository’s environment.yaml file.
  • Conda and pip.

Steps

  1. Create the environment with conda env create -f environment.yaml.

  2. Activate it with conda activate bert24.

  3. Install Flash Attention 2 with the command below.

    Documented command
    pip install "flash_attn==2.6.3" --no-build-isolation

Expected outcome

The documented Conda environment is created and the specified Flash Attention 2 package is installed.

Guide limitations

  • The README notes that H100 users may need to build Flash Attention 3; it also describes Flash Attention 2 as the alternative where FA3 is unsupported.
  • The README is described as under construction.
Documentation 1 ↗
Source-based practical guide

Configure and launch a ModernBERT training run

Use the repository’s example YAML configuration to launch a Composer training job.

Before you start

  • The documented GPU training environment.
  • Training data in a supported format: MDS, CSV/TSV or JSONL for StreamingTextDataset, or decompressed MDS format for NoStreamingDataset.
  • A training YAML such as yamls/main/modernbert-base.yaml.

Steps

  1. Review an example training YAML in the repository’s yamls folder.

  2. Choose the dataset class through the configuration’s streaming setting.

  3. Launch the documented example with composer main.py yamls/main/modernbert-base.yaml.

Expected outcome

Composer is instructed to run the training configuration on available GPUs.

Guide limitations

  • The repository warns that streaming data can distribute memory unevenly across accelerators.
  • The documented command is an example; no training result is specified.
Documentation 1 ↗
Evidence and transparency

Sources & review

Official repository ↗
Source record details
Source availability
Active
Sources checked
11 October 2026
Original website title
AnswerDotAI/ModernBERT · Official homepage title · retained in its source language
How we review sourcesSuggest a correction

Related options

ModernBERT Collection on Hugging Face — The repository identifies this separate collection as the route for users seeking ModernBERT models designed to integrate into common pipelines.