Practical overview
When it fits
Use it when you have a fine-tuned predecessor model and GLUE data for a compression experiment, or when you want to load the provided six-layer MNLI model for sentence-classification work.
Who it is for
- Researchers exploring BERT compression and progressive module replacement.
- Developers evaluating compressed language models on GLUE tasks.
- Sentence-classification practitioners looking to start from the provided six-layer MNLI model.
Documented uses
- Compress a fine-tuned BERT predecessor through progressive module replacement.
- Train and evaluate a compressed model on GLUE tasks.
- Load the pretrained MNLI model for sentence-classification experiments.
- Make predictions on GLUE tasks and prepare leaderboard submissions.
Requirements & limits
The repository is public and licensed under Apache-2.0.
- Using the implementation requires cloning and installing Hugging Face Transformers.
- The compression workflow expects a fine-tuned predecessor model saved locally and GLUE task data.
- The README says GLUE is the available task set and invites contributions for additional tasks.
- The pretrained model is six layers and is described as transferable to other sentence-classification tasks.
Check current requirements in the official documentation. Source-based guidance does not mean OpenViST has run or independently tested this project.
Load the pretrained MNLI model
Load the provided six-layer MNLI model with Hugging Face Transformers for sentence-classification use.
Before you start
- Hugging Face Transformers
- Python
Steps
Import AutoTokenizer and AutoModel from transformers.
Load the tokenizer from canwenxu/BERT-of-Theseus-MNLI.
Load the model from canwenxu/BERT-of-Theseus-MNLI.
Expected outcome
A tokenizer and model object for the pretrained six-layer MNLI model are loaded.
Guide limitations
- The README describes transfer to other sentence-classification tasks but does not provide a complete task-specific inference example.
Prepare a BERT compression run on a GLUE task
Set up the documented GLUE-based workflow for compressing a fine-tuned predecessor model.
Before you start
- Python
- A cloned and installed Hugging Face Transformers repository
- A fine-tuned predecessor model saved to a directory
- GLUE data for the selected task
Steps
Fine-tune a predecessor model using the Hugging Face GLUE instructions and save it to a directory.
Set GLUE_DIR to the GLUE data location and TASK_NAME to the selected task, as in the MRPC example.
Run run_glue.py with the saved predecessor, task data, training and evaluation options, and either a replacement scheduler or a constant replacing rate.
Expected outcome
The documented command configures a GLUE training and evaluation run that applies progressive module replacement and writes the successor model to the specified output directory.
Guide limitations
- The README provides example arguments and refers readers to the source code for detailed argument descriptions.
- The repository states that GLUE is the available task set.
Sources & review
Source record details
- Source availability
- Active
- Sources checked
- 17 October 2026
- Original website title
- JetRunner/BERT-of-Theseus · Official homepage title · retained in its source language