Practical overview
When it fits
Use it when you want a code-based research workflow for length-adaptive BERT or DistilBERT experiments on SQuAD 1.1 or GLUE, rather than a hosted model service.
Who it is for
- Researchers studying adaptive sequence lengths and Transformer efficiency.
- Practitioners evaluating BERT or DistilBERT workflows on SQuAD 1.1 or GLUE.
Documented uses
- Fine-tune a pretrained BERT or DistilBERT model on a documented benchmark task.
- Continue training with LengthDrop and Drop-and-Restore.
- Search for length configurations for anytime prediction after length-adaptive training.
- Measure FLOPs using the listed torchprofile dependency.
Requirements & limits
The public GitHub repository is open source under the Apache-2.0 license.
- Current documented coverage is limited to BERT and DistilBERT and the SQuAD 1.1 and GLUE tasks.
- The search stage follows length-adaptive training and uses the resulting checkpoint.
- Other use cases may require adaptation; the README does not detail those changes.
Check current requirements in the official documentation. Source-based guidance does not mean OpenViST has run or independently tested this project.
Train and search length configurations for SQuAD 1.1
Prepare the documented SQuAD data, train a standard BERT baseline, continue with length adaptation, then search configurations for anytime prediction.
Before you start
- Python 3, PyTorch, Hugging Face Transformers, and torchprofile.
- A pretrained BERT checkpoint.
- SQuAD 1.1 train-v1.1.json, dev-v1.1.json, and evaluate-v1.1.py files in a SQuAD data directory.
Steps
Prepare the SQuAD files in the directory referenced by SQUAD_DIR.
Use run_squad.py to fine-tune and evaluate the standard BERT baseline, saving its best checkpoint.
Continue fine-tuning that checkpoint with the documented length-adaptive options, including --length_adaptive, --num_sandwich, --length_drop_ratio_bound, and --layer_dropout_prob.
Run run_squad.py with --do_search on the resulting length-adaptive checkpoint to search for length configurations.
Expected outcome
A trained length-adaptive SQuAD checkpoint followed by an evolutionary search output directory for configurations intended for anytime prediction.
Guide limitations
- The documented task and model combination is SQuAD 1.1 with BERT.
- The guide describes the README workflow; it does not establish successful execution or results.
Train and search length configurations for a GLUE task
Prepare GLUE data, train a standard BERT task model, continue with length adaptation, and run the documented configuration search.
Before you start
- Python 3, PyTorch, Hugging Face Transformers, and torchprofile.
- A GLUE task dataset downloaded into the directory structure used by the scripts.
- A pretrained BERT checkpoint.
Steps
Use the referenced GLUE download script to prepare data in a GLUE directory.
Run run_glue.py for the selected task to fine-tune and evaluate a standard BERT checkpoint.
Continue training from the best standard checkpoint with the documented length-adaptive options.
Run run_glue.py with --do_search on the length-adaptive checkpoint.
Expected outcome
A length-adaptive GLUE task checkpoint and an evolutionary search output directory.
Guide limitations
- The README documents GLUE as a supported downstream benchmark and limits current model support to BERT and DistilBERT.
- The guide describes the README workflow; it does not establish successful execution or results.
Sources & review
Source record details
- Source availability
- Active
- Sources checked
- 18 October 2026
- Original website title
- clovaai/length-adaptive-transformer · Official homepage title · retained in its source language