Practical overview
When it fits
Use LiGO when you want to investigate the documented process of growing a pretrained BERT or RoBERTa model and have the required data, checkpoint and training environment available.
Who it is for
- Researchers studying efficient transformer training and pretrained-model reuse.
- Practitioners who want to reproduce the repository's documented BERT or RoBERTa model-growth workflows.
Documented uses
- Prepare and tokenize English Wikipedia data for the documented BERT or RoBERTa workflows.
- Train a LiGO operator from a smaller pretrained model checkpoint.
- Apply a saved operator while training a larger BERT or RoBERTa model.
Requirements & limits
The public GitHub repository displays an MIT license.
- The documented recipes focus on BERT and RoBERTa, with distinct corpus-processing pipelines for each.
- The workflow requires both a source-model checkpoint and prepared, tokenized training data.
- The README lists Transformers 4.21.0 and several supporting libraries; faiss-gpu is listed, with a PyTorch GPU-indexing alternative noted if faiss is unavailable.
- The repository provides command-line training examples rather than a hosted API.
Check current requirements in the official documentation. Source-based guidance does not mean OpenViST has run or independently tested this project.
Prepare Wikipedia data for BERT or RoBERTa
Download, preprocess and tokenize the English Wikipedia corpus for the selected model family.
Before you start
- The repository's Python dependencies, including PyTorch, torchvision, Transformers 4.21.0 and h5py 2.10 or later.
- A checkout of the LiGO repository and the required configuration and data scripts.
Steps
For BERT, run the command below; for RoBERTa, run the command below.
bash data/wiki/get_data_cased.bash enbash data/wiki/get_data_cased_untokenized.bash enFor BERT, run the command below; for RoBERTa, run the command below.
bash tokenization/tokenize_wiki_bert.bashbash tokenization/tokenize_wiki_roberta.bashUse the generated tokenized corpus files in the subsequent training workflow.
Expected outcome
The README says tokenization saves corpus, HDF5 and line files locally for use in training.
Guide limitations
- BERT and RoBERTa require different data preparation and tokenization pipelines.
- This guide covers data preparation only, not training or validation.
Train and apply a LiGO operator for BERT
Follow the documented two-stage command sequence to train an operator from a small BERT checkpoint and use it during larger-model training.
Before you start
- A prepared and tokenized BERT corpus and the repository configuration files.
- A small BERT checkpoint and writable paths for saving the LiGO operator and training output.
- The listed Python dependencies, including PyTorch, torchvision and Transformers 4.21.0.
Steps
Run the command below
python run_grow_distributed.py --config configs/bert_wiki.txt --config_name configs/bert-12L-768H.json --output_dir --tune_width --tune_depth --source_model_path --fuse_init_scheme stackbert_noisy rand --max_steps 100 --logging_steps 100 --ckpt_steps 100 --should_continueRun `python run_lm_distributed.py --config configs/bert_wiki.txt --config_name configs/bert-12L-768H.json --output_dir --grow_scheme ligo --source_model_path --pretrained_ligo_path --fuse_init_scheme stackbert_noisy rand --learning_rate 2e-4 --warmup_steps 0 --sho
Replace the documented path placeholders with the local checkpoint and output locations.
Expected outcome
The commands are intended to train a LiGO operator and then use it to grow and train a 12-layer, 768-hidden-size BERT model.
Guide limitations
- The README gives this recipe for its stated configuration and flags; it does not establish results for other models or environments.
- The sequence assumes data preparation and a source-model checkpoint are already available.
Sources & review
Source record details
- Source availability
- Active
- Sources checked
- 17 October 2026
- Original website title
- VITA-Group/LiGO · Official homepage title · retained in its source language