Practical overview
When it fits
Use Bert when you need repository-based BERT checkpoints and TensorFlow code to support research or fine-tuning, especially when you want to compare the listed compact English variants.
Who it is for
- Researchers building natural-language understanding experiments.
- Developers adapting a pre-trained language model for tasks such as text classification.
- Teams seeking smaller English BERT variants for computationally constrained settings.
Documented uses
- Start an NLP experiment with a pre-trained BERT checkpoint and TensorFlow code.
- Fine-tune BERT for a task such as text classification.
- Compare compact model sizes for a resource-constrained research setup.
Requirements & limits
- The repository lists 24 smaller English-only, uncased variants, with model dimensions ranging from two to 12 layers and hidden sizes from 128 to 768.
- The README says compact variants can be fine-tuned like the original models and are most effective in knowledge distillation using labels from a larger teacher.
- The repository is marked archived and carries an Apache-2.0 license.
Check current requirements in the official documentation. Source-based guidance does not mean OpenViST has run or independently tested this project.
Select and download a compact BERT checkpoint
Choose one of the smaller English, uncased BERT models and download its checkpoint for a restricted-compute environment.
Before you start
- A need for an English-only, uncased model; the compact models use WordPiece masking.
Steps
Compare the listed model sizes, such as BERT-Tiny (2 layers, hidden size 128), BERT-Mini (4, 256), BERT-Small (4, 512), or BERT-Medium (8, 512).
Select that model's linked ZIP file in the README table, or use the link for all 24 models.
Use the selected checkpoint for later fine-tuning; the smaller models can be fine-tuned in the same manner as original BERT models.
Expected outcome
A selected compact BERT checkpoint is downloaded and ready for a subsequent fine-tuning workflow.
Guide limitations
- The README does not provide the fine-tuning procedure in the material available here.
- The smaller models are described as most effective for knowledge distillation, with fine-tuning labels produced by a larger, more accurate teacher.
- The models are English-only and uncased.
Sources & review
Source record details
- Source availability
- Active
- Sources checked
- 12 October 2026
- Original website title
- google-research/bert · Official homepage title · retained in its source language