Practical overview
When it fits
Use UER-py when a text-model workflow needs an open-source PyTorch toolkit spanning pre-training, downstream fine-tuning and pretrained-model exploration, especially for medium-sized models.
Who it is for
- Researchers building or extending NLP pre-training models.
- Practitioners adapting pretrained language models to classification and other downstream text tasks.
- Teams exploring medium-sized text models with PyTorch.
Documented uses
- Preprocess text corpora for model pre-training.
- Pre-train models on general-domain text and fine-tune them for downstream tasks.
- Explore pretrained models in the project’s model zoo.
- Construct models from modular embedding, encoder, decoder and target components.
Requirements & limits
The repository is public and identifies Apache-2.0 as its license.
- The core stack requires Python 3.6 or later and PyTorch 1.1 or later, plus six, argparse, packaging and regex.
- Additional dependencies depend on the workflow: TensorFlow for certain model conversions, SentencePiece for that tokenizer, and other packages for selected downstream methods.
- The maintainers recommend TencentPretrain for multimodal models and large-model training, while describing UER-py for medium-sized text models.
Check current requirements in the official documentation. Source-based guidance does not mean OpenViST has run or independently tested this project.
Prepare a corpus for BERT pre-training
Convert a UTF-8 text corpus into the preprocessed dataset used as input to UER-py’s pretraining stage.
Before you start
- Python 3.6 or later and PyTorch 1.1 or later.
- six 1.12.0 or later, argparse, packaging and regex.
- A BERT-formatted corpus, vocabulary file and UER-py checkout.
Steps
Format the corpus with one sentence per line and blank lines between documents.
Run the repository’s documented preprocess.py command with corpus_path, vocab_path, dataset_path, processes_num and data_processor set for BERT.
Use the resulting dataset.pt as the input to pretrain.py.
Expected outcome
A preprocessed dataset file ready to be passed to the pretraining stage.
Guide limitations
- Preprocessing can be time-consuming; the README notes that multiple processes can speed it up.
- This recipe covers preprocessing and the next documented input stage, not a complete training run.
Sources & review
Source record details
- Source availability
- Active
- Sources checked
- 15 October 2026
- Original website title
- dbiir/UER-py · Official homepage title · retained in its source language
Related options
TencentPretrain — The repository identifies it as a refactored newer version of UER-py, with multimodal support and large-model training; UER-py is recommended there for medium-sized text models.