OpenViSTAI · Agents · Blockchain · Crypto
Submit a project
Menu
AI Models

Pert

Pert is a pre-trained BERT-structure language model that learns text semantics by predicting original token positions in permuted input, without introducing the [MASK] token. Its repository provides Chinese and English checkpoints in base and large sizes, along with Chinese reading-comprehension variants.

Explore the practical guide ↓

Independent OpenViST project profile, based on linked sources.

Pert — OpenViST project profile in AI Models.
OpenViST project illustration. See the linked sources for product details.
Understand the project

Practical overview

When it fits

Use PERT when you want to evaluate a BERT-structure model trained with a permuted language-modeling objective, or adapt its released weights to a downstream NLU task.

Who it is for

  • NLP practitioners exploring a BERT-based pre-trained model for Chinese or English language understanding.
  • Researchers comparing pre-training approaches or evaluating checkpoints on NLU benchmarks.
  • Developers adapting pre-trained weights for downstream fine-tuning or further pre-training.

Documented uses

  • Load a Chinese or English PERT checkpoint using Transformers.
  • Fine-tune PERT for a downstream language-understanding task such as text classification.
  • Evaluate PERT against other pre-trained models on a target benchmark.
  • Use the Transformer weights as an initialization for further pre-training.
Before you choose

Requirements & limits

The repository is public and lists an Apache-2.0 license.

  • Check language and size: the listed choices include Chinese and English base and large models.
  • The repository says PERT improves performance on some NLU tasks but performs worse on others, so results should be assessed on the intended task.
  • The original downloads are TensorFlow 1.15 weights; the repository also points to PyTorch and TensorFlow 2 model versions through Transformers.
  • The published weights include the Transformer component and can be used for downstream fine-tuning or as initialization for further pre-training.

Check current requirements in the official documentation. Source-based guidance does not mean OpenViST has run or independently tested this project.

Source-based practical guide

Load a PERT checkpoint with Transformers

Load a listed PERT model using the BERT tokenizer and model classes documented by the project.

Before you start

  • A PERT model name from the repository’s model list.
  • The Transformers library.

Steps

  1. Choose a Chinese or English PERT checkpoint from the model list.

  2. Pass its listed model name to BertTokenizer.from_pretrained.

  3. Load the same model name with BertModel.from_pretrained.

Expected outcome

A tokenizer and BERT-structure model object loaded from the selected PERT checkpoint.

Guide limitations

  • The repository documents this loading pattern for the PERT models; reading-comprehension models use BertForQuestionAnswering instead.
  • The project reports that performance varies across NLU tasks.
Documentation 1 ↗
Evidence and transparency

Sources & review

Official repository ↗
Source record details
Source availability
Active
Sources checked
14 October 2026
Original website title
ymcui/PERT · Official homepage title · retained in its source language
How we review sourcesSuggest a correction