OpenViSTAI · Agents · Blockchain · Crypto
Submit a project
Menu
AI Models

Distill Bert Textgen

Distill Bert Textgen is Python research code for reproducing IWSLT German-to-English text-generation experiments that distill knowledge learned in BERT into a sequence-to-sequence translation model.

Explore the practical guide ↓

Independent OpenViST project profile, based on linked sources.

Distill Bert Textgen — OpenViST project profile in AI Models.
OpenViST project illustration. See the linked sources for product details.
Understand the project

Practical overview

When it fits

Use it when you need a code-based reference for reproducing the repository’s BERT distillation workflow for German-to-English translation research.

Who it is for

  • Researchers studying BERT knowledge distillation for text generation.
  • Developers reproducing or adapting the repository’s German-to-English machine-translation experiments.

Documented uses

  • Prepare the project’s German-to-English experiment data.
  • Fine-tune a multilingual BERT teacher and extract hidden states and top-k logits.
  • Train a sequence-to-sequence translation model using BERT-derived knowledge.
  • Run inference on the development split and save experiment outputs.
Before you choose

Requirements & limits

The repository is publicly available under the MIT license. The documented setup targets Ubuntu 18.04 with an Nvidia GPU and Docker.

  • The documented pipeline requires an Nvidia GPU, Nvidia driver and nvidia-docker alongside Docker.
  • The authors say they do not plan to support other operating systems or CPU-only machines.
  • The repository includes a modified OpenNMT-py submodule and recommends Docker for reproducibility.
  • The README’s memory guidance is based on testing with one Nvidia Titan RTX GPU with 24 GB RAM.

Check current requirements in the official documentation. Source-based guidance does not mean OpenViST has run or independently tested this project.

Source-based practical guide

Prepare the IWSLT German-to-English data

Use the repository setup script to download raw data and preprocess it into the project’s expected data folders.

Before you start

  • Ubuntu 18.04 with an Nvidia GPU, as the documented test environment.
  • Docker, an Nvidia driver and nvidia-docker.
  • The repository cloned with its submodule.

Steps

  1. Clone the repository and submodule using the documented recursive clone command.

  2. Run source scripts/setup.sh to download and preprocess the data.

  3. Use the generated /dump contents as the preprocessed data mounted by the project container.

Expected outcome

The data folder contains downloaded and preprocessed German-to-English experiment files for later training and inference stages.

Guide limitations

  • The authors do not plan to support other operating systems or CPU-only machines.
  • The README recommends Docker for reproducibility and says non-Docker installations will not receive official troubleshooting support.
Documentation 1 ↗
Source-based practical guide

Run the documented distillation and translation workflow

Train the documented German-to-English model with BERT-derived knowledge and run development-set inference.

Before you start

  • Prepared data from the project setup script.
  • Docker, an Nvidia driver and nvidia-docker on the documented Ubuntu/Nvidia setup.
  • The repository and its modified OpenNMT-py submodule.

Steps

  1. Launch the container with source launch_container.sh .

  2. Inside the container, run the documented C-MLM fine-tuning command.

  3. Extract teacher hidden states and top-k logits with the documented dump_teacher_hiddens.py and dump_teacher_topk.py commands.

  4. Run the documented opennmt/train.py command with BERT knowledge distillation enabled.

  5. Run ./run_mt.sh /output/ 100000 dev 5 0.6 to translate the development split.

Expected outcome

The workflow produces a trained translation experiment and writes development-split inference results under the experiment output directory.

Guide limitations

  • The README reports testing on a single Nvidia Titan RTX with 24 GB RAM; it suggests reducing batch size and increasing gradient accumulation if out of memory.
  • The documented environment does not include CPU-only or other-operating-system support.
Documentation 1 ↗
Evidence and transparency

Sources & review

Official repository ↗
Source record details
Source availability
Active
Sources checked
17 October 2026
Original website title
ChenRocks/Distill-BERT-Textgen · Official homepage title · retained in its source language
How we review sourcesSuggest a correction