Practical overview
When it fits
Use it when building a Python training workflow that needs differential privacy for a supported Hugging Face transformer model, particularly when experimenting with ghost clipping.
Who it is for
- Developers experimenting with differentially private fine-tuning of Hugging Face models.
- Researchers comparing private training approaches, including ghost clipping.
- Teams building privacy-preserving NLP training pipelines.
Documented uses
- Attach a privacy engine to a transformer model's optimizer.
- Train using per-example losses passed to the optimizer's step method.
- Try ghost clipping when reducing peak GPU memory is important.
- Use the included examples as starting points for table-to-text and classification experiments.
Requirements & limits
The project is a public GitHub repository listed under the Apache-2.0 license. Installation instructions require Python 3.8 or later; the documented test suite requires pytest and a GPU.
- The optimizer step expects a one-dimensional per-example loss passed as a keyword argument; users should not call
loss.backward()directly. - Inputs must be batch-first, and the model must be in training mode for the underlying hooks to run.
- Ghost clipping is a one-setting change in the documented example, with a potential trade-off of slower training.
- The README lists specific supported model classes rather than claiming universal Transformers model support.
Check current requirements in the official documentation. Source-based guidance does not mean OpenViST has run or independently tested this project.
Set up Private Transformers
Install the Python package from its GitHub repository and run the documented test suite.
Before you start
- Python 3.8 or later
- pytest
- A GPU to run the test suite
Steps
Install with the command below.
pip install git+https://github.com/lxuechen/private-transformers.gitRun
pytest -s testsfrom the project checkout.
Expected outcome
The package is installed and the project's documented test suite has been invoked.
Guide limitations
- The test suite requires a GPU.
- This setup recipe does not cover training a model.
Add the privacy engine to a Hugging Face training loop
Use the documented per-example-loss workflow for differentially private transformer training.
Before you start
- An installed Private Transformers package
- A Hugging Face transformer model and optimizer
- Mini-batch inputs and a one-dimensional per-example loss
Steps
Create the transformer model and optimizer, then attach the optimizer to a
PrivacyEngine.Set the model to training mode and compute a one-dimensional loss with one value per example.
Pass the loss as a keyword argument to
optimizer.steporoptimizer.virtual_step; do not callloss.backward()directly.Repeat the loss computation and optimizer step for subsequent mini-batches.
Expected outcome
The documented training loop routes per-example losses through the privacy-enabled optimizer.
Guide limitations
- Inputs must use batch-first format.
- The project lists support for particular Hugging Face model classes; BART conditional generation requires frozen positional embedding layers.
- Ghost clipping can reduce memory use but may slow training.
Sources & review
Source record details
- Source availability
- Active
- Sources checked
- 17 October 2026
- Original website title
- lxuechen/private-transformers · Official homepage title · retained in its source language