Practical overview
When it fits
Use it when you want a code-based starting point for CIFAR-10 distillation experiments, with configurations and scripts for teacher–student training and hyperparameter exploration.
Who it is for
- Developers and researchers exploring teacher–student knowledge distillation in PyTorch.
- Practitioners comparing compact CNN students with pretrained or deeper teacher networks on CIFAR-10.
Documented uses
- Train a five-layer CNN using knowledge from a pretrained ResNet-18 teacher.
- Train ResNet-18 with knowledge distilled from deeper teacher models.
- Search experiment hyperparameters and synthesize recent search results into a table.
- Save and load training checkpoints.
Requirements & limits
The repository is public and MIT-licensed.
- The documented dataset is CIFAR-10; other datasets are not covered by the supplied examples.
- The README instructs users to install PyTorch and other dependencies from requirements.txt.
- Pretrained teacher checkpoints are distributed separately and must be placed under the expected experiments/ paths for the described runs.
- Training, hyperparameter search, and result synthesis are handled by separate scripts.
Check current requirements in the official documentation. Source-based guidance does not mean OpenViST has run or independently tested this project.
Set up the repository and teacher checkpoints
Prepare the documented project files and pretrained teacher checkpoints for the CIFAR-10 experiments.
Before you start
- Python environment with pip
- PyTorch and the dependencies listed in requirements.txt
- Access to the repository and the pretrained teacher-checkpoint archive
Steps
Clone the repository from its GitHub URL.
Install dependencies with pip install -r requirements.txt.
Download and extract the pretrained teacher-checkpoint archive.
Move the extracted subfolders into the repository's experiments/ directory, following the default path naming.
Expected outcome
The repository dependencies are installed and the pretrained checkpoints are arranged where the documented experiment paths expect them.
Guide limitations
- This guide covers setup only; training and search are separate documented stages.
- The README says to follow default path naming but does not detail troubleshooting for missing or differently named checkpoint files.
Run a documented CIFAR-10 distillation experiment
Start one of the repository's documented teacher–student training runs.
Before you start
- Repository dependencies, including PyTorch, installed from requirements.txt
- CIFAR-10 experiment configuration under experiments/
- Pretrained teacher checkpoints arranged according to the documented paths
Steps
Choose an included experiment configuration, such as experiments/cnn_distill or experiments/resnet18_distill/resnext_teacher.
Review or modify the hyperparameters in the configuration's params.json file.
Run python train.py --model_dir experiments/cnn_distill for the five-layer CNN with a ResNet-18 teacher, or use the documented ResNet-18/ResNeXt-29 experiment path.
Expected outcome
The training script is invoked for a documented CIFAR-10 teacher–student experiment.
Guide limitations
- The repository documentation does not guarantee a particular result from a new run.
- The guide covers the documented experiment paths and does not describe adapting the training code to other datasets.
Sources & review
Source record details
- Source availability
- Active
- Sources checked
- 16 October 2026
- Original website title
- haitongli/knowledge-distillation-pytorch · Official homepage title · retained in its source language