Practical overview
When it fits
Use it when you need a research codebase for exploring TM-CTC speech transcription across audio and visual inputs, or for following its documented LRS2 training and inference workflows.
Who it is for
- Researchers studying audio-visual speech recognition and lip-reading.
- Developers prototyping or comparing multimodal speech-to-text models.
Documented uses
- Generate transcripts from audio recordings.
- Generate transcripts from lip-reading video.
- Compare audio-only, video-only and combined audio-visual model results.
- Preprocess LRS2 data, train models, and evaluate test-set word error rate.
Requirements & limits
The repository is public and released under the MIT license.
- The README specifies an older software stack: Python 3.6.9, PyTorch 1.2.0 and CUDA 10.0 for NVIDIA GPU use.
- Training requires the LRS2 dataset; the README also describes using custom data with specifications and folder structure similar to LRS2.
- The repository provides pre-trained weights for the visual frontend, language model and AO, VO and AV models.
- Evaluation results reported in the README use LRS2 and include clean and noisy audio conditions, as well as greedy and beam-search decoding.
Check current requirements in the official documentation. Source-based guidance does not mean OpenViST has run or independently tested this project.
Generate predictions with a trained model
Configure the project and use demo.py to generate predictions for samples in a demo directory.
Before you start
- Python 3.6.9 and the listed project dependencies, including PyTorch 1.2.0 and FFmpeg 2.8.15.
- A trained model and any required pre-trained weights placed in the locations described by the README.
- Demo samples arranged according to the specifications in demo.py.
Steps
Set the required options in config.py.
Place the applicable model weights in the documented model locations.
Prepare samples in a demo directory following demo.py specifications.
Run demo.py to generate predictions for the samples.
Expected outcome
Predictions for the samples in the specified demo directory.
Guide limitations
- The README refers users to demo.py for sample specifications.
- The listed software versions are the repository's stated requirements and do not guarantee compatibility with current systems.
Train and evaluate a model on LRS2
Follow the repository’s documented preprocessing, curriculum pretraining, training and test-set evaluation stages.
Before you start
- The complete LRS2 dataset.
- Python 3.6.9, PyTorch 1.2.0, FFmpeg 2.8.15 and the other listed Python packages.
- CUDA 10.0 and cudatoolkit 10.0 if using an NVIDIA GPU.
Steps
Set the required configuration options in config.py.
Run preprocess.py to prepare the dataset samples.
Run pretrain.py for curriculum-learning iterations, changing PRETRAIN_NUM_WORDS in config.py between iterations as described.
Run train.py to train the model, then test.py to evaluate it on the test set.
Expected outcome
A trained model and test-set performance results.
Guide limitations
- The README describes 11 curriculum-learning iterations and says training is stopped when validation-set WER flattens and the learning rate reaches its minimum.
- The authors report that each model took around seven days to train using a GPU with 11 GB of memory; this is their reported experience, not a runtime guarantee.
Sources & review
Source record details
- Source availability
- Active
- Sources checked
- 15 October 2026
- Original website title
- smeetrs/deep_avsr · Official homepage title · retained in its source language