Practical overview
When it fits
Use it to study or adapt an end-to-end image-captioning experiment when you can work with the documented Python and PyTorch setup and provide the dataset and embedding-vector files it expects.
Who it is for
- Learners studying transformer-based image captioning and machine-learning workflows.
- Developers exploring a notebook-based PyTorch implementation with dataset preparation, training and inference stages.
- Researchers who want to inspect generated captions, reference captions, evaluation metrics and attention visualizations.
Documented uses
- Prepare MS COCO 2017 images and annotations into the project’s processed dataset files.
- Train or resume training of the image-captioning model on a CPU or GPU.
- Generate test-split captions using beam search and compare them with reference captions.
- Review caption-generation metrics and encoder-decoder cross-attention.
Requirements & limits
The repository is public and identifies its license as MIT.
- The documented workflow uses MS COCO 2017 and expects pre-trained embedding-vector files supported by torchtext.vocab.
- Training requires a configuration JSON and supports either CPU or GPU.
- The README specifies Python 3.8.12 and warns that attention visualization requires modifying PyTorch 1.8’s attention-weight handling.
- The project’s analysis reports early overfitting and variable BLEU-4 results, which matters when judging it as an experimental starting point.
Check current requirements in the official documentation. Source-based guidance does not mean OpenViST has run or independently tested this project.
Prepare an MS COCO dataset for the project
Create the processed image, caption and vocabulary files used by the training and inference scripts.
Before you start
- Python 3.8.12 and the libraries listed in requirements.txt.
- MS COCO 2017 train and validation images and annotations.
- A directory of supported pre-trained embedding-vector files and an output directory.
Steps
Install the project dependencies with pip install -r requirements.txt.
Provide the dataset directory, annotation and image paths, output directory, vector directory and vector dimensionality to code/create_dataset.py.
Run code/create_dataset.py; its outputs include HDF5 image files, JSON caption and length files, and vocab.pth.
Expected outcome
Processed files for train, validation and test splits that the project’s training and inference scripts expect.
Guide limitations
- The README describes MS COCO 2017 as the dataset used in the project.
- It notes that the data split is randomly created and that captions occurring fewer than three times are replaced with an unknown token.
Train the model and evaluate test-split captions
Use the project scripts to train the captioning model and produce test-split captions with evaluation and attention data.
Before you start
- The processed dataset files produced by code/create_dataset.py.
- A configuration JSON file and a device choice of CPU or GPU.
- The project dependencies installed in the documented Python 3.8.12 environment.
Steps
Run code/run_train.py with the dataset directory, configuration path and device; pass a checkpoint filename if resuming training.
Run code/inference_test.py with the processed dataset directory and an output save directory.
Expected outcome
The inference script saves a pandas DataFrame with generated captions, top-k candidates, ground-truth captions, cross-attention and BLEU, GLEU and METEOR metrics.
Guide limitations
- The README reports early overfitting in its training analysis and says BLEU-4 varied across the test set.
- The documented attention visualization setup requires a change to PyTorch 1.8 so attention weights are retained for each head.
Sources & review
Source record details
- Source availability
- Active
- Sources checked
- 14 October 2026
- Original website title
- zarzouram/image_captioning_with_transformers · Official homepage title · retained in its source language