OpenViSTAI · Agents · Blockchain · Crypto
Submit a project
Menu
AI Infrastructure & Serving

PyTorch C++ Inference

PyTorch C++ Inference is a public repository of C++ examples for running PyTorch models through the PyTorch C++ API. It demonstrates exporting a TorchScript ResNet18 image-classification model and serving predictions either from a standalone executable or an HTTP server.

Explore the practical guide ↓

Independent OpenViST project profile, based on linked sources.

PyTorch C++ Inference — OpenViST project profile in AI Infrastructure & Serving.
OpenViST project illustration. See the linked sources for product details.
Understand the project

Practical overview

When it fits

Use it when you need a concrete C++ pattern for running TorchScript inference locally, either as a command-line image classifier or as a prediction server.

Who it is for

  • C++ developers integrating PyTorch model inference into an executable or web service
  • Developers evaluating a Docker-based CPU or CUDA 10 inference example

Documented uses

  • Build a C++ executable that classifies an image using an exported TorchScript model.
  • Run a local HTTP prediction server for the example model on port 8181.
  • Use the documented CPU or CUDA 10 Docker environment as a starting point for the example.
Before you choose

Requirements & limits

The repository is publicly accessible and identified as open source. No license is specified in the supplied repository details.

  • The documented workflow requires LibTorch headers and shared libraries.
  • The provided model-export example uses Python to create a TorchScript ResNet18 model.
  • The Docker examples cover CPU and CUDA 10; other CUDA versions are not documented.
  • Replacing ResNet18 requires a model compatible with the PyTorch C++ API and the example's input and output handling.

Check current requirements in the official documentation. Source-based guidance does not mean OpenViST has run or independently tested this project.

Source-based practical guide

Build the documented C++ prediction server

Prepare the documented container environment and build the web-server executable for the ResNet18 example.

Before you start

  • The repository cloned locally
  • Docker
  • A choice of the documented CPU or CUDA 10 Dockerfile
  • The matching LibTorch package and unzip utility

Steps

  1. From the repository, build a Docker image in either the docker/cpu or docker/cuda10 directory using the documented docker build command.

  2. Start a container with the repository mounted and port 8181 forwarded, then enter the mounted repository directory.

  3. Download and unzip the matching LibTorch package so its headers and shared libraries are available.

  4. Run python3 resnet.py in models/resnet to export the ResNet18 model, then follow the server build instructions in inference-cpp/cnn-classification/server.

Expected outcome

A built server executable for the example model; the README documents running it at localhost:8181/predict and using test_api.py to send a request.

Guide limitations

  • The documented model is ResNet18 and the documented CUDA container uses CUDA 10.
  • The instructions do not establish compatibility with newer CUDA versions or other model input and output formats.
Documentation 1 ↗Documentation 2 ↗
Source-based practical guide

Build the standalone image-classification executable

Build and invoke the documented C++ executable with an image, exported model, labels file and GPU flag.

Before you start

  • The repository and LibTorch environment prepared as documented
  • An exported TorchScript model and labels file
  • An image file to classify

Steps

  1. Export the example ResNet18 model by running python3 resnet.py in models/resnet.

  2. Go to inference-cpp/cnn-classification and run ./build.sh.

  3. Run ./predict with the image path, exported model path, labels path and true or false GPU flag, following the README's CPU example.

Expected outcome

A standalone executable invocation for image classification using the supplied model and labels.

Guide limitations

  • The example and invocation documented are for image classification with ResNet18.
  • The repository's documented GPU container targets CUDA 10.
Documentation 1 ↗Documentation 2 ↗
COMPARE OPTIONS

Compare options for Model inference and serving

These projects document a common task and share a project role. This is a source-based comparison, not an execution benchmark or a guarantee of interchangeability.

ProjectDocumented purposePlatformAccess and requirements
PyTorch C++ InferenceBuild a C++ executable that classifies an image using an exported TorchScript model.Linux, Docker, ServerThe repository is publicly accessible and identified as open source. No license is specified in the supplied repository details.
Nano Vllm V1Generate text offline from prompts using a local model.Linux, Self-hostedThe repository is public and identifies the project as MIT-licensed. It is designed for Linux and self-hosted use.
MagicompilerCapture a Transformer inference workload across block boundaries for whole-graph compilation.LinuxThe repository is public and licensed under Apache-2.0. The README documents local installation and a prebuilt Docker image.
Evidence and transparency

Sources & review

Official repository ↗Guide documentation ↗
Source record details
Source availability
Active
Sources checked
23 October 2026
Original website title
Wizaron/pytorch-cpp-inference · Official homepage title · retained in its source language
How we review sourcesSuggest a correction