Serving: Machine-Learning Model Inference

Understand the project

Practical overview

When it fits

Use TensorFlow Serving after model training when you need a service to manage model versions and expose inference to clients. The repository recommends Docker for a straightforward setup and documents other deployment routes, including Kubernetes.

Who it is for

Teams that need to expose trained TensorFlow models to client applications.
Engineers building model-serving backends that need multiple models or versions and inference endpoints.
Developers extending a serving system to handle additional machine-learning model or data types.

Documented uses

Make a trained SavedModel available for prediction requests.
Serve multiple models or versions simultaneously and deploy new versions without changing client code.
Route inference requests through HTTP or gRPC endpoints.
Batch inference requests for joint GPU execution using configurable latency controls.
Extend serving to model and data types beyond the built-in TensorFlow integration.

Before you choose

Requirements & limits

The project is publicly available under the Apache-2.0 license. The repository documents Docker-based and non-Docker setup options.

TensorFlow SavedModel is the documented path for serving TensorFlow models.
HTTP and gRPC are both documented inference interfaces.
The README recommends Docker unless container use does not meet specific needs.
Non-TensorFlow model support is an extension capability rather than the built-in TensorFlow path.
The repository documents Kubernetes deployment, configuration, performance, and custom servable guidance.

Check current requirements in the official documentation. Source-based guidance does not mean OpenViST has run or independently tested this project.

Source-based practical guide

Serve the included TensorFlow demo model with Docker

Start TensorFlow Serving with the included half_plus_two SavedModel and send a REST prediction request.

Before you start

Docker
Git
The TensorFlow Serving repository’s included demo model files

Steps

Pull the TensorFlow Serving image with the command below.
Documented command
```
docker pull tensorflow/serving
```
Clone the repository with the command below and set TESTDATA="$(pwd)/serving/tensorflow_serving/servables/tensorflow/testdata".
Documented command
```
git clone https://github.com/tensorflow/serving
```
Run the container, mounting $TESTDATA/saved_model_half_plus_two_cpu at /models/half_plus_two, setting MODEL_NAME=half_plus_two, and publishing port 8501: the command below.
Documented command
```
docker run -t --rm -p 8501:8501 -v "$TESTDATA/saved_model_half_plus_two_cpu:/models/half_plus_two" -e MODEL_NAME=half_plus_two tensorflow/serving &
```
Send the documented prediction request: the command below. The README gives the expected response as { "predictions": [2.5, 3.0, 4.5] }.
Documented command
```
curl -d '{"instances": [1.0, 2.0, 5.0]}' -X POST http://localhost:8501/v1/models/half_plus_two:predict
```

Expected outcome

A TensorFlow Serving container exposes the demo model for REST inference on port 8501; the example request returns predictions for the supplied instances.

Guide limitations

This recipe covers the included CPU demo model and REST request only; it does not document production configuration or deployment.
The example uses --rm and runs the container in the background; persistent deployment procedures are not included here.

Documentation 1 ↗

COMPARE OPTIONS

Compare options for Model inference and serving

These projects document a common task and share a project role. This is a source-based comparison, not an execution benchmark or a guarantee of interchangeability.

Project	Documented purpose	Platform	Access and requirements
Serving	Make a trained SavedModel available for prediction requests.	Docker, Linux, Self-hosted, Server	The project is publicly available under the Apache-2.0 license. The repository documents Docker-based and non-Docker setup options.
Morphingdb	Import a libtorch model and create predictions from values in a PostgreSQL table.	Docker, Linux, Self-hosted, Server	The README documents a Docker image and a build-from-repository option, followed by connecting to PostgreSQL and enabling the pgdl extension.
Gateway — pyadav	Send chat-completion requests to a provider through a common server endpoint.	Server	The repository is public and identifies the project as Apache-2.0 licensed. README installation instructions include binaries, platform package managers and Docker.

Evidence and transparency

Sources & review

Official repository ↗

Source record details

Source availability: Active
Sources checked: 10 October 2026
Original website title: tensorflow/serving · Official homepage title · retained in its source language

How we review sources Suggest a correction

Serving