Mlserver: Serve models through REST and gRPC

Understand the project

Practical overview

When it fits

Use Mlserver when a model needs a protocol-based inference server, especially when REST and gRPC support, multi-model serving, or integration with Seldon Core or KServe is relevant.

Who it is for

Teams adding a REST or gRPC inference interface to trained models.
ML platform teams serving multiple models or frameworks in one process.
Teams integrating model inference with Kubernetes-native serving frameworks such as Seldon Core or KServe.

Documented uses

Serve trained models through REST or gRPC.
Run multiple models within one serving process.
Run inference in parallel across models using a pool of workers.
Group inference requests with adaptive batching.
Connect supported model frameworks through packaged runtimes or create a custom runtime.

Before you choose

Requirements & limits

The project is open source under the Apache License 2.0. Its README lists Python 3.9–3.12 as supported; it marks Python 3.7, 3.8, and 3.13 unsupported.

Both REST and gRPC interfaces implement the KFServing V2 Dataplane specification.
Optional model-framework runtimes may need to be installed separately; Scikit-Learn uses the mlserver-sklearn package in the documented example.
Companion libraries can use licenses different from Mlserver's Apache 2.0 license.
Kubernetes serving contexts named by the project include Seldon Core and KServe.

Check current requirements in the official documentation. Source-based guidance does not mean OpenViST has run or independently tested this project.

Source-based practical guide

Install Mlserver and the Scikit-Learn runtime

Set up the Mlserver package with the documented optional runtime for Scikit-Learn models.

Before you start

Python 3.9, 3.10, 3.11, or 3.12, listed as supported by the repository.
A Python package-installation environment.

Steps

Install the core package with the command below.
Documented command
```
pip install mlserver
```
Install the Scikit-Learn runtime with the command below.
Documented command
```
pip install mlserver-sklearn
```

Expected outcome

The core Mlserver package and its documented Scikit-Learn runtime package are installed.

Guide limitations

These steps cover package installation only; they do not configure or start model serving.
Software used alongside Mlserver may have different license terms.

Documentation 1 ↗

COMPARE OPTIONS

Compare options for Model inference and serving

These projects document a common task and share a project role. This is a source-based comparison, not an execution benchmark or a guarantee of interchangeability.

Project	Documented purpose	Platform	Access and requirements
Mlserver	Serve trained models through REST or gRPC.	Server	The project is open source under the Apache License 2.0. Its README lists Python 3.9–3.12 as supported; it marks Python 3.7, 3.8, and 3.13 unsupported.
Simple Tensorflow Serving	Host a TensorFlow SavedModel behind an inference endpoint.	Server, Cloud, Self-hosted	The project is open source under the Apache-2.0 license. The repository documents pip, source, Docker, Docker Compose and Kubernetes deployment paths.
Zhilight	Run inference for supported language models on local GPU infrastructure.	Linux, Server, Self-hosted	The project is a public GitHub repository licensed under Apache-2.0. The README documents building or installing from source and provides a Docker image.

Evidence and transparency

Sources & review

Official repository ↗

Source record details

Source availability: Active
Sources checked: 5 November 2026
Original website title: SeldonIO/MLServer · Official homepage title · retained in its source language

How we review sources Suggest a correction

Mlserver