OpenViSTAI · Agents · Blockchain · Crypto
Submit a project
Menu
AI Infrastructure & Serving

Rungpt

RunGPT is an open-source Python framework for loading and serving large language models, with a repository description that also refers to large multimodal models. It offers a model-loading interface and a command-line serving workflow that exposes HTTP and gRPC endpoints, plus an option to distribute a model across multiple GPUs.

Explore the practical guide ↓

Independent OpenViST project profile, based on linked sources.

Rungpt — OpenViST project profile in AI Infrastructure & Serving.
OpenViST project illustration. See the linked sources for product details.
Understand the project

Practical overview

When it fits

Use RunGPT when you need a Python-based serving layer for a model you will load on your own infrastructure, particularly when an application needs to send generation requests to an HTTP or gRPC endpoint.

Who it is for

  • Developers building a self-hosted language-model inference service.
  • Teams connecting model generation to an application through HTTP or gRPC.
  • Practitioners who need to partition a model across multiple GPUs.

Documented uses

  • Load a model and generate text from prompts using Python.
  • Start a local model-serving process with HTTP and gRPC interfaces.
  • Send generation requests to a running endpoint.
  • Distribute a model across available GPUs with the documented device-map option.
Before you choose

Requirements & limits

The repository describes RunGPT as open source and lists an Apache-2.0 license. Installation is documented through pip.

  • The README provides concrete text-generation examples; its broader multimodal serving description is not accompanied here by a specific multimodal model example.
  • The serving command documents HTTP on port 52000 and gRPC on port 51000.
  • The multi-GPU device_map option is supported by the Accelerate package.
  • The quickstart uses CUDA and fp16; the documentation notes a trade-off between fp16 memory use and text-generation accuracy.

Check current requirements in the official documentation. Source-based guidance does not mean OpenViST has run or independently tested this project.

Source-based practical guide

Start a model server and send a text-generation request

Load the documented StableLM example and submit a prompt to RunGPT's local HTTP server.

Before you start

  • Install RunGPT with pip.
  • Use the example model stabilityai/stablelm-tuned-alpha-3b.
  • The example specifies CUDA and fp16; the README notes fp32 can also be used.

Steps

  1. Install the package with the command below.

    Documented command
    pip install rungpt

  2. Start the server with rungpt serve stabilityai/stablelm-tuned-alpha-3b --precision fp16 --device_map balanced.

  3. Send a POST request with a prompt and generation settings to http://localhost:52000/generate.

Expected outcome

A local HTTP server accepts a text-generation request for the selected model.

Guide limitations

  • The documented server also listens for gRPC on port 51000; the HTTP example uses port 52000.
  • The README says first-time model download and loading can take several minutes.
  • The fp16 example uses less memory but may reduce accuracy on text generation; fp32 is offered as an alternative.
  • The guide follows repository documentation and is not a report of a run or compatibility test.
Documentation 1 ↗
COMPARE OPTIONS

Compare options for Model inference and serving

These projects document a common task and share a project role. This is a source-based comparison, not an execution benchmark or a guarantee of interchangeability.

ProjectDocumented purposePlatformAccess and requirements
RungptLoad a model and generate text from prompts using Python.Cloud, Self-hosted, ServerThe repository describes RunGPT as open source and lists an Apache-2.0 license. Installation is documented through pip.
MorphingdbImport a libtorch model and create predictions from values in a PostgreSQL table.Docker, Linux, Self-hosted, ServerThe README documents a Docker image and a build-from-repository option, followed by connecting to PostgreSQL and enabling the pgdl extension.
Gateway — pyadavSend chat-completion requests to a provider through a common server endpoint.ServerThe repository is public and identifies the project as Apache-2.0 licensed. README installation instructions include binaries, platform package managers and Docker.
Evidence and transparency

Sources & review

Official repository ↗
Source record details
Source availability
Active
Sources checked
17 October 2026
Original website title
jina-ai/rungpt · Official homepage title · retained in its source language
How we review sourcesSuggest a correction