OpenViSTAI · Agents · Blockchain · Crypto
Submit a project
Menu
AI Infrastructure & Serving

Openllm

OpenLLM is a Python-based tool for serving open-source language models as OpenAI-compatible API endpoints. It includes command-line model serving and chat, plus a browser chat interface for a launched server.

Explore the practical guide ↓

Independent OpenViST project profile, based on linked sources.

Openllm — OpenViST project profile in AI Infrastructure & Serving.
OpenViST project illustration. See the linked sources for product details.
Understand the project

Practical overview

When it fits

Use OpenLLM when you want to run an open-source model yourself and expose it through an API compatible with OpenAI-style clients.

Who it is for

  • Developers building applications that need an API connection to a self-hosted open-source language model.
  • Teams setting up local or cloud inference services and able to meet the selected model’s compute requirements.

Documented uses

  • Serve a supported open-source model behind an OpenAI-compatible API.
  • Connect compatible client libraries to a locally launched model server.
  • Try a served model through the browser chat UI or the command line.
Before you choose

Requirements & limits

  • GPU requirements depend on the model; the repository lists examples ranging from 12 GB to several 80 GB GPUs.
  • OpenLLM does not store model weights.
  • Gated models require Hugging Face access and a token set as HF_TOKEN.
  • The API key for client authentication is described as optional.
  • The repository describes Docker, Kubernetes and BentoCloud deployment workflows in addition to its local server example.

Check current requirements in the official documentation. Source-based guidance does not mean OpenViST has run or independently tested this project.

Source-based practical guide

Start a local server for a supported model

Launch the documented Llama 3.2 1B model as a local OpenAI-compatible API service.

Before you start

  • OpenLLM installed with pip.
  • Compute that meets the listed requirement for the selected model; the repository lists 24 GB GPU memory for llama3.2:1b.
  • For a gated model, a Hugging Face token and access to that model.

Steps

  1. Install OpenLLM with the command below.

    Documented command
    pip install openllm

  2. For a gated model, request access and set the Hugging Face token as the HF_TOKEN environment variable.

  3. Run openllm serve llama3.2:1b.

Expected outcome

The local server is documented at http://localhost:3000 and exposes OpenAI-compatible APIs.

Guide limitations

  • OpenLLM does not store model weights.
  • Model compute requirements vary; the cited GPU requirement is specific to the selected model.
  • The repository notes that model names can differ depending on the client tool.
Documentation 1 ↗
Source-based practical guide

Connect an OpenAI Python client to the local server

Send a chat-completion request to an OpenLLM server already running locally.

Before you start

  • A running OpenLLM server at http://localhost:3000.
  • The OpenAI Python client.

Steps

  1. Set the client base URL to http://localhost:3000/v1.

  2. Use the model name expected by the client and server.

  3. Submit a chat-completion request; the README example enables streaming.

Expected outcome

The client sends a chat request to the local OpenLLM API and can receive streamed response content.

Guide limitations

  • This recipe covers client use after the server is running; it does not install the client.
  • The API key is optional for OpenLLM client authentication, though the README example supplies a placeholder value.
  • The repository notes that model names may vary by client.
Documentation 1 ↗
COMPARE OPTIONS

Compare options for Model inference and serving

These projects document a common task and share a project role. This is a source-based comparison, not an execution benchmark or a guarantee of interchangeability.

ProjectDocumented purposePlatformAccess and requirements
OpenllmServe a supported open-source model behind an OpenAI-compatible API.Cloud, Self-hosted, Server
MorphingdbImport a libtorch model and create predictions from values in a PostgreSQL table.Docker, Linux, Self-hosted, ServerThe README documents a Docker image and a build-from-repository option, followed by connecting to PostgreSQL and enabling the pgdl extension.
Gateway — pyadavSend chat-completion requests to a provider through a common server endpoint.ServerThe repository is public and identifies the project as Apache-2.0 licensed. README installation instructions include binaries, platform package managers and Docker.
Evidence and transparency

Sources & review

Official repository ↗
Source record details
Source availability
Active
Sources checked
7 October 2026
Original website title
bentoml/OpenLLM · Official homepage title · retained in its source language
How we review sourcesSuggest a correction