OpenViSTAI · Agents · Blockchain · Crypto
Submit a project
Menu
AI Infrastructure & Serving

Paddler

Paddler is an open-source LLM and VLM inference serving platform and load balancer for self-managed infrastructure. A balancer distributes incoming requests to dynamically added agents, whose slots generate tokens and embeddings using a built-in llama.cpp engine. It includes an optional web admin panel for management, monitoring and testing.

Explore the practical guide ↓

Independent OpenViST project profile, based on linked sources.

Paddler — OpenViST project profile in AI Infrastructure & Serving.
OpenViST project illustration. See the linked sources for product details.
Understand the project

Practical overview

When it fits

Use Paddler when you need a self-managed inference serving layer that connects applications to LLM or VLM capacity across CPU or GPU infrastructure.

Who it is for

  • Product teams that need to serve LLM inference or embeddings in their features.
  • DevOps and LLMOps teams deploying and operating model inference.
  • Organizations that want to run inference on infrastructure they manage.

Documented uses

  • Serve tokens and embeddings to applications through a self-hosted inference service.
  • Distribute model requests across agents and their inference slots.
  • Add agents dynamically to integrate with autoscaling tools.
  • Manage, monitor and test a Paddler setup through its web admin panel.
Before you choose

Requirements & limits

The project is open source under the Apache-2.0 license. The README describes a self-contained binary available through GitHub releases or a source build.

  • The architecture has two deployable components: a balancer and agents.
  • The inference engine is based on llama.cpp; agents use Paddler's own slot implementation with per-slot context and KV cache.
  • The README lists request buffering, dynamic model swapping and observability metrics.
  • The web admin panel is optional in the documented balancer command.

Check current requirements in the official documentation. Source-based guidance does not mean OpenViST has run or independently tested this project.

Source-based practical guide

Install Paddler from a release or source

Make the Paddler binary available on your system.

Before you start

  • For a source build: Rust and Node installed.
  • For a CUDA source build: CUDA toolkit installed and the nvcc compiler available in PATH.

Steps

  1. Obtain the latest release binary from the GitHub releases page, or clone the Paddler repository to build from source.

  2. For a source build, from the repository root, run the hardware-specific build command: CPU: make; CUDA: the command below; Metal: the command below; Vulkan: the command below.

    Documented command
    make cuda
    Documented command
    make metal
    Documented command
    make vulkan

  3. Use the resulting binary at target/release/paddler for CPU, target/cuda/release/paddler for CUDA, target/metal/release/paddler for Metal, or target/vulkan/release/paddler for Vulkan.

  4. Optionally move the binary to a directory on your PATH to make it available globally; the appropriate location depends on the operating system.

Expected outcome

A Paddler binary is available on the system. Run `paddler --help` to list available commands and options.

Guide limitations

  • The source-build instructions list MSRV 1.95.0.
  • Global binary installation is system-specific.
  • CUDA builds require the CUDA toolkit and nvcc in PATH.
Documentation 1 ↗Documentation 2 ↗
COMPARE OPTIONS

Compare options for Model inference and serving

These projects document a common task and share a project role. This is a source-based comparison, not an execution benchmark or a guarantee of interchangeability.

ProjectDocumented purposePlatformAccess and requirements
PaddlerServe tokens and embeddings to applications through a self-hosted inference service.Self-hosted, ServerThe project is open source under the Apache-2.0 license. The README describes a self-contained binary available through GitHub releases or a source build.
MorphingdbImport a libtorch model and create predictions from values in a PostgreSQL table.Docker, Linux, Self-hosted, ServerThe README documents a Docker image and a build-from-repository option, followed by connecting to PostgreSQL and enabling the pgdl extension.
Gateway — pyadavSend chat-completion requests to a provider through a common server endpoint.ServerThe repository is public and identifies the project as Apache-2.0 licensed. README installation instructions include binaries, platform package managers and Docker.
Evidence and transparency

Sources & review

Official repository ↗Guide documentation ↗
Source record details
Source availability
Active
Sources checked
11 October 2026
Original website title
intentee/paddler · Official homepage title · retained in its source language
How we review sourcesSuggest a correction