OpenViSTAI · Agents · Blockchain · Crypto
Submit a project
Menu
AI Infrastructure & Serving

Willow Inference Server

Willow Inference Server is open-source Python server software for self-hosting speech recognition, text-to-speech and large language model inference. It offers WebRTC, REST and WebSocket interfaces, uses CTranslate2 for Whisper support, and primarily targets CUDA while also noting CPU-only operation.

Explore the practical guide ↓

Independent OpenViST project profile, based on linked sources.

Willow Inference Server — OpenViST project profile in AI Infrastructure & Serving.
OpenViST project illustration. See the linked sources for product details.
Understand the project

Practical overview

When it fits

Use WIS when you want to install and operate a speech-and-language inference server yourself, and your clients can connect through WebRTC, REST or WebSocket. Its README gives particular emphasis to latency-sensitive speech recognition and long-lived WebRTC audio sessions.

Who it is for

  • Operators who want to run speech or language inference on infrastructure they manage.
  • Developers connecting browser, WebRTC, device or application clients to server-side speech recognition.
  • Builders seeking a self-hosted text-to-speech service, including custom TTS voices.
  • Willow users who want to configure the client to use a WIS server.

Documented uses

  • Transcribe speech through streaming WebRTC sessions or posted audio files.
  • Provide speech output for assistant-style applications using TTS.
  • Expose language-model inference through the server's supported interfaces.
  • Configure a Willow client to use a WIS host.
Before you choose

Requirements & limits

The repository is public and identifies the project as open source under the Apache-2.0 license. The documented deployment is self-hosted.

  • CUDA is the primary hardware target; CPU-only operation is also noted.
  • The README gives a GTX 1060 with 6GB VRAM as its minimum supported hardware for ASR and TTS, and separately says ASR/STT can require 4GB VRAM in its Windows-with-WSL discussion.
  • WebRTC is suited to streaming audio and long-running sessions; REST and WebSocket are also supported.
  • The installation instructions require dependency setup, a build step, TLS configuration and starting the service.
  • The project reports that it detects available CUDA VRAM and platform support and may optimize or disable functionality accordingly.

Check current requirements in the official documentation. Source-based guidance does not mean OpenViST has run or independently tested this project.

Source-based practical guide

Install and start Willow Inference Server

Set up WIS and start its server process using the repository's documented installation sequence.

Before you start

  • A Linux environment; the README also describes Windows use through WSL.
  • For CUDA support, NVIDIA drivers for supported hardware; the README recommends driver version 530.
  • For the documented NVIDIA-container setup, nvidia-container-toolkit is required rather than nvidia-docker.

Steps

  1. Clone the repository and change into its directory.

  2. Install the documented dependencies, using the repository's Ubuntu script or the listed Arch Linux packages as applicable.

  3. Run ./utils.sh install.

  4. Generate a self-signed TLS certificate with ./utils.sh gen-cert [your hostname], or place a certificate and key at nginx/cert.pem and nginx/key.pem.

  5. Start WIS with ./utils.sh run.

Expected outcome

The documented setup sequence reaches a running WIS service; the README lists the REST documentation at `/api/docs` and WebRTC demo client at `/rtc` on the host.

Guide limitations

  • The README describes WIS as an early, rapidly advancing project and says installation involves a build requirement.
  • CUDA is the primary target; CPU-only operation is also noted, but the setup steps above do not configure or verify a particular workload.
Documentation 1 ↗
COMPARE OPTIONS

Compare options for Model inference and serving

These projects document a common task and share a project role. This is a source-based comparison, not an execution benchmark or a guarantee of interchangeability.

ProjectDocumented purposePlatformAccess and requirements
Willow Inference ServerTranscribe speech through streaming WebRTC sessions or posted audio files.Linux, Self-hosted, ServerThe repository is public and identifies the project as open source under the Apache-2.0 license. The documented deployment is self-hosted.
MorphingdbImport a libtorch model and create predictions from values in a PostgreSQL table.Docker, Linux, Self-hosted, ServerThe README documents a Docker image and a build-from-repository option, followed by connecting to PostgreSQL and enabling the pgdl extension.
Gateway — pyadavSend chat-completion requests to a provider through a common server endpoint.ServerThe repository is public and identifies the project as Apache-2.0 licensed. README installation instructions include binaries, platform package managers and Docker.
Evidence and transparency

Sources & review

Official repository ↗
Source record details
Source availability
Active
Sources checked
14 October 2026
Original website title
toverainc/willow-inference-server · Official homepage title · retained in its source language
How we review sourcesSuggest a correction