OpenViSTAI · Agents · Blockchain · Crypto
Submit a project
Menu
AI Infrastructure & Serving

Shimmy

Shimmy is a Rust-based, self-hosted inference server that runs GGUF language models locally through a WebGPU engine and exposes an OpenAI-compatible API.

Explore the practical guide ↓

Independent OpenViST project profile, based on linked sources.

Shimmy — OpenViST project profile in AI Infrastructure & Serving.
OpenViST project illustration. See the linked sources for product details.
Understand the project

Practical overview

When it fits

Use Shimmy when the target task is serving local GGUF inference through an OpenAI-compatible API, especially when an application already uses OpenAI SDKs or tools.

Who it is for

  • Developers connecting local language-model inference to applications that use OpenAI-compatible interfaces.
  • Teams or individual builders who want to serve GGUF models from a self-hosted inference process.

Documented uses

  • Run a local GGUF model behind an inference API.
  • Provide chat-completion, text-completion, streaming, and model endpoints to compatible tools and applications.
  • Choose among documented supported model-and-quantization combinations for a local serving setup.
Before you choose

Requirements & limits

The repository is public and identifies Apache-2.0 as the license. The documented install path uses Cargo.

  • Shimmy is the server; its README identifies Airframe as the WebGPU inference engine underneath it.
  • The documented quick-start path requires a local GGUF model file and uses Cargo to install the CLI.
  • The repository documents model- and quantization-specific certification; architecture recognition alone is not certification.
  • SafeTensors loading is described, but full Airframe-native inference for SafeTensors is identified as roadmap work.
  • The available evidence does not specify supported operating systems or deployment requirements.

Check current requirements in the official documentation. Source-based guidance does not mean OpenViST has run or independently tested this project.

Source-based practical guide

Install and start Shimmy with a local GGUF model

Install the CLI, launch the server with a model file, and send a chat-completions request to its local API.

Before you start

  • Rust/Cargo installation path, as shown by the documented cargo install command.
  • A GGUF model file at an absolute local path.

Steps

  1. Install Shimmy with the command below.

    Documented command
    cargo install shimmy

  2. Start the server with shimmy serve --model-path /absolute/path/to/model.gguf --bind 127.0.0.1:11435.

  3. In another terminal, use the documented curl request to http://127.0.0.1:11435/v1/chat/completions with a model name, user message, and max_tokens value.

Expected outcome

A local Shimmy server is started with the supplied model path, and the example request targets its chat-completions endpoint.

Guide limitations

  • The example assumes a model file is already available; the README points to the quick-start guide for model acquisition, GPU and VRAM sizing, and platform-specific build details.
  • The README's example request is not evidence of a runtime test or compatibility guarantee.
Documentation 1 ↗
COMPARE OPTIONS

Compare options for Model inference and serving

These projects document a common task and share a project role. This is a source-based comparison, not an execution benchmark or a guarantee of interchangeability.

ProjectDocumented purposePlatformAccess and requirements
ShimmyRun a local GGUF model behind an inference API.CLI, Self-hostedThe repository is public and identifies Apache-2.0 as the license. The documented install path uses Cargo.
MorphingdbImport a libtorch model and create predictions from values in a PostgreSQL table.Docker, Linux, Self-hosted, ServerThe README documents a Docker image and a build-from-repository option, followed by connecting to PostgreSQL and enabling the pgdl extension.
Gateway — pyadavSend chat-completion requests to a provider through a common server endpoint.ServerThe repository is public and identifies the project as Apache-2.0 licensed. README installation instructions include binaries, platform package managers and Docker.
Evidence and transparency

Sources & review

Official repository ↗
Source record details
Source availability
Active
Sources checked
10 October 2026
Original website title
Michael-A-Kuykendall/shimmy · Official homepage title · retained in its source language
How we review sourcesSuggest a correction