OpenViSTAI · Agents · Blockchain · Crypto
Submit a project
Menu
AI Infrastructure & Serving

Kvcached

Kvcached is a Python library and daemon that virtualizes key-value cache memory for LLM serving and training on shared GPUs. It separates virtual cache addressing from physical GPU-memory allocation, allowing physical memory to be mapped on demand as cache is used.

Explore the practical guide ↓

Independent OpenViST project profile, based on linked sources.

Kvcached — OpenViST project profile in AI Infrastructure & Serving.
OpenViST project illustration. See the linked sources for product details.
Understand the project

Practical overview

When it fits

Consider Kvcached when serving workloads on shared GPUs need demand-driven KV-cache allocation and the deployment uses SGLang or vLLM.

Who it is for

  • Teams operating LLM serving or training workloads on shared GPUs.
  • Engine integrators and infrastructure developers working with SGLang or vLLM.

Documented uses

  • Allocate and reclaim KV-cache memory dynamically as workload demand changes.
  • Let multiple LLMs share GPU memory without relying on rigid cache-memory partitions.
  • Apply memory limits and configure bounded prefix caching in supported serving engines.
  • Route requests to target models and put idle models to sleep, using the documented frontend features.
Before you choose

Requirements & limits

The project repository is public and lists the Apache-2.0 license. Installation instructions are provided for PyPI, source, and Docker.

  • The documented runtime prerequisites include Python 3.10–3.13, PyTorch 2.10 or newer, and SGLang or vLLM.
  • The README lists minimum engine versions of SGLang 0.5.11 and vLLM 0.17.0, and gives tested versions of 0.5.20 and 0.30.0.
  • Kvcached addresses KV-cache memory allocation; the documented engine integrations remain part of the serving setup.
  • The repository describes memory control, request routing, sleep mode, and prefix caching alongside elastic cache allocation.

Check current requirements in the official documentation. Source-based guidance does not mean OpenViST has run or independently tested this project.

Source-based practical guide

Install Kvcached from PyPI

Add Kvcached to an existing SGLang or vLLM environment.

Before you start

  • Python 3.10–3.13
  • PyTorch 2.10 or newer
  • An existing SGLang or vLLM environment

Steps

  1. Install the package with the command below.

    Documented command
    pip install kvcached --no-build-isolation

  2. Enable Kvcached by setting ENABLE_KVCACHED=true and KVCACHED_AUTOPATCH=1 in the environment.

Expected outcome

Kvcached is installed and enabled for the serving-engine environment; the README describes using the engine's benchmark scripts to exercise it.

Guide limitations

  • The README identifies SGLang 0.5.20 and vLLM 0.30.0 as tested versions; individual model, GPU, and deployment compatibility is not guaranteed.
  • Benchmark execution requires the relevant serving engine and its model setup.
Documentation 1 ↗
COMPARE OPTIONS

Compare options for Model inference and serving

These projects document a common task and share a project role. This is a source-based comparison, not an execution benchmark or a guarantee of interchangeability.

ProjectDocumented purposePlatformAccess and requirements
KvcachedAllocate and reclaim KV-cache memory dynamically as workload demand changes.Linux, ServerThe project repository is public and lists the Apache-2.0 license. Installation instructions are provided for PyPI, source, and Docker.
Cloud AI SDKAdapt and run documented model recipes for computer-vision and NLP inference.Cloud, ServerThe GitHub repository is public. Its public visibility does not establish license terms.
TurboquantReplace a model’s existing inference cache with a compressed cache.Cloud, Linux, macOS, Self-hostedThe project describes itself as open source. The README documents installation through pip or from a source checkout.
Evidence and transparency

Sources & review

Official repository ↗
Source record details
Source availability
Active
Sources checked
11 October 2026
Original website title
ovg-project/kvcached · Official homepage title · retained in its source language
How we review sourcesSuggest a correction