OpenViSTAI · Agents · Blockchain · Crypto
Submit a project
Menu
AI Infrastructure & Serving

Kvpacket

Kvpacket is a Python library for creating and reusing large language model key-value caches. It uses trainable soft-token header and trailer adapters to make independently precomputed document caches reusable across contexts, then combines them at serving time with rotary-position realignment. The repository includes multi-document tasks and controlled agentic-flow examples.

Explore the practical guide ↓

Independent OpenViST project profile, based on linked sources.

Kvpacket — OpenViST project profile in AI Infrastructure & Serving.
OpenViST project illustration. See the linked sources for product details.
Understand the project

Practical overview

When it fits

Use Kvpacket when developing an LLM inference or agent workflow that repeatedly relies on document context and can benefit from preparing reusable KV state offline.

Who it is for

  • Developers building LLM inference pipelines that reuse document context.
  • Teams implementing multi-document or agentic workflows with cache-reuse requirements.
  • Researchers exploring the library's separate paper experiments and evaluation materials.

Documented uses

  • Precompute document KV caches for later use in inference.
  • Combine reusable document caches at serving time without recomputing document tokens.
  • Build multi-document tasks and controlled agentic or skill-based flows using the library.
Before you choose

Requirements & limits

The repository is public and lists an MIT license. The package instructions specify Python 3.12 or newer and an appropriate PyTorch build for the compute platform.

  • Inference sessions require an explicit compute backend; the repository describes a Hugging Face backend.
  • Source preprocessing is separate from the compute backend. The documented text encoder handles raw text and local text files; other modalities require a framework-side source encoder.
  • Compression, HTTP/SSE transport and MCP support are optional package integrations.
  • The main branch is the maintained library; the paper-artifact branch preserves research experiments and related scripts.

Check current requirements in the official documentation. Source-based guidance does not mean OpenViST has run or independently tested this project.

Source-based practical guide

Install Kvpacket and review the development environment

Install the library and follow the repository's documented development-environment setup.

Before you start

  • Python 3.12 or newer
  • A PyTorch build appropriate for the compute platform before loading a model
  • For development from a repository clone, uv

Steps

  1. Install the package with the command below.

    Documented command
    pip install kvpacket

  2. For a clone-based development environment, run the command below.

    Documented command
    uv sync

  3. Run the command below to print the installed PyTorch version and CUDA availability.

    Documented command
    uv run python -c "import torch; print(torch.__version__, torch.cuda.is_available())"

Expected outcome

The package is installed, or the clone's locked development environment is set up and the PyTorch version and CUDA availability are printed.

Guide limitations

  • This covers installation and an environment check, not model loading or inference.
  • The appropriate PyTorch build depends on the compute platform.
Documentation 1 ↗
Evidence and transparency

Sources & review

Official repository ↗
Source record details
Source availability
Active
Sources checked
14 October 2026
Original website title
ChuangtaoChen-TUM/KVPacket · Official homepage title · retained in its source language
How we review sourcesSuggest a correction