OpenViSTAI · Agents · Blockchain · Crypto
Submit a project
Menu
AI Infrastructure & Serving

Kvpress

Kvpress is a Python library that implements training-free methods for compressing the key-value caches used by large language models. It integrates with Hugging Face Transformers for inference and includes methods and benchmarks aimed at developers and researchers working on cache efficiency and long-context processing.

Explore the practical guide ↓

Independent OpenViST project profile, based on linked sources.

Kvpress — OpenViST project profile in AI Infrastructure & Serving.
OpenViST project illustration. See the linked sources for product details.
Understand the project

Practical overview

When it fits

Use Kvpress when you are building or studying Transformers-based LLM inference and want to apply or compare training-free KV-cache compression, especially for context-heavy workloads.

Who it is for

  • Developers adding KV-cache compression to Transformers-based language-model inference.
  • Researchers comparing cache compression approaches for LLM workloads.

Documented uses

  • Apply a compression method to context tokens during model prefilling.
  • Use a compressed context for text generation and questions about that context.
  • Configure experimental periodic cache compression during token generation.
  • Compare compression approaches and investigate memory or timing measurements using the linked notebooks.
Before you choose

Requirements & limits

The project is a public GitHub repository identified as open source under the Apache-2.0 license. The README documents installation through pip or a local uv-managed checkout.

  • The custom kv-press-text-generation pipeline handles chat templates and tokenization, according to the README.
  • Prefill compression is the default; decoding compression is separately described as experimental.
  • DecodingPress uses a target cache size and compression interval rather than the prefill press's compression-ratio setting.
  • The README describes multi-GPU inference through Accelerate and links notebooks for memory and timing measurements.

Check current requirements in the official documentation. Source-based guidance does not mean OpenViST has run or independently tested this project.

Source-based practical guide

Install Kvpress and prepare a Transformers pipeline

Set up the documented text-generation pipeline for a model and a KV-cache compression method.

Before you start

  • Python environment with pip, or a local development environment with uv.
  • A Transformers model identifier; the example uses Qwen/Qwen3-8B.
  • The README's pipeline example specifies device_map="auto" and dtype="auto".

Steps

  1. Install the package with the command below, or clone the repository and run the command below from its directory.

    Documented command
    pip install kvpress
    Documented command
    uv sync

  2. Import pipeline from Transformers and a press such as ExpectedAttentionPress from kvpress.

  3. Create a pipeline with the kv-press-text-generation task and the chosen model identifier.

  4. Set a context, optionally provide a question, and create the press with a compression_ratio.

Expected outcome

A configured pipeline call can apply the selected press to context tokens and return an answer for a question about that context.

Guide limitations

  • The example describes API usage; it does not establish performance or hardware compatibility for a particular model.
  • Optional evaluation and flash-attention dependencies are installed separately using the documented uv extras.
Documentation 1 ↗
Source-based practical guide

Configure experimental decoding-time compression

Prepare a pipeline using the documented DecodingPress wrapper to compress the cache periodically during generation.

Before you start

  • Kvpress and Transformers installed.
  • A Transformers model identifier; the example uses meta-llama/Llama-3.1-8B-Instruct.
  • A ScorerPress, such as KnormPress, as the wrapper's base press.

Steps

  1. Initialize the kv-press-text-generation pipeline with the model and the documented device and model options.

  2. Create DecodingPress with a base ScorerPress, a compression interval, and a target cache size.

  3. Pass the wrapper as the press argument when calling the pipeline with context and a question.

Expected outcome

The configured wrapper is set to compress the cache at intervals during token generation toward the chosen target size.

Guide limitations

  • Decoding compression is identified as experimental.
  • Not all presses are compatible; the documentation supports ScorerPresses as base presses.
  • The example's CUDA device and flash-attention option are example configuration, not a guarantee of availability or compatibility.
Documentation 1 ↗
Evidence and transparency

Sources & review

Official repository ↗
Source record details
Source availability
Active
Sources checked
13 October 2026
Original website title
NVIDIA/kvpress · Official homepage title · retained in its source language
How we review sourcesSuggest a correction