OpenViSTAI · Agents · Blockchain · Crypto
Submit a project
Menu
AI Infrastructure & Serving

Rocketkv

Rocketkv is an open-source Python implementation of a training-free method for compressing transformer KV caches during large language model decoding. It first evicts lower-importance tokens coarsely, then applies hybrid sparse attention for finer token selection. The repository includes scripts for evaluating the approach on long-context benchmarks.

Explore the practical guide ↓

Independent OpenViST project profile, based on linked sources.

Rocketkv — OpenViST project profile in AI Infrastructure & Serving.
OpenViST project illustration. See the linked sources for product details.
Understand the project

Practical overview

When it fits

Use Rocketkv when you want to evaluate or study two-stage KV-cache compression during long-context LLM decoding, rather than obtain a hosted inference endpoint.

Who it is for

  • Researchers evaluating long-context LLM inference methods.
  • Engineers investigating KV-cache memory and decoding performance trade-offs.

Documented uses

  • Run RocketKV on supported long-context benchmark and model combinations.
  • Compare compressed decoding with full-KV or Exact-TopK evaluation modes.
  • Examine how token budgets relate to reported inference performance and memory use.
Before you choose

Requirements & limits

The project is identified as open source and its public repository is active. Model downloads for the documented accuracy experiments require configuring a Hugging Face access token.

  • The method is training-free and targets the decode phase.
  • Its two stages combine coarse cache-token eviction with fine-grained hybrid sparse attention.
  • The README reports results on an NVIDIA A100; performance on other hardware or workloads is not specified.
  • The repository setup requires FlashAttention 2.6.3 and experiment-specific model and dataset preparation.

Check current requirements in the official documentation. Source-based guidance does not mean OpenViST has run or independently tested this project.

Source-based practical guide

Run a documented LongBench comparison

Use the repository's LongBench scripts to compare RocketKV with the full-KV baseline or Exact-TopK for a supported model.

Before you start

  • Python environment with dependencies installed using the repository requirements file.
  • FlashAttention 2.6.3.
  • A supported model and LongBench data; the README gives a download script for LongBench.
  • Hugging Face access token configured as described in the repository for model downloads.

Steps

  1. Install dependencies with the command below.

    Documented command
    pip install -r requirements.txt

  2. Install FlashAttention with the command below.

    Documented command
    pip install flash-attn==2.6.3

  3. Prepare LongBench data with the command below and prepare the model as described in the README.

    Documented command
    python scripts/dataset_prep/download_longbench.py

  4. Run the relevant scripts/longbench/ model script with rocket, a results directory and a token budget; substitute baseline or topk for the documented comparison methods.

Expected outcome

The repository's LongBench evaluation script is set to produce results for the selected method and token budget.

Guide limitations

  • The documented scripts cover Llama 3.1 8B Instruct, Mistral 7B Instruct v0.2 and LongChat 7B v1.5 32k.
  • The README does not describe a hosted service or API for this workflow.
Documentation 1 ↗
Evidence and transparency

Sources & review

Official repository ↗
Source record details
Source availability
Active
Sources checked
18 October 2026
Original website title
NVlabs/RocketKV · Official homepage title · retained in its source language
How we review sourcesSuggest a correction