OpenViSTAI · Agents · Blockchain · Crypto
Submit a project
Menu
AI Infrastructure & Serving

Kvcache Factory

Kvcache Factory is a Python research toolkit that places KV-cache compression, retrieval, merging and quantization methods alongside a full-cache baseline under a shared evaluation interface for long-context autoregressive language-model inference.

Explore the practical guide ↓

Independent OpenViST project profile, based on linked sources.

Kvcache Factory — OpenViST project profile in AI Infrastructure & Serving.
OpenViST project illustration. See the linked sources for product details.
Understand the project

Practical overview

When it fits

Use it when a research or engineering workflow needs to compare or prototype KV-cache strategies for long-context inference, rather than a hosted model-serving endpoint.

Who it is for

  • Researchers comparing KV-cache methods for long-context language-model inference.
  • Developers prototyping cache compression, retrieval, merging or quantization approaches.
  • Practitioners evaluating cache strategies with LongBench, RULER or needle-in-a-haystack runners.

Documented uses

  • Compare multiple cache approaches against a full-cache baseline under a shared evaluation interface.
  • Configure long-context evaluations with selectable methods, model paths, datasets and cache budgets.
  • Explore quantized cache paths using KIVI, KVQuant or GEAR.
  • Inspect or reproduce attention visualizations with the provided notebook and visualization utilities.
Before you choose

Requirements & limits

The repository is public, open source and MIT-licensed. It is a self-hosted Linux Python library; setup uses pip and the repository’s requirements file.

  • The toolkit groups many cache methods, but supported runner and model coverage varies; newer methods may be narrower.
  • FlashAttention v2 experiments require flash-attn, while SDPA or eager attention can be used without it.
  • The optional MInference integration has separate dependencies.
  • HeadInfer is documented as lossless CPU offloading rather than cache compression, and requires FlashAttention v2.

Check current requirements in the official documentation. Source-based guidance does not mean OpenViST has run or independently tested this project.

Source-based practical guide

Set up Kvcache Factory for LongBench evaluation

Install the repository dependencies and configure the project path for running the documented LongBench workflow.

Before you start

  • A Linux self-hosted environment with Python and pip.
  • PyTorch and Transformers; the project requirements specify transformers==4.44.2.
  • A local or Hugging Face model path for the later evaluation stage.

Steps

  1. Clone https://github.com/Zefan-Cai/KVCache-Factory.git and change into the KVCache-Factory directory.

  2. Install the project dependencies with the command below.

    Documented command
    pip install -r requirements.txt

  3. Set PYTHONPATH to include the repository directory, as shown in the README.

  4. For FlashAttention v2 experiments, install flash-attn after torch; alternatively, the README lists SDPA and eager attention paths.

Expected outcome

The repository dependencies are installed and its directory is included in Python’s import path, ready for a separately configured evaluation run.

Guide limitations

  • FlashAttention v2 requires flash-attn; it is optional for SDPA or eager attention.
  • The optional MInference integration requires installing requirements-minference.txt separately.
  • The documented setup steps do not themselves run an evaluation.
Documentation 1 ↗
Source-based practical guide

Configure a LongBench run with PyramidKV

Use the documented LongBench command structure to select PyramidKV, a model path, a cache budget and an attention implementation.

Before you start

  • Kvcache Factory installed and its repository directory included in PYTHONPATH.
  • A local or Hugging Face model path.
  • An attention implementation supported by the chosen method and environment.

Steps

  1. Set CUDA_VISIBLE_DEVICES to the GPU device selection used for the run.

  2. Invoke run_longbench.py with --method pyramidkv and set --model_path to the model location.

  3. Set the cache budget with --max_capacity_prompts, choose an attention implementation, and specify a results directory with --save_dir.

  4. Optionally choose LongBench datasets with the --datasets argument; otherwise the documented default is the full 16-dataset list.

Expected outcome

The command is configured to run a LongBench evaluation using PyramidKV and save results in the specified directory.

Guide limitations

  • The README’s example uses FlashAttention v2; SDPA or eager are also listed attention choices, but method-specific constraints apply.
  • Some newer methods have narrower runner or model coverage; check runner argument choices before large jobs.
  • The guide describes the documented command configuration, not an independently executed or validated run.
Documentation 1 ↗
COMPARE OPTIONS

Compare options for Model inference and serving

These projects document a common task and share a project role. This is a source-based comparison, not an execution benchmark or a guarantee of interchangeability.

ProjectDocumented purposePlatformAccess and requirements
Kvcache FactoryCompare multiple cache approaches against a full-cache baseline under a shared evaluation interface.Linux, Self-hostedThe repository is public, open source and MIT-licensed. It is a self-hosted Linux Python library; setup uses pip and the repository’s requirements file.
Tiny TensorrtBuild a TensorRT engine from an ONNX model.Linux, Server, Self-hosted
PPLNNRun an ONNX model through an inference engine.The repository is public, identifies an Apache-2.0 license and provides source-build instructions.
Evidence and transparency

Sources & review

Official repository ↗
Source record details
Source availability
Active
Sources checked
3 November 2026
Original website title
Zefan-Cai/KVCache-Factory · Official homepage title · retained in its source language
How we review sourcesSuggest a correction