OpenViSTAI · Agents · Blockchain · Crypto
Submit a project
Menu
AI Infrastructure & Serving

Kivi

Kivi is an open-source Python implementation of tuning-free asymmetric quantization for large language model KV caches. It quantizes keys per channel and values per token, with the aim of reducing cache memory during inference.

Explore the practical guide ↓

Independent OpenViST project profile, based on linked sources.

Kivi — OpenViST project profile in AI Infrastructure & Serving.
OpenViST project illustration. See the linked sources for product details.
Understand the project

Practical overview

When it fits

Use Kivi when a Python-based LLM inference workflow needs lower-bit KV-cache handling or when evaluating cache quantization on the repository’s example and benchmark tasks.

Who it is for

  • LLM developers integrating cache compression into inference pipelines
  • Researchers evaluating quantized-cache behavior on language-model tasks
  • Practitioners investigating memory use in long-context inference

Documented uses

  • Reduce KV-cache memory use during LLM inference.
  • Load a Llama model with configurable key- and value-cache bit widths.
  • Run repository examples for GSM8K or passkey retrieval.
  • Evaluate supported Llama and Mistral models on LongBench.
Before you choose

Requirements & limits

The repository is public and released under the MIT License.

  • The method uses different quantization groupings for keys and values: per-channel keys and per-token values.
  • The README’s model-loading example provides 2- and 4-bit settings for both key and value caches, along with group-size and residual-length parameters.
  • A separate CUDA implementation is part of the documented setup.
  • The README describes GQA support and updates for Transformers 4.43 and the Llama 3 family; its LongBench instructions list Llama and Mistral.
  • Reported memory and throughput gains are project results and may not describe other models or workloads.

Check current requirements in the official documentation. Source-based guidance does not mean OpenViST has run or independently tested this project.

Source-based practical guide

Install Kivi and its CUDA implementation

Set up the Python package and the repository’s CUDA implementation.

Before you start

  • Conda
  • Python 3.10
  • A local checkout of the Kivi repository

Steps

  1. Create and activate a Conda environment with Python 3.10.

  2. Upgrade pip in the environment.

  3. From the repository root, install Kivi in editable mode with pip.

  4. Enter the quant directory and install its package in editable mode with pip.

Expected outcome

The README’s two package-installation stages are complete; model loading and inference are separate next steps in the repository example.

Guide limitations

  • This guide covers installation only, not inference execution.
  • The README specifies a CUDA implementation but does not state hardware requirements or compatibility guarantees.
Documentation 1 ↗
COMPARE OPTIONS

Compare options for Model inference and serving

These projects document a common task and share a project role. This is a source-based comparison, not an execution benchmark or a guarantee of interchangeability.

ProjectDocumented purposePlatformAccess and requirements
KiviReduce KV-cache memory use during LLM inference.Linux, macOS, WindowsThe repository is public and released under the MIT License.
Cloud AI SDKAdapt and run documented model recipes for computer-vision and NLP inference.Cloud, ServerThe GitHub repository is public. Its public visibility does not establish license terms.
TurboquantReplace a model’s existing inference cache with a compressed cache.Cloud, Linux, macOS, Self-hostedThe project describes itself as open source. The README documents installation through pip or from a source checkout.
Evidence and transparency

Sources & review

Official repository ↗
Source record details
Source availability
Active
Sources checked
14 October 2026
Original website title
jy-yuan/KIVI · Official homepage title · retained in its source language
How we review sourcesSuggest a correction