OpenViSTAI · Agents · Blockchain · Crypto
Submit a project
Menu
AI Infrastructure & Serving

Turboquant

Turboquant is an open-source Python library for compressing the key-value (KV) cache used during large language model inference. It offers configurable low-bit cache storage, including a 4-bit setting, and a Hugging Face integration that passes its cache into a model’s forward call. The repository also documents an OpenAI-compatible inference server and direct vector quantization utilities.

Explore the practical guide ↓

Independent OpenViST project profile, based on linked sources.

Turboquant — OpenViST project profile in AI Infrastructure & Serving.
OpenViST project illustration. See the linked sources for product details.
Understand the project

Practical overview

When it fits

Consider Turboquant when KV-cache memory is a constraint in a Hugging Face inference workload, especially at longer contexts. Compare it with the existing cache on the target model and workload because memory savings and throughput vary.

Who it is for

  • Developers integrating Hugging Face models who need to manage KV-cache memory during inference.
  • Teams serving long-context LLM workloads that want to evaluate lower-precision cache storage.
  • Developers exploring vector quantization through the library’s direct algorithm interface.

Documented uses

  • Replace a model’s existing inference cache with a compressed cache.
  • Adjust key and value bit widths, or preserve selected layers at FP16 precision.
  • Run an inference server using the documented command-line interface.
  • Quantize and dequantize vectors using the core quantization class.
Before you choose

Requirements & limits

The project describes itself as open source. The README documents installation through pip or from a source checkout.

  • The cache can be configured symmetrically at 4 bits or asymmetrically with different key and value widths; selected layers can be protected at FP16.
  • The README’s benchmark results vary substantially by model and context: some Qwen workloads show memory savings and improved throughput, while shorter-context cases can slow down and StableLM results show higher VRAM use
  • Documented Hugging Face integration uses past_key_values and use_cache=True; the package also includes a separate inference-server path and direct quantization utilities.
  • The README lists PyTorch 2.5.1, CUDA 12.1 and an RTX 4080 16 GB for benchmark conditions, not as universal compatibility requirements.

Check current requirements in the official documentation. Source-based guidance does not mean OpenViST has run or independently tested this project.

Source-based practical guide

Add a 4-bit cache to a Hugging Face model

Use Turboquant’s documented cache in a model forward pass.

Before you start

  • Install the turboquant package with pip.
  • Use a Hugging Face causal language model and tokenizer with Transformers and PyTorch, as in the README example.

Steps

  1. Import TurboQuantCache and create it with TurboQuantCache(bits=4).

  2. Tokenize input text and move the inputs to the model device.

  3. Call the model with past_key_values=cache and use_cache=True.

Expected outcome

The model call uses Turboquant’s symmetric 4-bit key-and-value cache.

Guide limitations

  • The README cautions that throughput depends on model and workload; benchmark the intended context and model.
  • The example shows cache integration, not a guarantee of a particular memory saving or speed.
Documentation 1 ↗
Source-based practical guide

Install Turboquant from the repository

Install the library from a local source checkout, as documented.

Before you start

  • Git and pip are needed for the listed checkout and installation commands.

Steps

  1. Clone https://github.com/back2matching/turboquant.

  2. Change into the turboquant directory.

  3. Run the command below.

    Documented command
    pip install -e .

Expected outcome

The repository’s README provides an editable local installation route.

Guide limitations

  • These steps cover installation only; they do not establish that a model or server is running.
Documentation 1 ↗
COMPARE OPTIONS

Compare options for Model inference and serving

These projects document a common task and share a project role. This is a source-based comparison, not an execution benchmark or a guarantee of interchangeability.

ProjectDocumented purposePlatformAccess and requirements
TurboquantReplace a model’s existing inference cache with a compressed cache.Cloud, Linux, macOS, Self-hostedThe project describes itself as open source. The README documents installation through pip or from a source checkout.
Cloud AI SDKAdapt and run documented model recipes for computer-vision and NLP inference.Cloud, ServerThe GitHub repository is public. Its public visibility does not establish license terms.
GptqmodelQuantize a language model using calibration data and save the resulting checkpoint.CLI, Linux, macOS, Windows
Evidence and transparency

Sources & review

Official repository ↗
Source record details
Source availability
Active
Sources checked
18 October 2026
Original website title
back2matching/turboquant · Official homepage title · retained in its source language
How we review sourcesSuggest a correction