OpenViSTAI · Agents · Blockchain · Crypto
Submit a project
Menu
AI Infrastructure & Serving

Vcache

Vcache is a Python library that caches LLM responses and reuses them for semantically similar prompts, using online-learned, embedding-specific decision boundaries and a configurable error-rate bound.

Explore the practical guide ↓

Independent OpenViST project profile, based on linked sources.

Vcache — OpenViST project profile in AI Infrastructure & Serving.
OpenViST project illustration. See the linked sources for product details.
Understand the project

Practical overview

When it fits

Consider Vcache when an LLM-backed application receives repeated or semantically similar prompts and you want to evaluate response reuse under a chosen error-rate bound.

Who it is for

  • Developers adding response caching to Python applications backed by LLMs.
  • Teams running agentic systems, retrieval-augmented generation pipelines, or database workflows that issue LLM-based SQL queries.

Documented uses

  • Reuse responses across prompts that are similar in meaning, not only exact matches.
  • Add a cache layer between an application server and an LLM inference backend.
  • Set an error-rate bound to govern semantic response reuse.
  • Evaluate cache hit rate, error rate, and latency improvement on representative prompt workloads.
Before you choose

Requirements & limits

  • The documented quick start uses OpenAI for both inference and embeddings and requires an OpenAI API key.
  • The default configuration lists HNSWLib vector storage, in-memory embedding metadata, string-comparison similarity evaluation, no eviction, and a verified decision policy with a stated maximum failure rate of 2%.
  • The README gives examples of swapping components, including the vector database, similarity evaluator, inference and embedding engines, and eviction policy.
  • The repository states that features and APIs may change while the project is in active development.
  • The README provides a benchmarking framework for cache hit rate, error rate, and latency improvement.

Check current requirements in the official documentation. Source-based guidance does not mean OpenViST has run or independently tested this project.

Source-based practical guide

Install and make a basic Vcache inference call

Set up the documented Python library and call infer with a selected error-rate bound.

Before you start

  • A local checkout of the vCache project
  • Python and pip
  • An OpenAI API key, set as OPENAI_API_KEY

Steps

  1. From the project directory, install in editable mode with the command below.

    Documented command
    pip install -e .

  2. Set the OPENAI_API_KEY environment variable to your key.

  3. In Python, import VCache, VCachePolicy, and VerifiedDecisionPolicy from vcache.

  4. Create a policy with VerifiedDecisionPolicy(delta=0.01), initialize VCache(policy=policy), and call vcache.infer("Is the sky blue?").

Expected outcome

A response is returned through Vcache using the configured policy and the documented default components.

Guide limitations

  • The example uses OpenAI by default for inference and embeddings.
  • The repository says its features and APIs may change during active development.
Documentation 1 ↗
COMPARE OPTIONS

Compare options for Model inference and serving

These projects document a common task and share a project role. This is a source-based comparison, not an execution benchmark or a guarantee of interchangeability.

ProjectDocumented purposePlatformAccess and requirements
VcacheReuse responses across prompts that are similar in meaning, not only exact matches.
Cloud AI SDKAdapt and run documented model recipes for computer-vision and NLP inference.Cloud, ServerThe GitHub repository is public. Its public visibility does not establish license terms.
TurboquantReplace a model’s existing inference cache with a compressed cache.Cloud, Linux, macOS, Self-hostedThe project describes itself as open source. The README documents installation through pip or from a source checkout.
Evidence and transparency

Sources & review

Official repository ↗
Source record details
Source availability
Active
Sources checked
18 October 2026
Original website title
vcache-project/vCache · Official homepage title · retained in its source language
How we review sourcesSuggest a correction