Practical overview
When it fits
Use it when an application sends recurring or near-repeated prompts and you want to evaluate whether local response reuse can reduce provider calls or latency. Assess cache behavior against the actual prompt distribution before relying on it.
Who it is for
- Teams operating RAG applications with recurring internal questions.
- Builders of AI agents with repeated reasoning or tool-use patterns.
- Developers of support bots that receive similar customer questions.
- Operators seeking a self-hosted cache layer between applications and LLM providers.
Documented uses
- Avoid another upstream generation request when a sufficiently similar prompt has a cached response.
- Route OpenAI-compatible chat-completions traffic through a semantic cache.
- Serve streamed responses through SSE on cache hits and misses.
- Configure similarity thresholds and optional gray-zone intent verification.
- Inspect management information and warm the cache using supplied prompt-response entries.
Requirements & limits
The repository is public and marked MIT licensed. The documented service is self-hosted; Docker Compose and running from source are described.
- Provider configurations cover OpenAI, Mistral AI and Anthropic/Claude; Claude embeddings use Voyage AI.
- BadgerDB can persist prompts, responses, embeddings and cache metadata; the configurable cache TTL defaults to 24 hours.
- Semantic matching is probabilistic and must not act as an authorization boundary across users, tenants or scopes.
- Management bearer-token authentication is enabled when API_AUTH_TOKEN is set; the README recommends setting it for non-local deployments.
- The inference endpoint does not provide application-level authorization, so deployments need appropriate external access controls.
Check current requirements in the official documentation. Source-based guidance does not mean OpenViST has run or independently tested this project.
Start PromptCache with Docker Compose
Set up the self-hosted service using the documented Docker quick start.
Before you start
- Git and Docker Compose
- An OpenAI API key for the documented provider configuration
Steps
Clone https://github.com/messkan/prompt-cache and change into the repository directory.
Set EMBEDDING_PROVIDER=openai and provide OPENAI_API_KEY.
Set API_AUTH_TOKEN, especially for a non-local deployment.
Run docker-compose up -d.
Expected outcome
The documented Docker Compose setup starts PromptCache with OpenAI selected for embeddings and verification.
Guide limitations
- This setup guide does not verify successful startup or application connectivity.
- The Docker quick start does not document a subsequent request in this recipe.
- Semantic matching is probabilistic and does not guarantee prompts are interchangeable.
Point an OpenAI-compatible client at PromptCache
Configure the documented Python OpenAI client example to send a chat-completions request through PromptCache.
Before you start
- A running PromptCache instance
- An OpenAI-compatible Python SDK
- A configured provider and the required provider credentials
Steps
Create an OpenAI client with base_url set to http://localhost:8080/v1.
Supply the API key used by the client configuration.
Call chat.completions.create with a model and messages.
Repeat a sufficiently similar request to allow PromptCache to determine whether it can serve a cached response.
Expected outcome
The first request is forwarded to the configured provider; a later sufficiently similar request may be served from cache.
Guide limitations
- Whether a later request matches is determined by semantic thresholds and is not guaranteed.
- The inference endpoint is not an application-level authorization system; use the access controls required by the application.
Sources & review
Source record details
- Source availability
- Active
- Sources checked
- 14 October 2026
- Original website title
- messkan/prompt-cache · Official homepage title · retained in its source language