Every time your application calls a large language model you pay for tokens and wait for a round trip. Semantic caching short-circuits that cost: Cache-Pot stores the LLM’s response alongside a vector embedding of the original prompt, and on the next request it checks whether the incoming prompt is semantically similar enough to any stored prompt to return the cached response directly. Because the comparison is over meaning rather than raw text, a rephrased question (“What’s the capital of France?” vs. “Which city is France’s capital?”) can still hit the cache — no exact-match required.
How it works
Store a response
Call SCACHE.SET with the original prompt, the LLM response, and an optional TTL. Cache-Pot sends the prompt to your configured embeddings endpoint and stores the resulting vector in an internal collection (__scache__).
Embed the incoming query
When your app calls SCACHE.GET, Cache-Pot embeds the incoming prompt using the same embeddings endpoint and queries the __scache__ collection for the nearest neighbour.
Threshold check
Cache-Pot compares the cosine similarity score of the best match against the threshold (default 0.90). If the score meets or exceeds the threshold, it returns the cached response.
Cache miss
If no stored prompt meets the threshold, SCACHE.GET returns nil. Your application then calls the LLM and, optionally, stores the new response with another SCACHE.SET.
Hit and miss counts are tracked and exposed via INFO and the built-in dashboard at http://localhost:8080. Every hit is a model call you didn’t pay for.
Requirements
Before using semantic cache commands, configure an OpenAI-compatible embeddings endpoint:
| Environment variable | Required | Description |
|---|
CACHEPOT_EMBED_URL | Yes | Base URL of your embeddings endpoint (e.g. https://api.openai.com/v1/embeddings). |
CACHEPOT_EMBED_KEY | If needed | API key for the endpoint. |
CACHEPOT_EMBED_MODEL | No | Model name to request (default: text-embedding-3-small). |
export CACHEPOT_EMBED_URL=https://api.openai.com/v1/embeddings
export CACHEPOT_EMBED_KEY=sk-...
# CACHEPOT_EMBED_MODEL defaults to text-embedding-3-small
Basic usage
# Cache a response with a 1-hour TTL
SCACHE.SET "What is the capital of France?" "The capital of France is Paris." TTL 3600
# Query with different wording — still a cache hit
SCACHE.GET "What's the capital city of France?"
# Returns: "The capital of France is Paris."
Adjusting the similarity threshold
The default threshold is 0.90 (90% cosine similarity). Lower the threshold to match more loosely-worded queries; raise it to require near-identical phrasing.
# More permissive: accept matches at 85% similarity
SCACHE.GET "capital of France" THRESHOLD 0.85
# Stricter: only accept very close matches
SCACHE.GET "What is the capital of France?" THRESHOLD 0.95
Start with the default 0.90. If you see too many cache misses for near-identical queries, lower the threshold to 0.85. If you see incorrect cache hits (wrong responses returned for clearly different prompts), raise it toward 0.95.
Dollar-savings example
# Suppose each LLM completion costs $0.01.
SCACHE.SET "summarize the BSD-3 license" "<summary text>" TTL 3600
SCACHE.GET "give me a summary of the BSD 3-clause license"
# HIT -> $0.01 saved, response returned in milliseconds
Persistence
Semantic cache entries are stored in the same keyspace as all other data. They survive restarts through the normal snapshot and AOF mechanisms — Cache-Pot logs a SCACHE.SET as its resulting VSET command in the AOF, so replay never re-calls the embeddings endpoint.
# Set a semantic cache entry with a TTL — it will survive a restart and
# expire naturally at the right wall-clock time.
SCACHE.SET "explain gradient descent" "<response>" TTL 86400
The internal collection __scache__ is a regular vector collection. You can inspect it with VCARD __scache__ and VDIM __scache__, and it respects all standard TTL and persistence behaviour.
