Skip to main content
Every time your application calls a large language model you pay for tokens and wait for a round trip. Semantic caching short-circuits that cost: Cache-Pot stores the LLM’s response alongside a vector embedding of the original prompt, and on the next request it checks whether the incoming prompt is semantically similar enough to any stored prompt to return the cached response directly. Because the comparison is over meaning rather than raw text, a rephrased question (“What’s the capital of France?” vs. “Which city is France’s capital?”) can still hit the cache — no exact-match required.

​
How it works

1

Store a response

Call SCACHE.SET with the original prompt, the LLM response, and an optional TTL. Cache-Pot sends the prompt to your configured embeddings endpoint and stores the resulting vector in an internal collection (__scache__).
2

Embed the incoming query

When your app calls SCACHE.GET, Cache-Pot embeds the incoming prompt using the same embeddings endpoint and queries the __scache__ collection for the nearest neighbour.
3

Threshold check

Cache-Pot compares the cosine similarity score of the best match against the threshold (default 0.90). If the score meets or exceeds the threshold, it returns the cached response.
4

Cache miss

If no stored prompt meets the threshold, SCACHE.GET returns nil. Your application then calls the LLM and, optionally, stores the new response with another SCACHE.SET.
Hit and miss counts are tracked and exposed via INFO and the built-in dashboard at http://localhost:8080. Every hit is a model call you didn’t pay for.

​
Requirements

Before using semantic cache commands, configure an OpenAI-compatible embeddings endpoint:

​
Basic usage

The second query uses different phrasing but returns the cached response because its prompt embedding is cosine-similar to the stored one.

​
Adjusting the similarity threshold

The default threshold is 0.90 (90% cosine similarity). Lower the threshold to match more loosely-worded queries; raise it to require near-identical phrasing.
Start with the default 0.90. If you see too many cache misses for near-identical queries, lower the threshold to 0.85. If you see incorrect cache hits (wrong responses returned for clearly different prompts), raise it toward 0.95.

​
Dollar-savings example

Track the running hit ratio on the dashboard — each hit line shows the similarity score of the matched prompt, so you can tune the threshold over real traffic.

​
Persistence

Semantic cache entries are stored in the same keyspace as all other data. They survive restarts through the normal snapshot and AOF mechanisms — Cache-Pot logs a SCACHE.SET as its resulting VSET command in the AOF, so replay never re-calls the embeddings endpoint.
The internal collection __scache__ is a regular vector collection. You can inspect it with VCARD __scache__ and VDIM __scache__, and it respects all standard TTL and persistence behaviour.