> ## Documentation Index
> Fetch the complete documentation index at: https://cache-pot.thatdevguy.in/llms.txt
> Use this file to discover all available pages before exploring further.

# Semantic caching: cache LLM responses by prompt meaning

> Cache-Pot's semantic cache stores LLM responses and retrieves them by meaning using vector embeddings — not just exact text match.

Every time your application calls a large language model you pay for tokens and wait for a round trip. Semantic caching short-circuits that cost: Cache-Pot stores the LLM's response alongside a vector embedding of the original prompt, and on the next request it checks whether the incoming prompt is *semantically similar enough* to any stored prompt to return the cached response directly. Because the comparison is over meaning rather than raw text, a rephrased question ("What's the capital of France?" vs. "Which city is France's capital?") can still hit the cache — no exact-match required.

***

## How it works

<Steps>
  <Step title="Store a response">
    Call `SCACHE.SET` with the original prompt, the LLM response, and an optional TTL. Cache-Pot sends the prompt to your configured embeddings endpoint and stores the resulting vector in an internal collection (`__scache__`).
  </Step>

  <Step title="Embed the incoming query">
    When your app calls `SCACHE.GET`, Cache-Pot embeds the incoming prompt using the same embeddings endpoint and queries the `__scache__` collection for the nearest neighbour.
  </Step>

  <Step title="Threshold check">
    Cache-Pot compares the cosine similarity score of the best match against the threshold (default `0.90`). If the score meets or exceeds the threshold, it returns the cached response.
  </Step>

  <Step title="Cache miss">
    If no stored prompt meets the threshold, `SCACHE.GET` returns `nil`. Your application then calls the LLM and, optionally, stores the new response with another `SCACHE.SET`.
  </Step>
</Steps>

<Note>
  Hit and miss counts are tracked and exposed via `INFO` and the built-in dashboard at `http://localhost:8080`. Every hit is a model call you didn't pay for.
</Note>

***

## Requirements

Before using semantic cache commands, configure an OpenAI-compatible embeddings endpoint:

| Environment variable   | Required  | Description                                                                         |
| ---------------------- | --------- | ----------------------------------------------------------------------------------- |
| `CACHEPOT_EMBED_URL`   | **Yes**   | Base URL of your embeddings endpoint (e.g. `https://api.openai.com/v1/embeddings`). |
| `CACHEPOT_EMBED_KEY`   | If needed | API key for the endpoint.                                                           |
| `CACHEPOT_EMBED_MODEL` | No        | Model name to request (default: `text-embedding-3-small`).                          |

<CodeGroup>
  ```bash OpenAI theme={null}
  export CACHEPOT_EMBED_URL=https://api.openai.com/v1/embeddings
  export CACHEPOT_EMBED_KEY=sk-...
  # CACHEPOT_EMBED_MODEL defaults to text-embedding-3-small
  ```

  ```bash Ollama (local) theme={null}
  export CACHEPOT_EMBED_URL=http://localhost:11434/v1/embeddings
  export CACHEPOT_EMBED_MODEL=nomic-embed-text
  # No API key needed for local Ollama
  ```
</CodeGroup>

***

## Basic usage

```bash theme={null}
# Cache a response with a 1-hour TTL
SCACHE.SET "What is the capital of France?" "The capital of France is Paris." TTL 3600

# Query with different wording — still a cache hit
SCACHE.GET "What's the capital city of France?"
# Returns: "The capital of France is Paris."
```

The second query uses different phrasing but returns the cached response because its prompt embedding is cosine-similar to the stored one.

***

## Adjusting the similarity threshold

The default threshold is `0.90` (90% cosine similarity). Lower the threshold to match more loosely-worded queries; raise it to require near-identical phrasing.

```bash theme={null}
# More permissive: accept matches at 85% similarity
SCACHE.GET "capital of France" THRESHOLD 0.85

# Stricter: only accept very close matches
SCACHE.GET "What is the capital of France?" THRESHOLD 0.95
```

<Tip>
  Start with the default `0.90`. If you see too many cache misses for near-identical queries, lower the threshold to `0.85`. If you see incorrect cache hits (wrong responses returned for clearly different prompts), raise it toward `0.95`.
</Tip>

***

## Dollar-savings example

```bash theme={null}
# Suppose each LLM completion costs $0.01.
SCACHE.SET "summarize the BSD-3 license" "<summary text>" TTL 3600
SCACHE.GET "give me a summary of the BSD 3-clause license"
# HIT -> $0.01 saved, response returned in milliseconds
```

Track the running hit ratio on the dashboard — each hit line shows the similarity score of the matched prompt, so you can tune the threshold over real traffic.

***

## Persistence

Semantic cache entries are stored in the same keyspace as all other data. They survive restarts through the normal snapshot and AOF mechanisms — Cache-Pot logs a `SCACHE.SET` as its resulting `VSET` command in the AOF, so replay never re-calls the embeddings endpoint.

```bash theme={null}
# Set a semantic cache entry with a TTL — it will survive a restart and
# expire naturally at the right wall-clock time.
SCACHE.SET "explain gradient descent" "<response>" TTL 86400
```

<Info>
  The internal collection `__scache__` is a regular vector collection. You can inspect it with `VCARD __scache__` and `VDIM __scache__`, and it respects all standard TTL and persistence behaviour.
</Info>