Catalog My learning Blog
LLM System Design Course: RAG, Vector Databases, Cost · Phase 4 · Lesson 18 · 9 min read

LLM Caching Strategies: Semantic and Prompt Caching

On this page
  1. Learning Objective
  2. Concept Explanation
  3. Worked Example: Cache Key Design and Prompt Ordering
  4. Real-World Examples
  5. Review Questions & Answers

Learning Objective

Distinguish the four caching layers available in an LLM system - exact-match response caching, semantic caching, provider prompt caching, and embedding caching - explain the correctness risk each one introduces, and design cache keys that do not leak data across tenants or serve stale grounded answers.

Concept Explanation

Caching in conventional systems is mostly a performance decision. In an LLM system it is also the single largest cost lever available after model routing, because a cache hit converts a multi-second, multi-cent operation into a lookup. Four distinct layers exist and they are frequently confused with one another, which matters because they have completely different hit-rate profiles, correctness risks, and failure modes. Understanding them as four separate mechanisms rather than as "caching" is what allows each to be applied where it is safe.

Exact-match response caching is the familiar kind: hash the fully assembled prompt along with the model, the parameters, and the retrieved context identifiers, and store the response. It is completely safe with respect to correctness, because an identical input legitimately maps to a stored output, and it is nearly free to implement. Its limitation is hit rate, since natural language inputs rarely repeat verbatim from real users. It earns its place anyway in three situations that are more common than they first appear: automated and programmatic traffic, retry storms where the same request arrives several times in seconds, and shared content such as a summary of a document that many users open.

Semantic caching raises the hit rate by embedding the incoming query and returning a cached response when a previous query lies within a similarity threshold. It genuinely works for high-volume, low-variation traffic such as a support assistant answering the same questions in a hundred different phrasings. It also introduces the most dangerous correctness risk in this lesson, because similarity is not equivalence. "How do I cancel my subscription" and "how do I cancel my subscription without losing my data" can sit above almost any reasonable threshold while having materially different correct answers, and negation is exactly where embeddings are weakest, so a query and its negation can be near neighbours. The mitigations are a conservative threshold tuned against real traffic rather than accepted from a default, scoping the cache narrowly to question types where near-matches are genuinely equivalent, short time-to-live values, and sampling cache hits into an offline check that compares the served answer against a freshly generated one.

Provider prompt caching is different in kind and is the layer most teams should adopt first, because it carries essentially no correctness risk. Rather than caching the response, the provider caches the internal computed state for a prefix of the prompt, so a subsequent request beginning with the identical prefix skips recomputing it. Anthropic shipped this in 2024 reporting cost reductions up to roughly ninety percent and latency reductions up to roughly eighty-five percent for long cached prefixes, and OpenAI shipped automatic prompt caching with a discount on cached input tokens for sufficiently long prompts. The design implication is concrete and often overlooked: the cache matches on an exact prefix, so your prompt must be ordered with everything stable first - system instructions, tool definitions, few-shot examples, and any long shared document - and everything variable last. A single dynamic value such as a timestamp placed near the top of the prompt invalidates the entire cache on every request.

The fourth layer is embedding caching, which is unglamorous and consistently worthwhile. Embeddings are deterministic for a given model and input, so an embedding for a given text and model version can be cached permanently. On the ingest side this saves re-embedding unchanged chunks during re-indexing, which turns a full corpus rebuild from an expensive operation into a mostly-cached one. On the query side it captures repeated queries cheaply. Across all four layers, two rules are non-negotiable. The cache key must include the tenant identifier and any permission-relevant scope, because a cache keyed on question text alone will eventually serve one customer's answer to another, which is a data breach rather than a stale-cache bug. And any cache holding grounded answers must be invalidated when the underlying documents change, or the system will confidently serve answers derived from a policy that was replaced last week.

LayerWhat Is CachedTypical Hit RateCorrectness RiskAdopt When
Exact-match responseThe full response for an identical assembled promptLow for human traffic, high for automatedNoneAlways; it is nearly free
SemanticA response for a sufficiently similar queryCan be high on repetitive trafficHigh; similar is not equivalentNarrow, well-understood question sets only
Provider prompt cacheInternal state for a stable prompt prefixHigh whenever the prefix is stableNoneFirst; reorder prompts to make the prefix stable
EmbeddingThe vector for a text and model versionVery high on re-indexNone; embeddings are deterministicAlways, especially for ingestion

Worked Example: Cache Key Design and Prompt Ordering

Two details decide whether a cache is an optimization or an incident: what goes into the key, and what order the prompt is assembled in. Both are shown below.

Code
# --- 1. CACHE KEY: what is missing here is a data breach, not a bug ---
def cache_key(question, chunks, model, params, principal, prompt_version):
    return sha256(json.dumps({
        'tenant': principal.tenant_id,        # REQUIRED - cross-tenant leak without it
        'acl': principal.permission_hash,     # REQUIRED - two users, same tenant,
                                              #            different visible documents
        'q': normalize(question),
        'chunks': sorted(c.id + ':' + c.version for c in chunks),  # stale grounding
        'model': model,
        'prompt_version': prompt_version,     # a prompt release must miss the cache
        'params': params,
    }, sort_keys=True))

# --- 2. PROMPT ORDER: the provider cache matches an exact PREFIX ---
# WRONG - the timestamp at the top invalidates everything after it, every call
prompt = [
    f'Current time: {now()}',        # <-- destroys the cache for the whole prompt
    SYSTEM_INSTRUCTIONS,             # 1,800 tokens, identical every call
    FEW_SHOT_EXAMPLES,               # 2,400 tokens, identical every call
    retrieved_context,
    user_question,
]

# RIGHT - stable prefix first, volatile content last
prompt = [
    SYSTEM_INSTRUCTIONS,             # cached
    FEW_SHOT_EXAMPLES,               # cached  -> ~4,200 tokens billed at a
    TOOL_DEFINITIONS,                # cached     fraction of the normal rate
    cache_breakpoint(),
    f'Current time: {now()}',        # volatile, and now cheap to vary
    retrieved_context,
    user_question,
]

# --- 3. SEMANTIC CACHE: guard it, do not just threshold it ---
hit = semantic_cache.lookup(embed(question), scope=principal.tenant_id)
if hit and hit.similarity >= 0.97 and hit.age < timedelta(hours=6):
    metrics.increment('cache.semantic.hit')
    if random() < 0.01:                       # 1% audit sample
        enqueue_shadow_check(question, hit.response)  # regenerate and compare
    return hit.response
The four caching layers of an LLM request path: embedding cache at ingest and query encode, provider prefix cache over the stable head of the prompt, semantic cache guarded by a similarity threshold and audit sampling, and exact-match response cache, each keyed with tenant and permission scope.
The four caching layers of an LLM request path: embedding cache at ingest and query encode, provider prefix cache over the stable head of the prompt, semantic cache guarded by a similarity threshold and audit sampling, and exact-match response cache, each keyed with tenant and permission scope.

Real-World Examples

  • Anthropic introduced prompt caching in 2024, reporting cost reductions of up to roughly ninety percent and latency reductions of up to roughly eighty-five percent for long prompts with a stable prefix, and OpenAI subsequently shipped automatic prompt caching applying a discount to cached input tokens for prompts above a length threshold. Both mechanisms match on an exact prefix, which is why prompt ordering became a concrete cost-engineering decision rather than a stylistic one.
  • GPTCache, an open-source semantic caching library released in 2023, made the pattern widely available by embedding incoming queries, searching a vector store of previous queries, and returning the stored response above a similarity threshold. Its own documentation stresses threshold tuning and eviction policy as the critical configuration, which reflects the central operational reality that the threshold is the boundary between a cost saving and a wrong answer served confidently.
WarningA semantic cache returns an answer written for a different question, and similarity is not equivalence. Embeddings are specifically weak at negation and at qualifying clauses, so a query and its opposite can score above a naive threshold. Never ship a semantic cache without a conservatively tuned threshold measured on real traffic, a short time-to-live, a tenant-scoped key, and an audit sample that regenerates a fraction of hits and compares them to what was served.

Review Questions & Answers

It delivers a large share of the available cost and latency saving with no correctness risk at all, because it caches the internal computation for a prefix rather than caching an answer, so the model still generates a fresh response for every request. What it requires is that the prompt be ordered with all stable content first: system instructions, tool definitions, few-shot examples, and any long shared document, with variable content such as timestamps, retrieved context, and the user question placed after. Because matching is on an exact prefix, a single dynamic value near the top invalidates the cache for everything that follows it, which is a common and entirely avoidable loss.

Because the answer is not a function of the question alone. It is a function of the question and the documents the asker is entitled to see. Two users in different tenants asking word-for-word the same question must receive answers grounded in entirely different corpora, and two users in the same tenant with different permissions may legitimately see different documents. A key based on question text alone will eventually return one customer's content to another, which is a data breach rather than a stale-cache inconvenience, and it will be invisible in every metric because the cache hit looks like a success.

Exact-match caching relies on the identity of the input, so a hit is correct by construction. Semantic caching relies on a similarity threshold, and similarity is not equivalence: a qualifying clause or a negation can change the correct answer entirely while barely moving the embedding, and negation is precisely where embeddings are weakest. Acceptable use requires a threshold tuned conservatively against real traffic rather than a library default, restriction to question categories where near-matches are genuinely interchangeable, a short time-to-live so the blast radius of a bad entry is bounded, tenant and permission scoping in the key, and an audit sample that regenerates a small fraction of hits and flags divergence.

Because the answer was derived from specific retrieved documents, and its correctness expires when those documents change. Including the identifiers and content versions of the chunks that grounded the answer means that editing a source document naturally produces a different key and therefore a cache miss, so the next request regenerates from the current material. Keying on the question alone leaves the system confidently serving an answer derived from a superseded policy with no mechanism that would ever notice, which is the same staleness failure the ingestion pipeline lesson warned about, arriving through the cache instead of through the index.