Stop 19 · System Design for AI ApplicationsTopics · 3/9

Caching LLM Responses

In short

During placement season, 500 students ask your bot 'What's the dress code?' Calling the model 500 times for the same answer is slow and wasteful. Cache it once, serve it 499 times.

Storing and reusing responses for repeated or similar queries to cut both latency and API cost — a high-leverage optimization for any AI product with predictable query patterns.

The blob pours chai from a big steel flask into a row of small glasses for a queue of office workers.

Think of it like this

The chaiwala's ready flask

At the 8 am office rush, the chaiwala doesn't brew each cup from scratch. He keeps a big flask of ready cutting chai, because everyone orders the same thing. A fresh batch only when the flask runs out or goes cold. That flask is your cache.

Watch it happen

A cache miss, then a hit

App

Cache

LLM

GET key sha256(...)

1/6The app hashes the model name and messages into a key and checks the cache first.

The big ideas

What you need to know

Exact-match cache

Hash model, prompt and settings into a key. Identical requests get the stored answer.

Semantic cache

Embed the question and reuse answers to very similar questions. Faster hits, but risk of wrong matches.

Provider prompt caching

Different idea: the provider caches a long fixed prompt prefix, so you pay less to resend it.

Expire and scope

Set a TTL, and never share cached answers that contain one user's personal data with another user.

See it in code

Walk through the code

python

1import hashlib, json, time2 3CACHE, TTL = {}, 36004 5def cache_key(model, messages):6    raw = json.dumps({"m": model, "msgs": messages}, sort_keys=True)7    return hashlib.sha256(raw.encode()).hexdigest()8 9def cached_call(model, messages, call):10    key = cache_key(model, messages)11    hit = CACHE.get(key)12    if hit and time.time() - hit[1] < TTL:13        return hit[0]14    answer = call(model, messages)15    CACHE[key] = (answer, time.time())16    return answer

1/3sort_keys keeps the JSON identical for identical requests, so the hash matches.

An exact-match cache with expiry. In production, swap the dictionary for Redis so all servers share it.

Save yourself the pain

Common mistakes

Oops

Caching answers to 'What's my exam seat number?' across all users.

Do this instead

Include the user id in the key for personal questions, or don't cache them at all.

Oops

Using a semantic cache with a very loose similarity threshold.

Do this instead

'Refund policy for flights' and 'refund policy for hotels' are close but different. Test thresholds on real questions.

Your mission

Measure your hit rate

~25 min

Quick check

Why include the model name and settings in the cache key?

Go deeper

1Implement exact-match response caching for a repeated query pattern.
2Learn semantic caching (matching similar, not identical, queries) at a conceptual level.
3Identify cache-invalidation risks specific to AI responses (stale facts, outdated context).