Stop 19 · System Design for AI ApplicationsTopics · 2/9

Latency & Cost Optimization

In short

Users leave if replies are slow. Founders panic if the bill doubles. Most AI latency and cost comes from a few things you can control: tokens, model choice and how many calls you make.

Reducing response time and per-request cost in an AI system — through smaller models for simple tasks, caching, batching and smarter prompt design.

The blob as a nurse with a clipboard points between a small door and a big specialist door.

Think of it like this

Triage at a hospital

At a busy hospital, a nurse sees everyone first. Fever and a cough? Handled in five minutes. Chest pain? Straight to the specialist. Sending everyone to the senior doctor would be slow and expensive. Good AI systems triage too: easy questions to small models, hard ones to big ones.

Watch it happen

Where the savings come from

Faster + cheaper

Fewer tokens

Shorter prompts

Cap output

Cheaper model

Route easy ones

Fewer calls

Cache answers

Batch jobs

1/5Cost is tokens times price. Latency is mostly how many tokens the model must read and write.

The big ideas

What you need to know

Measure first

Log tokens, latency and cost per feature before optimising. The expensive part is often a surprise.

Model routing

A cheap classifier or simple rules decide which model handles each request.

Prompt caching

Providers can cache a long, fixed prompt prefix, which cuts cost and latency on repeated calls.

Stream for feel

Streaming doesn't cut total time, but users see words sooner, which feels much faster.

See it in code

Walk through the code

python

1HARD_HINTS = ("prove", "design", "debug", "compare", "step by step")2 3def pick_model(question: str) -> str:4    q = question.lower()5    if len(q) < 200 and not any(h in q for h in HARD_HINTS):6        return "small-fast-model"7    return "large-smart-model"8 9for q in ["What time does the library close?",10          "Debug this recursion and compare it with an iterative version"]:11    print(pick_model(q), "<-", q[:40])

1/3Words that usually mean a harder task.

A rule-based router. Real systems often use a small classifier, but start with rules and check quality on both paths.

Save yourself the pain

Common mistakes

Oops

Optimising the prompt by guesswork without measuring tokens.

Do this instead

Log input and output tokens per feature. Fix the biggest line item first.

Oops

Routing to a small model without checking answer quality.

Do this instead

Run your eval set on both paths. Savings mean nothing if easy answers get worse.

Your mission

Cut a bill in half on paper

~30 min

Quick check

Your chatbot's answers average 900 output tokens but users only read the first paragraph. Best fix?

Go deeper

1Profile a real AI pipeline you've built and identify the slowest/most expensive step.
2Learn model-cascading: use a cheap model first, escalate to a stronger one only when needed.
3Practice estimating cost-per-request for a hypothetical AI feature at scale.