Stop 22 · Interview PreparationTechnical · 4/8

System design (AI/ML)

In short

"Design a system that auto-replies to support tickets." There's no single right answer. What interviewers grade is how you clarify, estimate, choose and admit trade-offs out loud, in about 45 minutes.

Designing an AI system end-to-end under interview constraints — expect to be asked to design a RAG system, recommendation engine, or ML pipeline on a whiteboard.

The purple blob points at rows of big cooking pots and serving counters, with a spare gas cylinder nearby.

Think of it like this

Catering a 2,000-guest wedding

A wedding caterer doesn't start cooking straight away. First: how many guests, veg or non-veg, what time is dinner? Then they plan kitchens, counters, staff and a backup gas cylinder. AI system design is the same: ask, estimate, plan the flow, then prepare for what breaks.

Watch it happen

A 45-minute answer structure

Clarify

Metrics

Data

Model

Serving

Monitor

1/6Ask: how many tickets a day, which languages, can it send replies itself or only draft them?

The big ideas

What you need to know

Clarify before drawing boxes

Spend the first five minutes on scale, users and constraints. Wrong assumptions ruin the rest.

Do the rough numbers

Requests per second, tokens per request, cost per day. Rough maths shows you think like an owner.

Name your trade-offs

Bigger model vs latency, auto-send vs human review, freshness vs cost. Say what you'd pick and why.

Plan for failure

What if the LLM provider is down or slow? Timeouts, fallbacks and a human handoff belong in the design.

See it in code

Walk through the code

python

1# Back-of-envelope for "AI replies to support tickets"2tickets_per_day = 50_0003peak_factor = 3                      # traffic bunches up in office hours4avg_qps = tickets_per_day / 86_4005peak_qps = avg_qps * peak_factor6 7tokens_per_call = 1_500 + 300        # prompt with retrieved context + answer8tokens_per_day = tickets_per_day * tokens_per_call9 10price_per_million = 1.0              # placeholder: check your provider's current pricing11cost_per_day = tokens_per_day / 1_000_000 * price_per_million12print(f"peak QPS ~{peak_qps:.1f}, tokens/day {tokens_per_day:,}, cost/day ~{cost_per_day:.0f} units")

1/3Turn daily volume into requests per second, then allow for peak hours.

The kind of rough estimate to do out loud. Exact numbers don't matter; showing that 50,000 tickets means about 90 million tokens a day does.

Save yourself the pain

Common mistakes

Oops

Jumping straight into "I'd fine-tune a large model" in the first minute.

Do this instead

Clarify the goal and constraints first, then start with the simplest design that could work.

Oops

Designing only the model and forgetting everything around it.

Do this instead

Cover data, serving, latency, cost, monitoring and human fallback. The model is one box of many.

Your mission

One mock design round

~45 min

Quick check

The interviewer says "Design a recommendation system." What should you do first?

Go deeper

1Practice 2-3 full mock system design interviews for AI-specific systems.
2Structure your answers: requirements, high-level design, deep dive, trade-offs.
3Always state assumptions about scale explicitly before designing.

More technical at this stop