Stop 19 · System Design for AI ApplicationsTopics · 1/9

Designing Scalable AI Systems

In short

An AI app that works for 10 friends can fall over at 10,000 users. Model calls are slow, costly and rate limited, so the design has to plan around them from day one.

Architecting AI applications to handle growing traffic and data volume — combines standard system design with AI-specific bottlenecks like model inference cost and latency.

The blob in a white cap sorts rows of steel tiffin boxes onto a long wooden crate at a railway platform.

Think of it like this

Mumbai's dabbawalas

Dabbawalas move lakhs of tiffins a day without fancy tech. Tiffins are grouped at stations, handed in batches, and each person does one simple step. Nobody carries one box across the whole city. Scalable AI systems work the same way: queues, batches and small, repeatable steps.

Watch it happen

A long AI job that doesn't block the user

Client

API

Queue

Worker

POST /summarise (40 pp)

1/6A user uploads a long PDF. Summarising it could take a minute, too long to hold an HTTP request open.

The big ideas

What you need to know

Queue slow work

Anything over a few seconds goes to a background queue. The API stays fast and workers scale separately.

Respect rate limits

Your provider's limits are a hard ceiling. Cap requests in flight across all workers.

Stateless API servers

Keep sessions and history in a database or cache, so any server can handle any request.

Measure per step

Track latency, tokens and errors for retrieval, model calls and post-processing separately.

See it in code

Walk through the code

python

1import asyncio2 3sem = asyncio.Semaphore(20)  # max calls in flight, under the rate limit4 5async def call_llm(prompt):6    await asyncio.sleep(0.2)  # stand-in for a real API call7    return prompt.upper()8 9async def handle(prompt):10    async with sem:11        return await call_llm(prompt)12 13async def main():14    results = await asyncio.gather(*(handle(f"q{i}") for i in range(100)))15    print(len(results), "done")16asyncio.run(main())

1/3A semaphore with 20 slots caps how many calls run at the same time.

100 jobs, but never more than 20 model calls at once. A semaphore is the simplest way to keep a worker under the rate limit.

Save yourself the pain

Common mistakes

Oops

Doing a 60-second LLM job inside the HTTP request.

Do this instead

Return a job id quickly, run the work in a background worker, and let the client poll or subscribe.

Oops

Scaling to 50 workers that all hit the same rate limit.

Do this instead

More workers won't help past the provider's limit. Share a global cap, or spread load across providers.

Your mission

Sketch the system

~30 min

Quick check

A request takes 45 seconds because of a long LLM call. Best design change?

Go deeper

1Design an AI system on a whiteboard for a concrete product (e.g. a customer support bot at scale).
2Identify the bottleneck component first — usually model inference, not the surrounding infrastructure.
3Practice explaining your design trade-offs out loud, as you would in an interview.