Stop 19 · System Design for AI ApplicationsTopics · 8/9

Handling High-Throughput Inference

In short

Serving one request at a time leaves an expensive GPU mostly idle. Batching many requests together can multiply throughput on the same hardware, which is how big AI services stay affordable.

Techniques (batching, model parallelism, dedicated inference servers) for serving very high volumes of model requests efficiently without linearly scaling cost.

The blob lifts the lid of a tall idli steamer full of stacked plates of round white idlis.

Think of it like this

The idli steamer

An idli stall doesn't steam one idli at a time. The steamer has plates for dozens, and they all cook together in the same time as one. A GPU is that steamer: running 32 requests in one batch costs little more time than running one.

Watch it happen

Dynamic batching at a model server

enqueue(req A)

req A

Front of the queue is coloured

1/6Request A arrives. The server waits a few milliseconds to see if others join.

The big ideas

What you need to know

Batching

Group requests so the GPU does more work per pass. Throughput rises, each request waits slightly longer.

Continuous batching

LLM servers like vLLM add and remove requests from the batch every step, so short replies don't wait for long ones.

Memory is the limit

The KV cache for each running request uses GPU memory. Tricks like paged attention fit more requests in.

Smaller and quantised

Quantised or distilled models use less memory and compute, so the same GPU serves more requests.

See it in code

Walk through the code

python

1from vllm import LLM, SamplingParams2 3llm = LLM(model="Qwen/Qwen2.5-0.5B-Instruct")4params = SamplingParams(max_tokens=64, temperature=0)5 6prompts = [f"Give one fun fact about the number {n}." for n in range(256)]7outputs = llm.generate(prompts, params)  # batched on the GPU automatically8 9print(len(outputs), "replies")10print(outputs[0].outputs[0].text)

1/3Load a small open model once onto the GPU.

vLLM batches 256 prompts for you. It needs an NVIDIA GPU; a free Colab GPU can run this small model.

Save yourself the pain

Common mistakes

Oops

Looping over prompts and calling generate once per prompt.

Do this instead

Send prompts together, or run a server that batches incoming requests, so the GPU stays busy.

Oops

Making the batch window so long that users wait.

Do this instead

Batching trades latency for throughput. Keep the wait short for interactive apps, longer for offline jobs.

Your mission

Feel the batching speed-up

~35 min

Quick check

What does continuous batching fix that simple batching doesn't?

Go deeper

1Learn dynamic batching and how it improves GPU utilization for inference servers.
2Read about dedicated inference-serving tools (e.g. vLLM, TensorRT-LLM) at an overview level.
3Understand the trade-off between batching for throughput and the added latency it introduces.