Stop 16 · AI APIs & Cloud AI PlatformsTopics · 5/9

Rate Limits & Cost Management

In short

Your demo works. Then 200 students use it during a placement drive and you get 429 errors and a scary bill. Rate limits and cost controls are what keep a real AI app alive.

LLM APIs charge per token and enforce request-rate limits — production AI features need deliberate cost tracking and backoff/retry logic to avoid surprise bills or failures.

The blob looks at a phone with a big battery-like gauge nearly empty, holding a small coin purse.

Think of it like this

Your daily mobile data pack

Your data pack gives you 2 GB a day and slows you down if you use too much too fast. Overspend and you pay extra. LLM APIs work the same: limits on requests and tokens per minute, and a bill that grows with every token.

Watch it happen

Retrying after a 429 with exponential backoff

try 1

now

try 2

try 3

try 4

try 5

1/5Try 1 fails with 429 Too Many Requests. You have hit the tokens-per-minute limit.

The big ideas

What you need to know

RPM and TPM

Providers limit requests per minute and tokens per minute, per model. Your account tier sets the numbers.

Back off on 429

Retry with growing waits and jitter. Respect a retry-after header if the response has one.

Cost = tokens x price

Multiply input and output tokens by the model's per-token prices. Log it per user and per feature.

Cheap wins first

Shorter prompts, prompt caching, smaller models for easy tasks and batch APIs for non-urgent jobs all cut the bill.

See it in code

Walk through the code

python

1import random, time2from openai import OpenAI, RateLimitError3 4client = OpenAI(max_retries=0)  # we handle retries ourselves here5 6def ask(prompt, tries=5):7    for attempt in range(tries):8        try:9            return client.responses.create(model="gpt-5-mini", input=prompt)10        except RateLimitError:11            wait = 2 ** attempt + random.random()  # 1, 2, 4, 8... plus jitter12            print(f"429, waiting {wait:.1f}s")13            time.sleep(wait)14    raise RuntimeError("Still rate limited, please try later")

1/3Turn off the SDK's own retries so you can see the logic clearly.

Exponential backoff by hand. Official SDKs already retry a couple of times by default; this shows what they do.

Save yourself the pain

Common mistakes

Oops

Retrying a 429 instantly in a tight loop.

Do this instead

Instant retries make the overload worse. Wait, grow the wait, and add jitter.

Oops

Using the biggest model for every request, including 'say hi back'.

Do this instead

Route easy tasks to a small cheap model and save the big one for hard questions.

Your mission

Build a cost log

~30 min

Quick check

Why add random jitter to backoff waits?

Go deeper

1Add token-usage logging and a rough cost estimate to every API call in a project.
2Implement exponential backoff and retry logic for rate-limit errors.
3Learn techniques (caching, smaller models for simple tasks) to control cost at scale.