Stop 17 · MLOps & Model DeploymentTopics · 8/10

Batch vs Real-time Inference

In short

Does your prediction need to be ready in 100 ms while the user waits, or by tomorrow morning for a report? That one question shapes your whole serving setup.

Batch inference processes large volumes of data on a schedule; real-time inference responds to individual requests instantly — the choice shapes your entire serving architecture.

The blob stirs a giant mess cooking pot on one side while a tiny single noodle bowl steams on a small stove beside it.

Think of it like this

Mess dinner vs the Maggi point

The hostel mess cooks dinner for 400 students at once at 7 pm: cheap and efficient, but you eat what's ready. The Maggi point outside makes one plate the moment you order. Batch inference is the mess, real-time inference is the Maggi point.

Watch it happen

Two ways to serve predictions

Batch

Score millions of rows on a schedule
Results saved to a table
Cheap, minutes or hours of delay

Real-time

One request, answer in milliseconds
Model always running behind an API
Fresh input, higher serving cost

Use batch when results can wait, like nightly churn scores. Use real-time when the user is waiting, like fraud checks at payment.

The big ideas

What you need to know

Batch = scheduled jobs

A job reads a whole table, predicts for every row and writes results back. Apps then just look up the saved answer.

Real-time = always on

A server keeps the model loaded and answers each request. Latency and uptime now matter a lot.

Freshness is the trade-off

Batch scores can be hours old. If the input changes every minute, like a cart, batch may be too stale.

Mix them

Many systems precompute heavy parts in batch and do a quick real-time step on top.

See it in code

Walk through the code

python

1import joblib2import pandas as pd3 4FEATURES = ["orders_30d", "days_since_last_order", "complaints_90d"]5model = joblib.load("churn_model.joblib")6 7df = pd.read_csv("customers_today.csv")8df["churn_risk"] = model.predict_proba(df[FEATURES])[:, 1]9df[["customer_id", "churn_risk"]].to_csv("churn_scores_today.csv", index=False)10print(f"Scored {len(df)} customers")

1/3Same features and model as training, loaded once for the whole job.

A nightly batch job. A scheduler like cron or Airflow runs it, and the app just reads churn_scores_today.csv.

Save yourself the pain

Common mistakes

Oops

Building a real-time API for a dashboard that refreshes once a day.

Do this instead

If nobody waits for the answer, a batch job is cheaper and simpler.

Oops

Looping over rows and calling predict one by one in a batch job.

Do this instead

Predict on the whole DataFrame, or in large chunks. It can be many times faster.

Your mission

Same model, two modes

~30 min

Quick check

A bank must decide in under a second whether a card payment looks like fraud. Which mode?

Go deeper

1Identify which of your projects genuinely need real-time inference vs. could run in batch.
2Learn the cost and latency trade-offs between the two approaches.
3Build a small batch inference job and a small real-time endpoint to feel the difference directly.