Stop 04 · SQL & Data Engineering BasicsTopics · 5/9

Batch vs Streaming Data

In short

Does your model need yesterday's data by morning, or this second's data right now? A fraud check on a UPI payment can't wait for tonight's batch. A monthly sales report can.

Batch processing handles data in scheduled chunks; streaming processes it continuously as it arrives — the choice determines your pipeline architecture and latency guarantees.

The purple blob carries a full bucket while a nearby tap fills a glass.

Think of it like this

A bucket or a running tap

Batch is filling a bucket and carrying it once: lots of water, one trip, some waiting. Streaming is a tap that runs all the time: each drop arrives the moment it's ready. Both deliver water. The question is how soon you need it.

Watch it happen

Same orders, two ways

Batch

Runs on a schedule, like 2 am
Processes a whole day at once
Simpler, cheaper, easy to rerun

Streaming

Processes each event as it arrives
Results in seconds
Needs always-on systems like Kafka

Start with batch. Move to streaming only when a decision truly needs fresh data, like fraud checks or live recommendations.

The big ideas

What you need to know

Batch = scheduled chunks

Process all of yesterday's data in one go. Most training data and reports work fine this way.

Streaming = event by event

Each click or payment is handled within seconds. Tools like Kafka carry the events.

Latency is the deciding question

Ask how stale the data can be. Hours? Batch. Seconds? Streaming. Minutes? Small, frequent batches often work.

Streaming is harder

Late events, duplicates and keeping state are trickier when data never stops. It costs more to run and debug.

See it in code

Walk through the code

python

1import pandas as pd2 3# batch: run once a day on the full file4day = pd.read_csv("orders_2026-03-01.csv")5print(day.groupby("city")["amount"].sum())6 7# streaming: handle each event the moment it arrives8def order_events():9    yield {"user": 7, "amount": 300}10    yield {"user": 7, "amount": 45_000}11    yield {"user": 9, "amount": 120}12 13for event in order_events():          # in real life: a Kafka consumer loop14    if event["amount"] > 20_000:15        print("flag for review:", event)

1/3Batch: load a complete day and summarise it in one go.

The generator stands in for a real event stream. The key difference: batch sees the whole day at once, streaming decides on each event without waiting.

Save yourself the pain

Common mistakes

Oops

Building a streaming system for a dashboard people check once a day.

Do this instead

Use batch. It's simpler, cheaper and far easier to fix when something goes wrong.

Oops

Assuming streaming events always arrive in order and only once.

Do this instead

Design for late and duplicate events, for example by using event ids and timestamps.

Your mission

Batch it, then stream it

~20 min

Quick check

Which use case most needs streaming instead of batch?

Go deeper

1List AI use cases that need streaming (fraud detection) vs. batch (monthly reports).
2Learn the basic concepts behind a streaming tool (Kafka) at an overview level.
3Understand the added complexity streaming introduces before reaching for it.