Stop 08 · Data Cleaning & WranglingTopics · 8/9

ETL Basics (Extract, Transform, Load)

In short

Every dashboard you've used sits on top of a pipeline that moves data from where it's created to where it's analysed. Understanding ETL helps you debug numbers and talk to data engineers.

The standard pattern for moving data from a source system into an analysis-ready destination — extracting raw data, transforming/cleaning it, then loading it somewhere queryable.

The jaadu feeds sugarcane into a crusher as juice pours into a glass.

Think of it like this

The sugarcane juice stall

The juice wala takes cane from the pile (extract), crushes it, adds lemon and ginger, strains it (transform), and pours it into your glass (load). Same three steps, every glass, in order. Skip the straining and you get pulp in your juice.

Watch it happen

A nightly sales pipeline

Schedule

Extract

Transform

Validate

Load

1/5At 2 AM a scheduler starts the job, when source systems are quiet.

The big ideas

What you need to know

Extract

Read from sources: databases, APIs, files. Pull only new or changed rows when you can.

Transform

Clean, join, aggregate, apply business rules. This is where your cleaning skills live.

Load

Write into the destination, usually a warehouse. Append new rows or replace a partition.

ELT is common now

Many teams load raw data into the warehouse first, then transform with SQL there, often using dbt.

See it in code

Walk through the code

python

1def extract(day):2    orders = pd.read_sql(f"SELECT * FROM orders WHERE order_date = '{day}'", src_engine)3    payments = pd.read_csv(f"exports/payments_{day}.csv")4    return orders, payments5 6def transform(orders, payments):7    orders = clean(orders)8    df = orders.merge(payments, on="order_id", how="left", validate="one_to_one")9    df["net_revenue"] = df["amount"] - df["discount"].fillna(0)10    return df11 12def load(df, day):13    df.to_sql("fact_orders", dwh_engine, if_exists="append", index=False)14 15load(transform(*extract("2026-10-26")), "2026-10-26")

1/4Extract one day of orders from the database and that day's payments file.

A tiny ETL job as three functions. Real pipelines add logging, retries and a scheduler, but the shape is the same.

Save yourself the pain

Common mistakes

Oops

Rerunning a failed day and appending the same rows twice.

Do this instead

Make loads safe to rerun: delete that day's rows first, or upsert by key.

Oops

Transforming in five different dashboards, each a little differently.

Do this instead

Transform once in the pipeline so every report uses the same definition of revenue.

Your mission

Your first mini pipeline

~40 min

Quick check

In ELT, where does the transformation usually happen?

Go deeper

1Diagram a simple ETL flow for a dataset you've worked with.
2Learn the difference between ETL and the increasingly common ELT (load first, transform after).
3Understand where this connects to the data warehouses covered in Phase 13.