Stop 04 · SQL & Data Engineering BasicsTopics · 7/9

Airflow Basics (Overview)

In short

Once you have five pipelines, someone has to run them in the right order, on time, and retry the ones that fail. Airflow is the tool many teams use for exactly that.

A workflow orchestration tool for scheduling and monitoring data pipelines as DAGs — the industry-standard way to run recurring ETL and ML training jobs.

The purple blob holds a clipboard and watch, directing helpers carrying garlands and lights.

Think of it like this

The wedding planner's schedule

Mehendi must finish before haldi, haldi before the sangeet, and the caterer must arrive before guests. The planner keeps the list, starts each task on time and calls a backup if the decorator doesn't show. Airflow is the planner for data tasks: order, timing and retries.

Watch it happen

One night of the daily_sales DAG

Scheduler

Worker

Web UI

run extract

1/5At the scheduled time, the scheduler sees the first task is ready and sends it to a worker.

The big ideas

What you need to know

A DAG is the plan

A Directed Acyclic Graph lists tasks and which ones depend on which. No loops allowed, so there's always a clear order.

Schedules and dates

Each DAG has a schedule, like @daily. Every run knows which date it's for, so reruns process the right data.

Retries and alerts built in

Set retries and a retry delay per task. Failures are visible in the UI and can trigger alerts.

Pipelines as Python code

DAGs are Python files in Git, so they're reviewed, versioned and tested like any other code.

See it in code

Walk through the code

python

1from datetime import datetime2from airflow.sdk import dag, task   # Airflow 2: from airflow.decorators3 4@dag(schedule="@daily", start_date=datetime(2026, 1, 1), catchup=False)5def daily_sales():6    @task(retries=2)7    def extract() -> str:8        return "/data/raw/sales.csv"9    @task10    def transform(path: str) -> str:11        return path.replace("raw", "clean")12    @task13    def load(path: str) -> None:14        print("loading", path)15    load(transform(extract()))16daily_sales()

1/4The DAG runs daily from 1 Jan 2026. catchup=False skips backfilling old dates.

With the TaskFlow style, calling one task with another's output sets the dependency. Real tasks would read and write files or tables instead of passing paths around.

Save yourself the pain

Common mistakes

Oops

Doing heavy work, like reading a big file, at the top level of the DAG file.

Do this instead

Keep top-level code light. Airflow parses DAG files often, so real work belongs inside tasks.

Oops

Passing whole DataFrames between tasks.

Do this instead

Pass small things like file paths or table names. Save the data to storage in one task and read it in the next.

Your mission

Run your first DAG locally

~45 min

Quick check

Why can't an Airflow DAG contain a loop where task C depends back on task A?

Go deeper

1Read Airflow's core concepts: DAGs, tasks, operators, scheduling.
2Run the Airflow quickstart locally and trigger a simple DAG.
3Understand when a project needs an orchestrator vs. a simple cron job.