Stop 06 · Python for Data AnalysisTopics · 12/12

Basic Web Scraping (Overview)

In short

Sometimes the data you need is only on a web page, with no download button and no API. Scraping reads the page's HTML and pulls out the parts you want, if you're allowed to.

Extracting data directly from web pages (using tools like BeautifulSoup) when no API is available — a fallback technique, and one to use carefully and ethically.

The jaadu copies details from a notice board full of pinned papers into a diary.

Think of it like this

Copying from the notice board

Results are pinned on the college notice board. You copy your branch's rows into your diary. Fine. But photographing every board every five minutes would annoy everyone. Scraping is copying from a site's pages, and manners matter just as much.

Watch it happen

A polite scrape, step by step

Allowed?

Fetch

Parse

Select

Save

1/5Check for an API or download first. Read the site's terms and robots.txt. If scraping isn't allowed, stop.

The big ideas

What you need to know

Permission first

Prefer APIs and official downloads. Respect terms of service and robots.txt, and never scrape personal data.

HTML is a tree

Pages are nested tags. Right-click, Inspect in your browser shows the tag and class holding your data.

Tables are the easy case

pd.read_html(url) can grab simple HTML tables straight into DataFrames. Try it before writing a parser.

Be gentle

Add a delay between requests and cache what you fetch. Many sites also block heavy traffic.

See it in code

Walk through the code

python

1import time2import requests3import pandas as pd4from bs4 import BeautifulSoup5 6url = "https://example.com/prices"7html = requests.get(url, timeout=10, headers={"User-Agent": "student-project"}).text8soup = BeautifulSoup(html, "html.parser")9 10rows = []11for item in soup.select("div.product"):12    rows.append({"name": item.select_one("h2").get_text(strip=True),13                 "price": item.select_one(".price").get_text(strip=True)})14time.sleep(2)15pd.DataFrame(rows).to_csv("prices.csv", index=False)

1/4Fetch the HTML with a timeout and an honest user agent.

A minimal scraper for a page you're allowed to scrape. example.com and the selectors are placeholders; real pages need their own.

Save yourself the pain

Common mistakes

Oops

Scraping a site whose terms forbid it, or collecting people's personal details.

Do this instead

Check terms and robots.txt, use the API if one exists, and leave personal data alone.

Oops

Expecting the scraper to work forever.

Do this instead

Sites change their HTML. Keep selectors in one place and check output each run.

Your mission

Grab a public table

~20 min

Quick check

A site offers a free API and also shows the same data on its pages. What should you use?

Go deeper

1Try a simple scraping tutorial on a page that explicitly allows it.
2Learn to always check a site's robots.txt and terms of service before scraping.
3Understand why an API (Phase 6's previous topic) is almost always preferable when available.