Stop 03 · Search Engine Optimization (SEO)Topics · 1/12

How Search Engines Work

In short

Google doesn't search the whole web when you type. It searches its own giant copy that it built earlier. SEO is about getting into that copy and ranking well inside it.

Search engines crawl the web, index pages, and rank them for a given query using hundreds of signals — understanding crawling, indexing and ranking as distinct steps clears up most SEO confusion.

The blob librarian carries a tall stack of books toward a giant card catalogue cabinet.

Think of it like this

A librarian who reads every new book

Imagine a librarian who visits every bookshop daily, reads new books, files them in a huge catalogue, and when you ask a question, hands you the three best books in seconds. Crawling, indexing and ranking, in one person.

Watch it happen

From a new page to a search result

Crawl

Index

Rank

Serve

1/4Googlebot finds your new page through links and your sitemap, then downloads it.

The big ideas

What you need to know

Crawling

Bots follow links from page to page. A page nothing links to is hard for them to find.

Indexing

Google stores and understands pages. If a page isn't indexed, it can't rank at all.

Ranking

Google uses many signals, like relevance to the query, content quality, links and page experience.

Intent matters most

Google tries to show what the searcher actually wants, not just pages that repeat the words.

See it in code

Walk through the code

text

1# robots.txt at chaibox.in/robots.txt2User-agent: *3Disallow: /cart/4Disallow: /admin/5 6Sitemap: https://chaibox.in/sitemap.xml

1/3These rules apply to all crawlers.

robots.txt tells crawlers which paths not to crawl and where your sitemap lives. It controls crawling, not indexing.

Save yourself the pain

Common mistakes

Oops

Using robots.txt to hide a page from Google results.

Do this instead

A blocked page can still appear if others link to it. Use a noindex tag on a crawlable page instead.

Oops

Thinking Google searches the live web each time.

Do this instead

It searches its index. New pages need to be crawled and indexed first, which can take days.

Your mission

Check what Google has indexed

~15 min

Quick check

A new blog post isn't showing on Google even for its exact title. What's the most likely reason?

Go deeper

1Learn the 3-step process: crawling, indexing, ranking, and what could go wrong at each.
2Check whether a website of yours (or a sample one) is indexed using a 'site:' search.
3Read a beginner-friendly explainer on how Google's ranking algorithm broadly works.