Stop 14 · Retrieval-Augmented Generation & Vector DatabasesTopics · 4/9

RAG Architecture

In short

LLMs don't know your college's rules or yesterday's news. RAG fetches the right passages first and hands them to the model with the question. It's the most common way to build useful AI assistants.

Retrieval-Augmented Generation — retrieving relevant context from a knowledge base and injecting it into a prompt so the model answers grounded in real data instead of memory alone.

The blob in an open-book exam flips to a bookmarked page and writes its answer.

Think of it like this

The open-book exam

In an open-book exam, you don't memorise everything. You find the right pages using bookmarks, read them, then write your answer in your own words. RAG is the model taking an open-book exam: Retrieve the pages, then Generate the answer.

Watch it happen

One question through a RAG app

Student

App

Vector DB

LLM

Min attendance?

1/6A student asks: 'What's the minimum attendance to sit for exams?'

The big ideas

What you need to know

Indexing happens first

Offline: load documents, chunk, embed and store. Online: embed the question, retrieve, then generate.

Fresh knowledge, no retraining

Update the documents and re-index. The model's answers change without any fine-tuning.

Citations build trust

Return sources with each answer so users can check, and so you can debug wrong ones.

Retrieval is the weak link

If the right chunk isn't retrieved, the best LLM can't answer. Most RAG fixes are retrieval fixes.

See it in code

Walk through the code

python

1import chromadb2 3docs = chromadb.PersistentClient(path="./db").get_collection("rulebook")4 5def answer(question: str) -> str:6    hits = docs.query(query_texts=[question], n_results=4)7    context = "\n\n".join(8        f"[p.{m['page']}] {d}" for d, m in zip(hits["documents"][0], hits["metadatas"][0])9    )10    prompt = (f"Answer using only the context. Cite pages like [p.12]. "11              f"If the answer isn't there, say so.\n\n<context>\n{context}\n</context>\n\n"12              f"Question: {question}")13    return ask_llm(prompt)

1/4Open the collection you indexed earlier, with page numbers as metadata.

The online half of RAG in a dozen lines: retrieve, build a grounded prompt, generate. ask_llm is your helper.

Save yourself the pain

Common mistakes

Oops

Blaming the LLM when answers are wrong.

Do this instead

First print the retrieved chunks. Often the right passage was never retrieved.

Oops

Stuffing 30 chunks into the prompt 'just in case'.

Do this instead

More context adds noise and cost. Retrieve fewer, better chunks, and rerank if needed.

Your mission

RAG over one PDF

~45 min

Quick check

A RAG bot gives a wrong answer. What should you check first?

Go deeper

1Build the full RAG loop end-to-end: embed, store, retrieve, inject into prompt, generate.
2Learn where each failure mode (bad retrieval, bad chunking, weak prompt) shows up in outputs.
3Compare RAG output quality against the same model with no retrieved context.