Stop 19 · System Design for AI ApplicationsTopics · 5/9

Designing a RAG System at Scale

In short

RAG over 50 PDFs is a weekend project. RAG over 5 million documents with permissions, daily updates and sub-second answers is a system design interview. Here's how the big version fits together.

A classic AI system design interview question — architecting a RAG pipeline that stays accurate and fast as the document corpus and query volume both grow large.

The blob pulls folders from tall rows of archive drawers stacked high in a big records room.

Think of it like this

The newspaper archive room

A newspaper archive files every day's clippings by topic and date. A researcher asks a question, the archivist pulls the ten best clippings, checks the most relevant, and hands over three. New papers are filed every night. Large-scale RAG is that archive, run by software.

Watch it happen

One question through a production RAG stack

User

RAG API

LLM

Intern leave policy?

1/6The request comes with the user's id, so the system knows which documents they may see.

The big ideas

What you need to know

Ingestion is a pipeline

New and edited documents are chunked, embedded and indexed continuously. Deleted ones must leave the index too.

Filter by permission

Store access info with each chunk and filter at search time. Never let the LLM see documents the user can't.

Hybrid search plus rerank

Combine vector and keyword search for recall, then rerank a few dozen results for precision.

Evaluate retrieval separately

Track whether the right chunk was retrieved, not just whether the final answer looked good.

See it in code

Walk through the code

python

1def rrf(rankings, k=60):2    scores = {}3    for ranking in rankings:4        for rank, doc_id in enumerate(ranking):5            scores[doc_id] = scores.get(doc_id, 0) + 1 / (k + rank + 1)6    return sorted(scores, key=scores.get, reverse=True)7 8vector_hits = ["d7", "d2", "d9", "d4"]9keyword_hits = ["d2", "d5", "d7", "d1"]10print(rrf([vector_hits, keyword_hits])[:3])  # ['d2', 'd7', 'd5']

1/3Each list adds 1/(k + rank) for every document. Higher rank, bigger share.

Reciprocal rank fusion merges vector and keyword results without tuning score scales. Documents high on both lists win.

Save yourself the pain

Common mistakes

Oops

Filtering permissions after the LLM has already written the answer.

Do this instead

Filter at retrieval time. Once restricted text is in the prompt, it can leak into the answer.

Oops

Never removing deleted or outdated documents from the index.

Do this instead

Ingestion must handle updates and deletes, or the bot keeps quoting old policies.

Your mission

Design it on one page

~40 min

Quick check

Why add a reranker after retrieving the top 50 chunks?

Go deeper

1Practice designing a RAG system on paper: ingestion, indexing, retrieval, generation, caching.
2Think through how you'd scale each component independently as load grows.
3Consider freshness — how do you keep the knowledge base updated without full re-indexing?