RAG Chatbot with a Custom Knowledge Base
Part of the AI Engineer Roadmap
In short
Almost every company wants a bot that answers from its own documents without making things up. Build one with citations and an honest "I don't know", and you show the exact skill they hire for.
A production-quality RAG system over a real document set — the single most commonly requested AI engineer portfolio piece because it maps directly to real hiring needs.
Think of it like this
The college librarian
Ask a good librarian a question and she doesn't guess from memory. She walks to the right shelf, pulls the exact book, opens the page and answers from it, pointing at the line. A RAG chatbot does the same: fetch the right pages first, then answer only from them.
Watch it happen
One question through the RAG bot
"How many leaves?"
1/7An employee asks about the HR policy in the chat window.
The big ideas
What you need to know
Use real, messy documents
PDFs with tables, scanned pages and old versions. Handling them well is the project, not the chat UI.
Citations on every answer
Show which chunk each answer came from. Users trust it more, and you can debug wrong answers.
Say "not found" honestly
When retrieval finds nothing relevant, the bot should say so instead of inventing an answer.
An eval set of 30+ questions
Write questions with known answers. Measure retrieval hits and answer accuracy every time you change something.
See it in code
Walk through the code
1from sentence_transformers import SentenceTransformer2import numpy as np3 4model = SentenceTransformer("all-MiniLM-L6-v2")5chunks = ["Leave: 24 days per year...", "WFH: 2 days a week...", "Payroll runs on the 28th..."]6vecs = model.encode(chunks, normalize_embeddings=True)7 8def retrieve(question, k=2):9 q = model.encode([question], normalize_embeddings=True)[0]10 scores = vecs @ q # cosine similarity, since vectors are normalised11 top = np.argsort(-scores)[:k]12 return [(chunks[i], float(scores[i])) for i in top]13 14context = retrieve("How many leaves do I get?")15prompt = f"Answer only from this context and cite it. If it isn't there, say so. Context: {context}"1/3Embed every chunk once, ahead of time. Normalised vectors make a dot product equal cosine similarity.
Retrieval in its smallest form. Once this works, swap the list for a vector database and add chunking, metadata and a reranker.
Save yourself the pain
Common mistakes
Oops
Judging the bot by chatting with it a few times and feeling good.
Do this instead
Build a fixed test set of questions and expected answers. Rerun it after every change to chunking, prompts or models.
Oops
Blaming the LLM for every wrong answer.
Do this instead
First check whether the right chunk was even retrieved. Most RAG failures are retrieval failures.
Your mission
Milestone one: answer from five PDFs
Quick check
Your RAG bot gives a wrong answer. What should you check first?
Go deeper
- 1Use a genuinely useful knowledge base (docs, a book, a company's public content).
- 2Add source citations and a clear 'I don't know' fallback for unanswerable questions.
- 3Include a short write-up of your chunking, retrieval and evaluation choices.
More projects at this stop
End-to-End ML Pipeline with Deployment
Fine-Tuned LLM for a Niche Domain
AI Agent that Automates a Real Workflow
Computer Vision App (e.g. Defect Detection)
Multi-Modal AI Application
AI-Powered SaaS MVP
Kaggle Competition Submission
Stuck on this topic? Ask an Insider
Get 1:1 guidance from people who've walked this exact path — on the InsideEdge app.
