Stop 14 · Retrieval-Augmented Generation & Vector DatabasesTopics · 6/9

Reranking

In short

Vector search is fast but rough. A reranker reads the top 20 or 50 results closely and puts the truly best ones first. It's often the biggest quality boost for the least effort.

Using a second, more precise model to re-score and reorder an initial set of retrieved candidates — improves final result quality beyond what fast approximate retrieval alone provides.

The blob selector in a cap watches a short row of players through binoculars, with a crowd behind.

Think of it like this

The cricket selector

Selectors can't watch every player in the country closely. They shortlist 30 from scores and reports, quickly. Then they watch those 30 carefully in the nets and pick the final XI. Retrieval is the shortlist. Reranking is the careful watch.

Watch it happen

Reordering the top 5

1/5Vector search returns five chunks in this order for 'hostel fee refund rules'.

The big ideas

What you need to know

Bi-encoder: fast, rough

Embedding search encodes query and chunks separately. Fast across millions, but misses fine detail.

Cross-encoder: slow, sharp

A reranker reads the query and chunk together, so it understands how well one answers the other.

Retrieve many, keep few

Fetch 20 to 50 with vector or hybrid search, rerank, then send only the top few to the LLM.

Watch latency

Reranking adds time per chunk. Limit how many you rerank, or use a hosted rerank API.

See it in code

Walk through the code

python

1# pip install sentence-transformers2from sentence_transformers import CrossEncoder3 4reranker = CrossEncoder("cross-encoder/ms-marco-MiniLM-L-6-v2")5 6question = "hostel fee refund rules"7chunks = ["Mess timings are 7 to 9 am...", "Hostel fees are due by July...",8          "Refunds of hostel fees are given if you leave before...", "Wi-Fi rules...",9          "Refund requests must be filed within 30 days..."]10 11scores = reranker.predict([(question, c) for c in chunks])12ranked = sorted(zip(scores, chunks), reverse=True)13top = [c for _, c in ranked[:3]]

1/3Load a cross-encoder trained on search relevance data.

A small open cross-encoder reranker. It scores each (question, chunk) pair directly.

Save yourself the pain

Common mistakes

Oops

Retrieving only 3 chunks, then reranking those 3.

Do this instead

Rerank a larger pool, like 20 to 50, so the reranker can find good chunks the first stage ranked low.

Oops

Reranking thousands of chunks per query.

Do this instead

Cross-encoders are slow. Use fast retrieval to narrow down first.

Your mission

See reranking help

~30 min

Quick check

Why is a cross-encoder reranker usually more accurate than plain embedding similarity?

Go deeper

1Add a reranking step (e.g. a cross-encoder model) after initial vector retrieval.
2Compare top-result quality with and without reranking.
3Learn the latency/quality trade-off rerankers introduce.