The notes of Justin Abrahms

Recently updated

Using LLMs as a Reranker for RAG A Practical Guide

2 min read

Using LLMs as a Reranker for RAG: A Practical Guide

rw-book-cover

Metadata

  • Author: Dhruv Patel
  • Full Title: Using LLMs as a Reranker for RAG: A Practical Guide
  • Category: articles
  • Summary: This post explains how using large language models (LLMs) as rerankers improves retrieval-augmented generation (RAG) by selecting the most relevant passages. The author describes engineering techniques like parallel pointwise reranking to speed up processing and reduce errors. Their approach boosts answer quality significantly, but added latency led them to train a custom reranker based on LLM guidance.
  • URL: https://fin.ai/research/using-llms-as-a-reranker-for-rag-a-practical-guide/

Highlights

  • reranking, which reorders the results from vector search so the final answer is grounded on the most relevant passages. (View Highlight)
  • There are three ways to prompt the LLM for reranking [arxiv]: β€’ Pointwise reranking: ask the LLM to rate how relevant each passage is on a scale from 1 to 10. Example output with passage ids: [("id0", 6), ("id1", 10), ("id2", 5), ("id3", 10), ("id4", 4)] β€’ Listwise reranking: ask the LLM to order the passages by relevance. Example output: "id1" > "id3" > "id0" > "id5" > "id2" > "id4" β€’ Pairwise reranking: build a ranking through pairwise comparisons, asking the model which of two passages (𝑝𝑖p_i or 𝑝𝑗p_j​) is more relevant to the query. If you implement the LLM as a comparator inside a sort, you typically need 𝑂(𝐾log𝐾)O(K \log K) or 𝑂(𝐾2)O(K^2) comparisons, making it the most expensive. Studies often find pairwise prompting best on quality, though at higher cost [arxiv]. (View Highlight)
  • When cutting latency, reducing output tokens is a good first step (View Highlight)
  • switched to a Dict format instead of List[Tuple]. (View Highlight)
    • Note: I’m honestly a little surprised that there aren’t efficiencies b/c (" should tokenize to the same thing over and over again.
  • thresholding: instructed the LLM to omit passage ids if their score is below 5. (View Highlight)
  • Vector search already imposes an ordering bias (higher semantic similarity first). If you naively split passages into consecutive chunks, for example the first 𝐾𝑁\frac{K}{N} for worker 1, the next 𝐾𝑁\frac{K}{N} for worker 2, etc., you’ll overweight the first shard with the β€œbest” candidates. We instead assign round-robin by index (View Highlight)

Graph View