Vector Search Isn't Always Enough for RAG

Vector search is powerful because it retrieves documents based on semantic meaning, not just exact words.

But semantic similarity doesn't always mean retrieval relevance.

Consider the query:

"AWS cost reduction"

A semantic search might retrieve documents about:

• Cloud optimization
• AWS architecture best practices
• Resource utilization
• Infrastructure efficiency

These are semantically related.

But a keyword search might find an exact document titled:

"AWS Cost Reduction Strategies"

Neither approach is universally better.

Keyword search is strong at exact terms, product names, error codes, and technical terminology.

Semantic search is strong when users express the same intent using different words.

That's why production RAG often combines both:

Query
  ↓
Keyword Search + Semantic Search
  ↓
Hybrid Retrieval
  ↓
Candidate Documents
  ↓
Reranking
  ↓
Best Context
  ↓
LLM

Reranking takes the candidate documents retrieved by keyword/semantic search and scores them again based on how relevant each document is to the specific query. It then moves the most useful documents to the top.

For example:

Hybrid Retrieval → 20 candidates
                     ↓
                  Reranker
                     ↓
                 Top 3–5
                     ↓
                    LLM

So retrieval becomes a two-stage process: retrieve broadly, then rank precisely.

The goal of retrieval isn't to find similar documents.

The goal of retrieval isn't to find similar documents. It's to find the most useful context for answering the question.

Tags: #RAG, #Semantic, #ReRanking

Share: LinkedIn, Twitter, Email