Retrieval-Augmented Generation (RAG) is often described as a simple architecture:

Documents → Embeddings → Vector Database → LLM → Answer

That description is useful for understanding the concept, but it hides the real engineering challenge.

A production RAG system is not one component. It is a multi-stage retrieval and generation pipeline, and every stage can introduce failure.

That is why evaluating only the final answer is not enough.

RAG quality must be evaluated end-to-end because a wrong answer can originate anywhere in the pipeline.

1. RAG Is a Pipeline, Not a Single Model

A typical RAG system looks like:

Documents
    ↓
Ingestion
    ↓
Chunking
    ↓
Embedding
    ↓
Vector Storage
    ↓
Query Embedding
    ↓
Retrieval
    ↓
Top-K / Filtering
    ↓
Context Construction
    ↓
Prompt Grounding
    ↓
LLM Generation
    ↓
Final Answer

There are multiple opportunities for failure.

For example:

• The document parser may extract incorrect text.
• Chunking may separate information that belongs together.
• The embedding model may fail to capture the relevant semantics.
• Retrieval may return irrelevant chunks.
• Top-K may exclude the correct chunk.
• Context construction may discard useful information.
• The LLM may misunderstand the retrieved context.
• The model may generate unsupported claims.

If you only evaluate the final answer, you know something went wrong, but you don't necessarily know where.

2. The Most Important Distinction: Retrieval vs Generation

Consider this question:

"How many days of annual leave do employees receive?"

Suppose the knowledge base contains:

"Employees receive 20 days of annual leave per year."

But the retriever returns:

Employee Benefits
Holiday Calendar
Office Policies

The correct policy wasn't retrieved.

The LLM receives incorrect context:

Question
   +
Wrong Context
   ↓
LLM
   ↓
Wrong Answer

This is a retrieval failure.

Now consider a different situation.

The retriever correctly returns:

"Employees receive 20 days of annual leave per year."

But the LLM answers:

"Employees receive 25 days."

This is a generation failure.

Both systems produced an incorrect answer, but the engineering fixes are completely different.

That's why:

Final-answer accuracy alone cannot diagnose RAG failures.

3. Evaluate Retrieval Independently

Before evaluating the LLM, evaluate the retriever.

Suppose you have a test dataset:

Question:
How many days of annual leave do employees receive?

Relevant document:
annual-leave-policy.pdf

Run the query through your retriever.

If the relevant chunk appears in the top 5:

Top 5

1. Employee Benefits
2. Holiday Calendar
3. Annual Leave Policy  ← Relevant
4. Security Policy
5. Office Policy

then the retriever successfully found the evidence.

If it doesn't:

Top 5

1. Holiday Calendar
2. Employee Benefits
3. Office Policy
4. Security Policy
5. IT Policy

retrieval failed.

Useful retrieval metrics include:

Recall@K

Did the relevant document appear in the top K?

Recall@5 = 1

if the relevant chunk appears somewhere in the first five results.

Precision@K

How many of the retrieved documents are actually relevant?

MRR

Mean Reciprocal Rank measures how high the first relevant result appears.

NDCG

Normalized Discounted Cumulative Gain evaluates ranking quality when multiple documents have different relevance levels.

These metrics let you answer:

Is my retriever actually finding the right information?

4. Retrieval Quality Is Not Just Vector Similarity

A common mistake is to look at similarity scores and assume:

0.92 = good
0.45 = bad

That isn't universally true.

Similarity scores depend on factors such as:

• Embedding model
• Corpus
• Vector normalization
• Chunking strategy
• Query characteristics
• Distance metric

A score of 0.82 might be excellent in one system and mediocre in another.

Therefore, retrieval quality should be evaluated against a labeled evaluation dataset, not arbitrary similarity thresholds.

For example:

Query                         Expected Chunk
------------------------------------------------
Forgot password               Password Recovery
Change subscription           Billing Policy
Employee leave                Leave Policy
API authentication            API Security

Then measure retrieval performance against those expectations.

5. Chunking Needs Evaluation Too

Imagine this document:

Employees receive 20 days of annual leave per year. Employees must complete six months of service before becoming eligible.

A poor chunking strategy could produce:

Chunk 1:
Employees receive 20 days...

Chunk 2:
Employees must complete six months...

Depending on the query, retrieving only one chunk may provide incomplete information.

For example:

"How much leave do I receive and when am I eligible?"

The retriever might find only:

"Employees receive 20 days..."

The answer is incomplete because the eligibility condition was separated into another chunk.

This means chunking should be evaluated through its impact on retrieval.

You can experiment with:

• Fixed-size chunks
• Sentence-based chunks
• Heading-aware chunks
• Overlapping chunks
• Semantic chunks

Then measure which strategy produces better retrieval results.

6. Context Relevance Matters

Even when retrieval returns documents, the retrieved context may contain too much irrelevant information.

Suppose Top-K returns:

1. Annual Leave Policy       ✓
2. Employee Benefits         ✓
3. Security Policy           ✗
4. Office Parking            ✗
5. IT Support                ✗

The LLM now has to reason through unnecessary information.

This can increase:

• Context length
• Latency
• Token cost
• Confusion
• Risk of unsupported generation

Therefore, evaluate:

Is the retrieved context actually relevant to the question?

This is sometimes called context relevance.

7. Context Sufficiency Is Different From Relevance

A context can be relevant but still incomplete.

For example:

Context:

Employees receive 20 days of annual leave.

Question:

"How much leave do I receive and when can I use it?"

The context is relevant.

But it doesn't contain the eligibility information.

So we need another question:

Does the retrieved context contain enough information to answer the question correctly?

This is closer to context recall / context sufficiency.

A strong RAG evaluation therefore asks both:

Is the context relevant?
        +
Does the context contain the required evidence?

8. Groundedness / Faithfulness

Now we reach the generation stage.

Suppose the retrieved context says:

"Employees receive 20 days of annual leave."

The LLM responds:

"Employees receive 20 days of annual leave and can carry over up to 10 unused days."

If the context never mentioned carry-over, the second claim is unsupported.

The answer may sound reasonable, but it isn't grounded in the retrieved evidence.

This is a groundedness / faithfulness problem.

The evaluation question becomes:

Are the claims in the answer supported by the retrieved context?

This is fundamentally different from asking whether the answer sounds good.

9. Answer Correctness

An answer can be grounded but still fail to answer the user's question properly.

For example:

Question:

"How many days of annual leave do employees receive?"

Context:

"Employees receive 20 days of annual leave."

Answer:

"The employee handbook contains an annual leave policy."

The answer is technically grounded in the context, but it doesn't actually answer the question.

Therefore we also need:

Answer correctness / relevance

The evaluation should consider:

• Is the answer factually correct?
• Does it answer the question?
• Is it complete?
• Is it appropriately concise?
• Does it follow the expected format?

10. Source Attribution Must Be Evaluated

If your RAG system displays citations:

Answer:
Employees receive 20 days of annual leave.

Source:
Employee Handbook — Page 42

you should evaluate whether the citation actually supports the claim.

A dangerous failure is:

Answer:
Employees receive 20 days of annual leave.

Source:
Security Policy — Page 12

The answer might happen to be correct, but the attribution is wrong.

Therefore, source attribution should be evaluated separately:

Does the cited source actually support the generated claim?

11. End-to-End Evaluation

Now we can define a complete evaluation framework.

                 RAG Evaluation
                       │
        ┌──────────────┴──────────────┐
        ↓                             ↓
   Retrieval                       Generation
        │                             │
   Recall@K                      Answer Correctness
   Precision@K                   Groundedness
   MRR                           Completeness
   NDCG                          Citation Accuracy
        │                             │
        └──────────────┬──────────────┘
                       ↓
                End-to-End Quality

You want to know both:

Did we retrieve the right evidence?

and:

Did the LLM use that evidence correctly?

12. Build an Evaluation Dataset

The most valuable investment in a RAG system is often not another model.

It's a good evaluation dataset.

Create something like:

{
  "question": "How many days of annual leave do employees receive?",
  "expected_sources": [
    "annual-leave-policy"
  ],
  "expected_answer": "20 days"
}

Build perhaps:

100–500 representative questions

covering:

• Easy questions
• Ambiguous questions
• Multi-hop questions
• Questions with paraphrasing
• Questions with missing information
• Questions requiring multiple documents
• Out-of-domain questions

This becomes your RAG evaluation set.

13. Test Retrieval Before Generation

For every evaluation question:

Question
   ↓
Retriever
   ↓
Top-K

Record:

Question
Expected Source
Retrieved Sources
Rank
Similarity

For example:

Now you know exactly where retrieval is failing.

14. Then Test Generation

Once retrieval is good:

Question
+
Retrieved Context
↓
LLM
↓
Answer

Evaluate:

Correctness

Is the answer correct?

Groundedness

Are the claims supported?

Completeness

Did it include all necessary information?

Citation accuracy

Do the citations support the claims?

This isolates generation quality from retrieval quality.

15. Then Test the Entire System

Finally:

Question
 ↓
Retrieval
 ↓
Context
 ↓
LLM
 ↓
Answer

Measure the complete user experience.

For example:

End-to-end answer accuracy = 87%

Retrieval Recall@5 = 94%

Groundedness = 96%

Citation accuracy = 91%

Now you have a much more useful picture.

Perhaps your retriever is excellent, but the final answer accuracy is only 87%.

That tells you:

Retrieval isn't necessarily the bottleneck. Investigate generation or context construction.

16. Evaluate Failure Cases, Not Just Averages

A system reporting:

Accuracy = 90%

can hide serious problems.

Suppose:

Easy questions       98%
Complex questions    71%

The average hides the important weakness.

Break evaluation down by category:

Simple factual       97%
Multi-document        84%
Paraphrased           92%
Long-context          78%
No-answer             69%
Access-controlled     81%

Now you know where engineering work is needed.

17. Retrieval Failure vs Generation Failure

This is perhaps the most useful debugging framework.

Case 1

Retriever ❌
LLM ✓

The correct evidence wasn't retrieved.

Focus on:

• Chunking
• Embeddings
• Metadata
• Query transformation
• Top-K
• Reranking
• Hybrid search

Case 2

Retriever ✓
LLM ❌

The evidence was available, but generation failed.

Focus on:

• Prompt design
• Context ordering
• Model selection
• Structured outputs
• Grounding instructions
• Context length

Case 3

Retriever ❌
LLM ❌

Fix retrieval first.

There is little value in optimizing generation while the model receives bad evidence.

18. Evaluation Should Also Include Production Metrics

Quality isn't the only dimension.

A production RAG system also needs:

Latency

Retrieval: 80 ms
LLM: 1.8 sec
Total: 1.95 sec

Cost

Track:

• Embedding costs
• Input tokens
• Output tokens
• Retrieval infrastructure
• Reranking
• Cache hit rates

Reliability

Track:

• API failures
• Timeouts
• Rate limits
• Empty retrieval
• Model failures

Observability

Log enough information to reconstruct failures:

request_id
query
retrieval results
scores
model
prompt version
latency
tokens
answer
sources

Without this, debugging production RAG becomes extremely difficult.

19. Evaluation Creates an Engineering Feedback Loop

The real value of evaluation isn't producing a score.

It's enabling iteration.

For example:

Evaluation
    ↓
Recall@5 = 72%
    ↓
Investigate retrieval
    ↓
Improve chunking
    ↓
Recall@5 = 84%
    ↓
Add metadata filtering
    ↓
Recall@5 = 91%
    ↓
Add reranking
    ↓
Recall@5 = 95%

Then:

Groundedness = 88%
    ↓
Improve context construction
    ↓
Groundedness = 94%

Now you're doing engineering based on evidence, rather than changing models based on intuition.

20. The Most Common RAG Evaluation Mistake

One of the biggest mistakes is:

"The answer looks good, so the RAG system works."

A few manually tested examples are not an evaluation strategy.

LLMs can produce answers that sound extremely convincing.

You need systematic testing.

The correct mindset is:

Don't ask:
"Does this answer look good?"

Ask:
"Why did this answer happen?"

Trace:

Question
   ↓
What did we retrieve?
   ↓
Was it relevant?
   ↓
Did we retrieve the required evidence?
   ↓
What context did we send?
   ↓
What did the LLM generate?
   ↓
Were the claims grounded?
   ↓
Were the sources correct?

21. A Practical RAG Evaluation Framework

For a production system, I would think about evaluation across these layers:

This is what end-to-end RAG evaluation really means.

Final Takeaway

RAG isn't a single model that you can evaluate with one accuracy number.

It is a chain:

Ingestion
   ↓
Chunking
   ↓
Embedding
   ↓
Retrieval
   ↓
Context
   ↓
Generation
   ↓
Attribution

A failure anywhere in that chain can affect the final answer.

Therefore:

Don't evaluate only the answer. Evaluate the path that produced the answer.

The most effective RAG teams don't simply ask:

"Which LLM gives the best answer?"

They ask:

"Where does our system fail, why does it fail, and which stage should we improve?"

That shift—from model-centric evaluation to pipeline-centric evaluation—is one of the most important steps from building an RAG demo to engineering a production RAG system.

Tags: #RAG, #Evaluation

Share: LinkedIn, Twitter, Email