Lesson 16.34: RAG Evaluation & Retrieval Quality

1. Where RAG Pipelines Fail

In the preceding lesson, we mastered Golden Datasets, Unit Tests, and CI regression prevention. Now, we apply those testing principles to quantitatively audit the performance of an AI Knowledge Assistant's Retrieval-Augmented Generation (RAG) pipeline.

Consider a customer asking: "What is the monthly API limit for the Professional plan?" Your knowledge corpus holds thousands of documents. The definitive truth resides inside an updated pricing specification. But if your retriever returns obsolete pricing tables, marketing brochures, or irrelevant blogs, the downstream model is doomed to fail.

The breakdown is rarely confined to the LLM itself. Failures emerge across retrieval, chunking, embeddings, metadata filtering, reranking, or context assembly. A production RAG pipeline executes through distinct operational stages:

Official accuracy guidance emphasizes: Retrieval quality and generation behavior must be evaluated as decoupled failure domains. Without authoritative context, even frontier models fail; conversely, pristine context does not preclude model reasoning blunders.

2. Decoupling Retrieval vs. Answer Generation Evaluation

Suppose the actual warranty duration for an enterprise product is 24 months. Your vector retriever pulls an obsolete legacy policy stating 12 months.

If the LLM dutifully follows the provided context and answers: "The warranty is 12 months"—observe the outcome: The model's generation is 100% faithful to the context, yet the end-user receives an entirely erroneous answer!

Now examine the reverse scenario: The retriever delivers the pristine 24-month document, yet the model hallucinates: "The warranty is 12 months." Here, retrieval was flawless, but synthesis collapsed.

3. Building a Retrieval Evaluation Dataset

The foundation of empirical retrieval evaluation is a curated benchmark dataset. Each record pairs an inquiry with verified relevant_chunk_ids and explicit Graded Relevance scores.

{"id": "pricing-001", "query": "What is the monthly API limit for the Professional plan?", "relevant_chunk_ids": ["pricing-professional-v3"], "relevance": {"pricing-professional-v3": 3, "pricing-starter-v2": 1, "marketing-brochure-v1": 0}}
{"id": "refund-001", "query": "How many days do customers have to request a refund?", "relevant_chunk_ids": ["refund-policy-v2"], "relevance": {"refund-policy-v2": 3, "returns-faq-v1": 2, "shipping-policy-v1": 0}}
{"id": "security-001", "query": "How do I enable two-factor authentication?", "relevant_chunk_ids": ["account-security-guide-v2"], "relevance": {"account-security-guide-v2": 3, "password-reset-guide-v1": 1}}

Graded Relevance Scale Convention:

Gold Label Curation Best Practices:

• Label Multiple Valid Sources: Both an authoritative contract and an FAQ section might legitimately satisfy a query. Labeling only one unfairly penalizes valid alternate retrievals.
• Version & Temporal Diligence: Mark obsolete or superseded documentation explicitly as irrelevant (Grade 0).
• Unanswerable Inquiry Probes: Include intentional queries with empty relevant lists (relevant_chunk_ids: []) to benchmark abstention accuracy.

4. Recall@k: How Much Authoritative Evidence Was Retrieved?

Recall@k quantifies the proportion of all known ground-truth relevant chunks ($R$) that your retriever successfully captured within its top-$k$ returned candidate list ($K_k$).

Suppose an exhaustive answer requires evidence dispersed across 4 authoritative chunks:

Your vector search engine queries Qdrant and returns the top 5 ($k=5$) chunks:

Within $K_5$, two authoritative items were captured: $A$ and $B$ (while $X, Y, Z$ represent irrelevant distractors).

When Does Recall Take Priority?

• Preventing Evidence Omission: Auditing whether crucial clauses vanish before reaching context assembly.
• Benchmarking Query Rewriters: Confirming whether query expansion actually surfaces missing ground-truth chunks.
• Validating Chunk Boundary Overlap: Detecting whether facts were split and destroyed across arbitrary chunk boundaries.

Trade-off: Inflating $k$ (e.g. from $k=5$ to $k=15$) monotonically boosts recall. However, it pumps distractor noise into the prompt, inflates token spend, increases latency, and degrades synthesis quality.

5. Precision@k: What Fraction of Retrieved Chunks is Signal?

Precision@k calculates the proportion of retrieved chunks within the top-$k$ set that are genuinely relevant to answering the query.

Revisiting our previous candidate set ($A, X, B, Y, Z$), exactly 2 chunks ($A$ and $B$) are authoritative.

6. Mean Reciprocal Rank (MRR): Where Does the First Hit Land?

Not all queries require ingesting ten dense chunks. In direct Q&A scenarios (e.g. FAQ lookups, support tickets), the paramount operational question is: At what position does the very first authoritative hit appear?

Consider the ranked candidate stream returned by a vector database:

• Rank 1: Irrelevant marketing landing page
• Rank 2: Irrelevant community blog post
• → Rank 3: Authoritative pricing specification chunk A
• Rank 4: Helpful supplementary chunk B

Here, the rank of the first authoritative match is 3.

Mean Reciprocal Rank (MRR):

Across a benchmark evaluation suite of $N$ queries, the arithmetic mean of all individual reciprocal ranks yields the MRR:

Worked Example: For 3 queries with RR values of 1.0, 0.5, and 0.0, the aggregate MRR = (1.0 + 0.5 + 0.0) / 3 = 0.50.

7. Why NDCG is Essential: Measuring Ranking Trajectory

While Recall and MRR provide critical insights, they suffer from structural blind spots: they cannot evaluate the holistic rank distribution or differentiate between varying shades of graded relevance.

Consider two competing search engines (Retriever A and Retriever B) that retrieve the identical 3 chunks, but present them in radically different ordering:

Notice: Recall@3 for both systems is identical (100%)! Yet Retriever A places the most actionable document at the apex, whereas Retriever B introduces cognitive friction. To quantify this discrepancy, we use Normalized Discounted Cumulative Gain (NDCG).

8. DCG & NDCG Mathematical Formulation

NDCG relies on two complementary mathematical transformations:

1. Exponential Gain: The term $(2^{rel_i} - 1)$ exponentially rewards top-grade authoritative documents.
2. Logarithmic Discount: The denominator $\\log_2(i + 1)$ progressively penalizes evidence discovered late in the list.

9. Python Implementation: Project Hierarchy

Rather than introducing heavyweight external black-box dependencies, standardizing retrieval metrics in a zero-dependency, pure Python module delivers auditable, transparent verification logic.

byteion-ai-backend/
├── app/
│   └── retrieval/
│       ├── metrics.py              # Pure Python mathematical metrics
│       └── service.py              # Qdrant & BM25 Retrieval Service
└── tests/
    └── evals/
        ├── datasets/
        │   └── retrieval_v1.jsonl  # Ground truth queries & relevance
        ├── test_metrics.py         # Unit tests verifying metric math
        └── run_retrieval_eval.py   # Benchmark evaluation runner

10. Retrieval Metric Functions Implementation

The following pure Python module implements Recall@k, Precision@k, Reciprocal Rank, DCG@k, and NDCG@k with comprehensive boundary protections:

from collections.abc import Sequence
from math import log2

def recall_at_k(
    retrieved_ids: Sequence[str],
    relevant_ids: set[str],
    k: int,
) -> float:
    if k <= 0:
        raise ValueError("k must be strictly greater than zero")
    if not relevant_ids:
        raise ValueError("Recall is undefined for an empty relevant ground-truth set")
    top_k = retrieved_ids[:k]
    hits = set(top_k) & relevant_ids
    return len(hits) / len(relevant_ids)

def precision_at_k(
    retrieved_ids: Sequence[str],
    relevant_ids: set[str],
    k: int,
) -> float:
    if k <= 0:
        raise ValueError("k must be strictly greater than zero")
    top_k = retrieved_ids[:k]
    hits = set(top_k) & relevant_ids
    # Fixed-k convention: missing results are penalized as non-relevant
    return len(hits) / k

def reciprocal_rank(
    retrieved_ids: Sequence[str],
    relevant_ids: set[str],
) -> float:
    for rank, chunk_id in enumerate(retrieved_ids, start=1):
        if chunk_id in relevant_ids:
            return 1.0 / rank
    return 0.0

def dcg_at_k(
    retrieved_ids: Sequence[str],
    relevance: dict[str, int],
    k: int,
) -> float:
    if k <= 0:
        raise ValueError("k must be strictly greater than zero")
    score = 0.0
    for rank, chunk_id in enumerate(retrieved_ids[:k], start=1):
        rel = relevance.get(chunk_id, 0)
        if rel < 0:
            raise ValueError("Relevance scores cannot be negative")
        gain = (2**rel - 1) / log2(rank + 1)
        score += gain
    return score

def ndcg_at_k(
    retrieved_ids: Sequence[str],
    relevance: dict[str, int],
    k: int,
) -> float:
    actual_dcg = dcg_at_k(retrieved_ids, relevance, k)
    ideal_ids = sorted(relevance, key=lambda c_id: relevance[c_id], reverse=True)
    ideal_dcg = dcg_at_k(ideal_ids, relevance, k)
    if ideal_dcg == 0.0:
        return 0.0
    return actual_dcg / ideal_dcg

11. Unit Testing the Metric Mathematics

Before orchestrating large-scale retrieval benchmarks, we must rigorously verify that our measurement math is bug-free. These Pytest assertions confirm that ideal orderings achieve 1.0 and degraded rank orders penalize scores:

import pytest
from app.retrieval.metrics import (
    recall_at_k,
    precision_at_k,
    reciprocal_rank,
    ndcg_at_k,
)

def test_recall_at_k_calculation():
    retrieved = ["A", "X", "B", "Y"]
    relevant = {"A", "B", "C", "D"}
    # Hits in top 3: {"A", "B"} out of 4 total → 2/4 = 0.5
    assert recall_at_k(retrieved, relevant, 3) == pytest.approx(0.5)

def test_precision_at_k_fixed_k():
    retrieved = ["A", "X", "B", "Y"]
    relevant = {"A", "B", "C"}
    # Hits in top 4: {"A", "B"} / 4 = 0.5
    assert precision_at_k(retrieved, relevant, 4) == pytest.approx(0.5)

def test_reciprocal_rank_third_position():
    retrieved = ["X", "Y", "B", "A"]
    relevant = {"A", "B"}
    assert reciprocal_rank(retrieved, relevant) == pytest.approx(1.0 / 3.0)

def test_reciprocal_rank_zero_hits():
    assert reciprocal_rank(["X", "Y"], {"A"}) == 0.0

def test_ndcg_ideal_order_is_one():
    retrieved = ["A", "B", "C"]
    relevance = {"A": 3, "B": 2, "C": 1}
    assert ndcg_at_k(retrieved, relevance, 3) == pytest.approx(1.0)

def test_ndcg_inverted_ranking_penalized():
    relevance = {"A": 3, "B": 2, "C": 1}
    ideal = ndcg_at_k(["A", "B", "C"], relevance, 3)
    inverted = ndcg_at_k(["C", "B", "A"], relevance, 3)
    assert inverted < ideal

12. Retrieval Service Interface Contract

Production retrieval engines must conform to an explicit protocol boundary enforcing tenant boundaries, active document versions, and candidate thresholds:

from typing import Protocol, TypedDict

class RetrievedChunk(TypedDict):
    chunk_id: str
    score: float
    content: str
    source_id: str

class Retriever(Protocol):
    async def search(
        self,
        *,
        tenant_id: str,
        query: str,
        top_k: int,
    ) -> list[RetrievedChunk]:
        """Query vector or hybrid index with strictly enforced tenant filtering."""
        ...

13. Automated Benchmark Runner & Macro Averaging

The benchmark runner ingests retrieval_v1.jsonl, computes individual inquiry metrics, and compiles aggregate Macro Averages.

Why Macro Averaging? Macro averaging assigns equal mathematical weight to every distinct user inquiry, ensuring high-volume document questions do not skew systemic evaluation results.

import json
from pathlib import Path
from statistics import mean
from app.retrieval.metrics import (
    recall_at_k,
    precision_at_k,
    reciprocal_rank,
    ndcg_at_k,
)

async def evaluate_retriever(
    retriever,
    dataset_path: Path,
    tenant_id: str,
    top_k: int = 5,
) -> dict:
    cases = []
    with dataset_path.open("r", encoding="utf-8") as file:
        for line in file:
            if line.strip():
                cases.append(json.loads(line))

    per_case = []
    for case in cases:
        results = await retriever.search(
            tenant_id=tenant_id,
            query=case["query"],
            top_k=top_k,
        )
        retrieved_ids = [item["chunk_id"] for item in results]
        relevant_ids = set(case["relevant_chunk_ids"])
        relevance = case["relevance"]

        per_case.append({
            "id": case["id"],
            f"recall@{top_k}": recall_at_k(retrieved_ids, relevant_ids, top_k),
            f"precision@{top_k}": precision_at_k(retrieved_ids, relevant_ids, top_k),
            "rr": reciprocal_rank(retrieved_ids, relevant_ids),
            f"ndcg@{top_k}": ndcg_at_k(retrieved_ids, relevance, top_k),
        })

    return {
        "total_cases": len(per_case),
        f"recall@{top_k}": mean(c[f"recall@{top_k}"] for c in per_case) if per_case else 0.0,
        f"precision@{top_k}": mean(c[f"precision@{top_k}"] for c in per_case) if per_case else 0.0,
        "mrr": mean(c["rr"] for c in per_case) if per_case else 0.0,
        f"ndcg@{top_k}": mean(c[f"ndcg@{top_k}"] for c in per_case) if per_case else 0.0,
    }

14. Context Relevance, Coverage & Groundedness

Even when Recall@5 reaches 0.95, an end-to-end RAG system can fail catastrophically! If the retrieved chunks contain excessive filler, outdated provisions, or duplicate paragraphs, the downstream model easily becomes distracted and hallucinates.

15. Strategy Benchmarking: Dense vs. BM25 vs. Hybrid

Selecting an enterprise retrieval architecture must be grounded in rigorous benchmarking rather than developer intuition. We evaluate competing configurations against the identical ground-truth suite:

16. Ablation Testing Methodology

To isolate the exact marginal gain contributed by each architectural component, conduct Ablation Testing—toggling individual layers sequentially while holding all other parameters constant:

Invariant Principle: Every experiment must evaluate across the identical test suite, tenant boundary filters, and cut-off $k$.

17. Chunking Strategy Evaluation & Trade-offs

Chunk segmentation parameters dictate the granular resolution of your vector space. Granular 256-token chunks isolate razor-sharp facts but sever surrounding context. Conversely, 1024-token blocks preserve narrative continuity but dilute embedding centroids with distractor prose.

18. Query Rewriting & Transformation Evaluation

End-users frequently submit colloquial shorthand (e.g. "Pro plan api limit koto?"). Expanding queries via lightweight LLM prompts into formal inquiries can dramatically improve vector retrieval recall.

However, unchecked query rewriting occasionally mutates core entities or swaps product tiers. Rigorous rewriting evaluation audits 3 indicators:

• Entity & Constraint Preservation: Are product names, dates, and numerical constraints preserved?
• Recall Delta: Does the rewritten variant produce a statistically significant boost in Recall@k?
• Latency Overhead: Does the intermediate rewriting step introduce unacceptable end-to-end latency?

19. Reranker Evaluation & Theoretical Ceilings

Cross-encoder rerankers rescore initial candidates (e.g. top-20) with deep bidirectional attention, propelling authoritative evidence into positions 1 and 2. Evaluate reranker efficacy via Before vs. After NDCG@5 and MRR deltas.

20. Multi-Tenant Security & Access Control Verification

In multi-tenant SaaS architectures, surfacing Tenant B's confidential pricing or financial records in response to Tenant A's inquiry represents a critical security compromise.

Regardless of high Recall@5 or stellar NDCG scores, a single cross-tenant leakage event represents an immediate, absolute release blocker.

import pytest

@pytest.mark.security
async def test_retrieval_never_leaks_cross_tenant_chunks(retriever):
    # Tenant A executes a query that semantically matches Tenant B's private contract
    results = await retriever.search(
        tenant_id="tenant_acme",
        query="What are the confidential severance clauses?",
        top_k=10,
    )

    # Assert 100% of returned chunks strictly belong to caller's verified tenant
    unauthorized = [
        item for item in results
        if item.get("tenant_id") != "tenant_acme"
    ]
    assert len(unauthorized) == 0, f"CRITICAL LEAK: {len(unauthorized)} foreign chunks detected!"

21. Production RAG Benchmark Report Structure

Every RAG pipeline experiment must emit a structured audit artifact to maintain longitudinal optimization telemetry across deployment iterations:

22. Automated Release Gate Policies

Before merging retrieval or embedding changes into the mainline branch, our CI pipeline enforces 4 strict verification gates: