Create a golden set before tuning
A golden set pairs representative queries with expected permitted evidence and labels. Include semantic, lexical identifier, filtered, ambiguous, stale, and no-answer query classes.
Version the corpus snapshot, query set, relevance labels, prompt, retrieval configuration, and judge configuration so an apparent improvement is reproducible.
Measure retrieval and answer behaviour separately
Metrics map to different decisions
| Metric | Use |
|---|---|
| ContextCoverage | Bedrock built-in measure of retrieved context coverage |
| ContextRelevance | Bedrock built-in measure of retrieved context relevance |
| citation and faithfulness | Answer is supported by selected evidence |
| abstention | System declines unsupported requests correctly |
| p50/p95 and cost | Experience and operating trade-offs |
Course-synthesis RAG failure isolation
Loading diagram...
Use Bedrock RAG evaluation as evidence, not a substitute for judgement
Bedrock RAG evaluation metrics can help structure retrieval and generation assessment. AgentCore evaluation types distinguish online, on-demand, and batch evaluation for agent traces; use the mode that matches continuous monitoring, a focused investigation, or a versioned comparison. Treat scores as an investigation signal: review sampled evidence, evaluate authorization and abstention separately, and compare like-for-like corpus and configuration versions.
Debug through versioned experiments
- Classify the failure: source, parse, index, filter, retrieval, context, generation, or presentation.
- Change one stage and rerun the same golden set.
- Record Bedrock evaluation results where applicable, p50/p95 latency, token and infrastructure cost, and failure samples.
- Require a human review path for high-impact queries and no-answer behaviour.
- Promote and roll back complete versioned configurations, not isolated model changes.
Author a golden set that can be replayed and audited
A golden set is an executable agreement about the query, permitted evidence, and expected outcome. It must represent the decisions your retrieval system makes: answer from a particular source, reject an unauthorized source, prefer a current revision, or abstain when evidence is missing.
Keep source IDs and configuration identifiers stable. Store sensitive question text in an appropriately controlled evaluation store; the JSONL shape below is a teaching schema, not an Amazon Bedrock upload contract. A missing label is a data-quality defect, not permission to let an evaluator guess the expected result.
Golden-set JSONL record
One illustrative line for a versioned, permission-aware evaluation dataset.
Click on an annotation to highlight it in the JSON
Evaluate retrieval before evaluating generated answers
Amazon Bedrock supports retrieve-only jobs and retrieve-and-generate jobs. Use retrieve-only first to establish whether the allowed evidence was found and ranked. Then use retrieve-and-generate to inspect whether a response uses that evidence appropriately. This order prevents a fluent generator from masking a retrieval failure.
Two-stage evidence contract
| Stage | Control | Observable result |
|---|---|---|
| Retrieve-only | Freeze corpus, principal, filter, and query | Expected permitted source appears; forbidden source does not |
| Retrieve-and-generate | Freeze prompt and selected evidence | Answer cites and stays within retrieved support or abstains |
| Human sample | Review disagreements and high-impact cases | Labels or product policy are corrected deliberately |
Triage failures into a scorecard and rollback decision
When a case fails, follow the evidence path rather than retuning the model first. If the expected source is absent, inspect ingestion, parsing, chunking, embeddings, index version, query transformation, and filters. If it is present but omitted from context, inspect fusion, reranking, and budget. If it is in context but the answer is unsupported, inspect prompt, generator, citation mapping, and abstention policy.
Failure triage tree
Loading diagram...
A release scorecard should name the baseline and candidate, corpus and golden-set versions, per-slice retrieval outcomes, unsupported-answer samples, authorization failures, latency/cost observations, reviewers, and rollback owner. Roll back the complete versioned configuration if a critical slice regresses or a forbidden source is returned; do not silently patch a single metric.
Primary reference: Amazon Bedrock RAG evaluations
AWS distinguishes retrieve-only from retrieve-and-generate evaluation and writes evaluation reports to the configured output location. Verified 2026-08-25; supported models and Regions are time-sensitive.
AWS Bedrock: evaluate RAG sourcesCalibrate evaluator variance before trusting a trend
An evaluator can rank alternatives usefully while still disagreeing with expert labels on individual cases. Before gating a release, sample cases across positive, no-answer, permission, and ambiguous slices; compare the evaluator verdict with a human rubric; and retain the evaluator model, prompt, metric definition, and report location. A changed evaluator is a changed measurement instrument.
Evaluator calibration record
| Field | Decision use | Failure response |
|---|---|---|
| Evaluator and prompt version | Makes score changes comparable | Invalidate blended comparison |
| Human disagreement sample | Finds systematic false passes/fails | Narrow metric or revise labels |
| Repeated/replayed cases | Observes unstable verdicts | Escalate to human review |
| Slice distribution | Prevents easy cases dominating | Rebalance before promotion |
Amazon Bedrock reports evaluation results through the console or the output S3 location you specify, and stores custom metric definitions with custom-metric outputs. Treat those artifacts as a versioned evidence receipt. They do not remove the need to inspect retrieval logs, deterministic policy assertions, or a representative human review sample.
Pair probabilistic metrics with a deterministic assertion ledger
Use an assertion ledger for invariants that should never depend on an LLM judge. Assertions can prove that a returned citation belongs to a retrieved source, a forbidden source ID is absent, the selected corpus version is expected, a response includes an abstention marker when no evidence is permitted, or a rollout remains reversible. These checks turn evaluation from a score-only activity into a production control.
Deterministic assertion ledger entry
A teaching artifact for a release pipeline; values and thresholds belong to the workload.
Click on an annotation to highlight it in the JSON
Treat RAG evaluation as an architecture pipeline, not a spreadsheet afterthought
AWS’s September 2025 SageMaker AI example makes the delivery shape visible: source preparation, chunking and ingestion, retrieval tools, RAG evaluation, and experiment tracking are connected as one versioned system. It is a source-specific RAGOps implementation; its value for this course is the reproducibility boundary rather than any requirement to adopt its exact frameworks.
- Version the corpus, parsing or chunking policy, embedding route, index, retrieval configuration, prompt, and evaluator together.
- Record retrieval-only evidence before accepting a generated-answer quality result.
- Keep experiments comparable by freezing identity scope and the labelled case set, then review why a candidate changed a result.
- Promote or roll back the complete compatible set rather than hand-editing one metric or model after an incident.
Glossary: terms first introduced here
These are the technical terms introduced in this step. Later modules build on them rather than redefining them from scratch.
New vocabulary for this step
| Term | Plain-language meaning | Why it matters here |
|---|---|---|
| Golden set | A curated set of representative questions with expected evidence, answer behavior, or both. | It turns a claimed retrieval improvement into a repeatable comparison. |
| Ablation | A controlled comparison that removes or changes one component to identify its actual contribution. | It prevents several changed settings from being mistaken for one clear improvement. |
| Faithfulness | Whether a generated answer stays supported by the evidence supplied to it. | A relevant retrieval result does not guarantee that the final answer uses it correctly. |
Knowledge Check
Test your understanding with this quiz. You need to answer all questions correctly to mark this section as complete.