Climb the decision ladder only when the evidence demands it
Agentic RAG is a control loop around retrieval, not a replacement for retrieval engineering. Start with a deterministic cited answer. Add routing when query classes need different retrieval paths, then corrective or reflective loops when evaluation shows a bounded retry can recover a named failure class.
Escalate architecture in response to measured failure modes
| Level | Control | Evidence to advance |
|---|---|---|
| Deterministic RAG | Fixed retrieve, rank, answer, cite path | Baseline fails a known query class |
| Routed retrieval | Classify request to a permitted retrieval strategy | A router improves the labelled class without harming others |
| Corrective loop | Inspect insufficiency, retry a named retrieval action | The retry recovers failures within a bounded budget |
| Agentic retrieval | Plan, act, observe, and stop across approved tools | Multi-step decisions improve an evaluated user outcome |
Make router quality a first-class test subject. Its false-positive cost is unnecessary agent complexity; its false-negative cost is a static answer that may abstain or miss a recoverable multi-step path. Include examples close to the decision boundary, deliberately similar requests with different authorization scopes, and requests whose source state changes during the run. The router should emit a reason code tied to a known query class rather than a free-form rationale. If it cannot classify confidently, choose the safer static path or a review route. This static-first default limits autonomy while preserving a simple baseline that operators can still understand and validate independently.
Define sufficient evidence before a model can stop
A plan should not continue merely because another tool is available. Encode an answer contract: required evidence types, allowed source domains, freshness or version rules, citation granularity, authorization scope, and an abstention condition. The controller can stop when the contract is met, not when a model reports confidence.
Illustrative evidence-sufficiency record
Application-owned control fields, not an AWS response schema.
Click on an annotation to highlight it in the JSON
If sources conflict, preserve both provenance records, apply the declared authority rule, and escalate where the contract requires a human decision. Do not let a reflective prompt silently choose the more fluent source.
Make the plan, act, observe loop bounded and inspectable
A production controller treats planning output as a request for a permitted action, not an instruction to execute. Before every action, validate tool arguments, identity scope, tenant context, data classification, remaining step, time, and cost budgets. Record the observation and the exact reason for the next transition.
Bounded agentic retrieval state machine
Loading diagram...
Authorize the action, preserve the provenance
Identity and authorization travel with every retrieval and tool step. The controller should pass the authenticated principal, tenant or workspace scope, request purpose, and trace identifier to a policy-enforcing tool boundary. A retrieved document is evidence, not permission to call another system or widen a query filter.
Per-step governance record
| Record | Why it exists |
|---|---|
| Principal and tenant scope | Prevents a plan from losing the user or tenant boundary |
| Tool schema and validated arguments | Makes the requested action reviewable and rejects malformed input |
| Authorization decision | Separates policy result from model reasoning |
| Evidence and source version | Supports answer citations, replay, and incident investigation |
| Trace and termination reason | Connects a final answer to the route and budget that produced it |
Redact or minimize sensitive content in traces. Retain enough structured metadata to reconstruct the policy decision and source versions without treating logs as an unrestricted copy of the corpus.
Design recovery before the first write-capable tool call
Classify failures before adding retries. A malformed request, denied authorization, stale index, temporary dependency failure, conflicting evidence, budget exhaustion, and side-effect ambiguity require different handling. Retry only failures that are safe and expected to recover, with a bounded policy and preserved idempotency key.
Recovery design for agentic actions
| Failure class | Controller response |
|---|---|
| Invalid or unauthorized action | Stop, record denial, and request correction or approval |
| Transient read dependency | Bounded retry or a retrieval-only fallback |
| Unknown write outcome | Reconcile using an idempotency key before compensating |
| Evidence conflict | Surface the conflict under the authority rule or escalate |
| Budget exhausted | Return safe partial state, abstain, or hand off without hidden continuation |
Checkpoint durable state before a side effect and make compensation explicit. For example, a provisioning request may need a cancel or review workflow, whereas a failed search should never trigger a compensating write. Human approval is a state transition with identity and rationale, not a free-text prompt response.
Evaluate trajectories as well as answers
An answer can look correct after an unsafe or needlessly expensive route. Evaluate the output against its citations and separately inspect trajectory properties: allowed tools only, authorization outcomes, evidence sufficiency, termination reason, retries, side effects, latency, and cost. Observability gives you traces; evaluation decides whether those traces meet the contract.
- Freeze a labelled workload with simple, ambiguous, denied, unavailable, and high-impact cases.
- Compare deterministic, routed, and agentic variants using the same corpus, policies, and answer contract.
- Review trajectory samples for unneeded calls, policy denials, missing provenance, and unsafe recovery.
- Promote a complete versioned controller configuration only when quality and operational constraints improve together.
- Keep a static retrieval or abstention route available for degraded operation and rollback.
Promote from a static hybrid baseline only for a named failure class
Agentic retrieval is a conditional design choice, not an upgrade. Begin with a versioned static hybrid path that applies permissions, retrieves and reranks evidence, cites sources, and abstains when it cannot meet the answer contract. Classify the requests that fail it: perhaps a question needs a permitted second lookup after finding a conflicting policy, or a durable workflow needs an approved tool sequence. Then compare a static-first router with the proposed agentic loop on the identical corpus snapshot, identity scope, prompt, model, and evaluation cases. Promote only if the additional state and calls improve the declared outcome without causing unacceptable denials, unsafe paths, tail latency, or operating cost.
Static-first promotion gate
| Gate | Decision evidence |
|---|---|
| Problem fit | A labelled failure class requires a bounded multi-step decision, not just better retrieval tuning. |
| Baseline | Static hybrid retrieval is versioned and evaluated on allowed, denied, stale, ambiguous, and unavailable cases. |
| Incremental value | The routed or agentic variant improves the named outcome on the same workload. |
| Operational fit | Termination, authorization, traceability, fallback, and rollback evidence all meet the workload contract. |
Persist a bounded controller state, not hidden model intent
A controller should externalize the state that determines safe behavior: the request class, authenticated scope, allowed actions, evidence already seen, remaining budgets, and why it will transition or stop. The model may propose an action, but a deterministic controller validates it against that state before invocation. Keep the state compact and redact user content when it is not needed for recovery. The AWS basic reasoning pattern describes a repeated reasoning and tool-use loop; production use still needs workload-owned limits, policy enforcement, and an observable stopping condition.
Bounded agentic retrieval state
Illustrative controller payload, not an AWS service response schema.
Click on an annotation to highlight it in the JSON
Test termination and adversarial behavior before widening autonomy
A correct answer on a happy-path demo says little about a controller under pressure. Build a fixed suite that checks prompt-injected retrieved text, a model request for an unlisted tool, forged or cross-tenant identifiers, conflicting authoritative sources, a stale index, repeated dependency timeouts, and an ambiguous write outcome. For every case assert the final route, allowed tools, policy result, maximum calls, terminal reason, surfaced uncertainty, and whether a side effect occurred. A run that ends safely with authorization_denied, budget_exhausted, dependency_unavailable, or human_approval_required is successful when that is the declared contract.
- Run the static hybrid route first and record the evidence it can provide without escalation.
- Allow the router to escalate only for the labelled query class and only when a remaining budget exists.
- Inject a denied action, timeout, conflict, or malicious instruction and assert the controller reaches its declared terminal state.
- Compare result and trajectory with the baseline; retain a static retrieval or abstention fallback for rollback and degraded operation.
AWS pattern for basic reasoning agents
Primary AWS Prescriptive Guidance for a reasoning-and-tool-use loop. Use it with workload-owned authorization, limits, and evaluation rather than assuming an agentic path is universally superior.
Read the basic reasoning agent patternRecover task state and reconcile side effects deterministically
Keep durable task state separate from a conversational transcript. Before a side-effecting action, record the task identifier, authenticated scope, approved action, validated parameters, idempotency key, prior state, and the next safe recovery step. After dispatch, record a receipt or an explicit unknown outcome. On restart, reconcile with the target system using the idempotency key or status query before retrying. This protects against the common failure where a controller restarts after sending a write and repeats it because the model context no longer remembers the first attempt. For read-only retrieval, a bounded retry may be suitable; for writes, recovery must be target-specific.
Task recovery and idempotency state
Application-owned durable state for a potentially side-effecting agent action.
Click on an annotation to highlight it in the JSON
Test restart at each transition: before dispatch, after dispatch but before receipt, after a denial, and after a dependency timeout. The expected result is a single reconciled side effect or a safe terminal handoff, never hidden continuation. Preserve the static retrieval or abstention route so an incident does not force write-capable autonomy.
Calibrate evidence sufficiency against abstention and escalation
A model confidence phrase is not evidence sufficiency. Define observable checks such as required source classes present, permitted citation locations resolved, conflict rule satisfied, and answer scope no wider than retrieved evidence. Then calibrate the controller on held-out cases that include answers, abstentions, and escalations. Measure false completion, unnecessary escalation, unsupported citation, and time or tool budget use by query class. Tune a threshold only after reviewing where it fails: a higher threshold may correctly increase abstention, while a lower threshold can conceal missing evidence. The correct calibration is workload-specific and should be versioned alongside the router and corpus.
Evidence-sufficiency calibration cases
| Case | Expected terminal route | Failure signal |
|---|---|---|
| Two current permitted sources agree | Cited answer | Unnecessary extra tool call. |
| Required source absent | Abstain or retrieve within budget | Confident unsupported answer. |
| Authoritative sources conflict | Surface conflict or human escalation | Fluent silent selection. |
| Source is denied | Denied-safe response | Citation leaks inaccessible evidence. |
Use calibration results to simplify as well as extend. If a static answer path meets the sufficiency contract for a request class, remove it from the agentic escalation population and verify that the router no longer spends tool or state budget there. When calibration changes, preserve the previous threshold and cases so a quality regression is reversible. Review representative false-completion samples with the authority owner, because a numerical score cannot decide whether a cited source actually supports a high-impact conclusion.
Study Amazon Q Business as a product-specific self-correcting knowledge system
AWS launched Agentic RAG for Amazon Q Business in August 2025. Its published design is a concrete organisational knowledge-base example: analyse the question and conversation, choose retrieval tools, decompose complex work, retrieve in parallel where useful, assess the response, then seek more evidence or disambiguate when needed. It is a product-specific implementation, not a generic recipe to recreate component-for-component.
Turn the published Amazon Q Business design into reusable design questions
| Published capability | Bounded interpretation | Evidence to require in your system |
|---|---|---|
| Query decomposition and parallel retrieval | Split only a documented multi-part query class and preserve its parent request. | Per-part source trace, partial-result policy, and a comparison with the static baseline. |
| Tool selection across retrieval modes | Select from an allow-listed tool set rather than grant free-form access. | Tool schema, identity scope, denial tests, budget, and an explainable terminal reason. |
| Response quality checks with more retrieval or disambiguation | Define the specific insufficiency signal and maximum corrective action. | False-completion, unnecessary-retry, latency, and citation-validity slices. |
Glossary: terms first introduced here
These are the technical terms introduced in this step. Later modules build on them rather than redefining them from scratch.
New vocabulary for this step
| Term | Plain-language meaning | Why it matters here |
|---|---|---|
| Agentic RAG | A retrieval system that can plan, choose permitted retrieval tools, inspect results, and take bounded follow-up actions. | It adds useful capability for some complex questions but also state, cost, and failure modes that need evidence. |
| Tool | A bounded operation an agent may invoke, such as a retriever, a database query, or an approved business API. | A plan is not permission: each tool call needs its own schema, authorization, and audit boundary. |
| Corrective retrieval | A bounded follow-up retrieval action taken after a defined insufficiency or failure signal. | It is safer than open-ended retries because the trigger, action, budget, and fallback are explicit. |
| Evidence sufficiency | A workload-defined rule for deciding that permitted evidence is adequate for a specific answer. | It provides a testable stop condition for an agent instead of relying on a confidence phrase. |
Knowledge Check
Test your understanding with this quiz. You need to answer all questions correctly to mark this section as complete.