Corpus Engineering and Ingestion

Before parsing, assign every source a trust tier, accountable owner, version, permitted audience, retention rule, and retrieval purpose.

Propagate source locator, version, ACL or tenancy, and freshness metadata to each chunk so authorization, citations, and correction remain possible after indexing.

For an S3 data source, place a source-metadata sidecar named POL-17-v4.pdf.metadata.json beside policies/POL-17-v4.pdf. Its values are document metadata: Amazon Bedrock applies them to every chunk produced from that document. This is useful for shared provenance and retrieval filters, but it is not yet a distinct per-chunk schema.

includeForEmbedding: false keeps a field available for metadata filtering without adding it to the embedding input. When true, Amazon Bedrock concatenates that key and value into the embedding input; it does not return the metadata as raw chunk text. Treat title inclusion as a measurable relevance hypothesis, not an automatic improvement.

If each passage needs its own chunk_id, section_heading, or page span, use native chunking with a POST_CHUNKING custom-transformation Lambda. That boundary can write unique metadata for each generated chunk; when a chunk-level key collides with file-level metadata, the chunk value wins. Preserve source_id and source_version so every override remains traceable to the document that supplied it.

Retain headings, tables, page or section boundaries, and source locations before chunking. Fixed chunks are a baseline; semantic chunks use a foundation model to identify meaning-aware breaks; hierarchical chunks pair precise children with broader parents; multimodal content follows model-level or parser-specific handling.

View full size

Amazon Bedrock User Guide: Knowledge Bases preprocessing figure (CC BY-SA 4.0)Last verified: 2026-08-26

Document structure is retrieval preparation. AWS writing guidance recommends descriptive headings, sequential procedures, local context, defined abbreviations, textual treatment of tables and graphics, and smaller self-contained documents so a retrieved passage remains interpretable outside its original page.

For a customer-managed Knowledge Bases data source, start with the information that must survive ingestion, not with a parser preference. The default parser extracts text only and has no parser usage charge. Choose it when text is the evidence boundary and use representative files to prove headings, locators, metadata, and retrieval are sufficient.

Parser strategy type cannot be changed in place after the data source is connected: moving between BDA, a foundation-model parser, and the default strategy requires a new data source. Within the same parser strategy, AWS documents updating parameters such as a model ARN or parsing prompt by retrieving and resubmitting the complete configuration. Managed Knowledge Bases are a separate, more-opinionated path and currently use SMART_PARSING; do not infer that their configuration or availability matches a customer-managed data source.

View full size

AWS Samples: multimodal RAG sample architecture (Apache-2.0)Last verified: 2026-08-26

Chunking is an ingestion commitment: it fixes what retrieval can see before query-time ranking or prompting begins. Inventory prose, nested policies, tables, code, atomic records, and multimodal media alongside the questions users actually ask. The Agentic AI Lens explicitly warns against a single fixed chunk size for every document type.

Treat native Nova multimodal embedding as a representation decision, not as a fourth parser. For audio, video, and images, its media chunking occurs at the embedding-model layer, while BDA converts media to text and then applies text chunking. The connected advanced-retrieval module goes deeper on native multimodal retrieval; here, choose the path that preserves the evidence your workload must retrieve and evaluate it on media-specific cases.

Inspect result count, parent context, metadata size, and citation resolution for the corpus and store you will actually operate.

1. Detect source, ACL, parser-version, update, and deletion changes.
2. Use source and chunk versions so indexing is idempotent.
3. Propagate deletion and permission changes before retrieval.
4. Quarantine malformed, unsupported, and ambiguously owned documents.
5. Measure sync lag, stale-result rate, and failure reason.

A source inventory is not enough. An acceptance manifest makes the ingestible unit explicit: what entered, who approved it, which audience may retrieve it, what parser and chunking policy applied, and how the team will prove that a later update or deletion took effect.

Reject a manifest with an unknown owner, ambiguous permitted use, or no update and deletion test. The Generative AI Lens frames data architecture around governed data use; the security reference architecture adds the principle that access to data and systems should be controlled rather than assumed from a model or vector-store choice. Those are architecture controls, not metadata decoration.

Define a workload-specific freshness SLO before choosing a refresh mechanism: for example, the maximum permitted time from an approved source or ACL change to verified retrieval behavior. Record the source version, parser and chunking profile, expected retrieval outcome, and rollback owner. An ingestion request or a changed object timestamp is not proof that the RAG corpus meets that SLO.

Enable and retain ingestion evidence at the right operational boundary. Knowledge Bases APPLICATION_LOGS can report a job status and resource-level status reasons plus chunk create, update, delete, and failure statistics. Join that evidence to the source event and retrieval checks: a parser fallback, a failed document, or a deleted source is an investigation signal even when the overall job completes.

View full size

AWS Samples: automatic sync for Bedrock Knowledge Bases architecture (MIT-0)Last verified: 2026-08-26

Quarantine is not a silent retry queue. It protects users from a corpus that is half parsed, incorrectly classified, or missing access controls, while giving the data owner a precise repair task. Separate a transient connector failure from a content failure: the former may be retried under an idempotent job policy; the latter needs a changed source, parser, or approval decision before re-ingestion.

Build a small controlled corpus with one approved document, one intentionally denied document, and one document that will be replaced. The lab proves lifecycle behaviour by querying the corpus, not by trusting upload timestamps or a successful job submission.

1. Record the manifest, source locator, version, expected query, and expected access result for each document.
2. Start the documented data-source sync and capture its final status plus any failure reasons.
3. Use a retrieval-only query to confirm the approved locator and metadata are present and the denied material is not exposed.
4. Replace the approved document, sync again, and verify that a query returns the new version rather than a mixture of old and new passages.
5. Delete the source through the documented lifecycle, then verify with a targeted retrieval query that the removed locator no longer appears; escalate if stale evidence remains.

Keep the evidence set small enough for a reviewer to inspect every result. If a deletion cannot be verified at retrieval time, treat the corpus as not ready for that source class and investigate the data-source lifecycle, metadata filters, and application caching boundaries before widening use.

A retrievable chunk is only useful for investigation if the team can trace it back to the source version, owner, parser, access classification, and lifecycle event that created it. Treat source lineage as a change envelope that travels beside ingestion work, not as a note in a release ticket.

AWS documents custom data sources as a path that can directly ingest or delete documents, while other data-source changes require a sync. It also documents metadata as a retrieval input. Do not generalize one lifecycle to every connector: record the selected source type and use the documented operation for that type. If the event cannot name the expected metadata and verification queries, keep it out of automated publication.

Lineage also prevents a misleading repair: re-running ingestion against the same source version and parser does not fix a bad semantic extraction. Require the event owner to identify what changed in the source, parser profile, metadata contract, or access decision before another attempt is eligible. That record distinguishes an idempotent retry from a new corpus candidate and preserves a defensible audit trail when a user reports stale evidence.

Run semantic and ACL checks in the same release harness. A parser change can preserve text relevance while dropping metadata; an ACL change can preserve metadata while unexpectedly changing chunk selection. AWS’s metadata guidance shows that metadata fields influence how a knowledge base can query data, while the security reference architecture treats controlled data access as a core architecture concern. The only useful regression result is therefore one that checks both evidence relevance and permitted visibility.

Include a deliberately ambiguous document in the harness, such as a source with a similar title but a different audience. This detects an overly broad semantic match that looks relevant but should never become visible to the test principal. If the candidate returns it, preserve the failing source identifiers, quarantine the change, and correct the metadata or retrieval policy before retrying. Do not hide the case by removing it from the corpus sample.

1. Version a small golden corpus with expected source locators and authorized principal classes.
2. Run the cases before and after each parser, chunking, metadata, or connector change.
3. Diff the source identifiers and access outcomes, not only free-text answer quality.
4. Quarantine the candidate corpus and restore the prior accepted source/index version when a rollback trigger occurs.