Enterprise AI Project Wiki · Data, knowledge, and models
Document chunking
Also calledChunking · Text chunking · Content chunking · Document segmentation · Document splitting · 文档切分
Document chunking is the process of converting a parsed source document into retrievable content units with stable identities and source relationships. Each chunk should preserve a topic, fact, rule, procedure, or structural unit that remains understandable for the target task, together with metadata sufficient to resolve its document, version, location, heading path, applicability, and access rights.
Chunking turns a document into retrieval units without severing accountability to the original
A policy may run to dozens of pages while search needs to find two relevant paragraphs in seconds. Document chunking makes that local retrieval possible, but each passage must still answer where it came from, whom it applies to, and whether it remains current.
A 120-page policy can be one formal document but is rarely an appropriate single retrieval result. When a user asks about one approval exception, the system should locate the applicable clause, conditions, and explanation rather than pass the entire file to a model. Chunking organises parsed content into smaller retrieval units that search can resolve to a specific topic and location.
A chunk is a derived representation, not a new original. Its text, added headings, table transcription, image description, or generated context must resolve to the original version and location. Withdrawal, access changes, or parsing errors in the source must propagate to every affected chunk.
Smaller is not automatically more accurate. A small chunk may match one term but lose a definition, negation, precondition, table header, or exception. A large chunk preserves more context but can mix topics, weaken ranking precision, and consume model context. Chunking is a validated trade-off among retrieval precision, semantic integrity, citation, cost, and maintenance.
Confirm the original, parsing output, and reading order before chunking
Chunking cannot repair a document that was parsed incorrectly. Headers inserted into prose, reversed columns, or missing table cells will simply be propagated, so input quality needs its own check first.
Original identity
Confirm source ID, file or record ID, version, publication and effective time, owner, state, language, licence, and access scope. Do not build a formal index from a personal download or superseded copy.
Text acquisition
Distinguish native text, OCR, speech transcription, image descriptions, and manual entry. Selectable characters do not prove correctness; scans, rotation, columns, footnotes, and unusual fonts can change meaning.
Layout and reading order
Recognise heading levels, paragraphs, lists, tables, figures, headers, footers, annotations, text boxes, and cross-page relationships. Incorrect coordinate ordering in a multi-column PDF can join unrelated sentences.
Structural semantics
Preserve sections, clauses, Q&A, steps, fields, code blocks, table rows and columns, and attachment relationships. A chunker cannot recover meaning that upstream parsing has already destroyed.
Noise handling
Make removal of repeated headers, footers, navigation, copyright text, contents pages, and watermarks configurable and auditable. Removing a repeated formal warning can also damage knowledge.
Parsing quality gate
Fail or route empty pages, garbled text, bad order, broken tables, undescribed figures, missing attachments, wrong language, and over-limit files to review instead of silently chunking them.
A useful chunk includes identity, location, and applicability—not text alone
Putting a paragraph into a vector store does not yet make it a manageable content unit. An enterprise system also needs its document identity, exact location, inherited access, and a dependable way to withdraw it when the original changes.
- Stable chunk ID
Use a reproducible rule or controlled identifier and distinguish retained, added, modified, moved, and deleted chunks during reprocessing.
ID convention, old-to-new map, change list
- Source reference
Link source ID, document version, snapshot, or checksum and make derivation among original, parsed, cleaned, and chunked forms explicit.
Provenance chain, version, checksum
- Precise location
Record page, section, heading path, paragraph, character or element range, table, and attachment ID so a user can open and verify the location.
Locator, original anchor, link-back test
- Text and structure
Separate original text from repeated ancestor headings, table headers, neighbouring context, and generated descriptions. Supplemental text must not masquerade as a quotation.
Content fields, structure fields, derivative marker
- Time and state
Inherit publication, effective, expiry, withdrawal, and historical states. Similar clauses across versions must retain their time boundary.
Validity interval, state, replacement relationship
- Access and scope
Carry tenant, organisation, role, object, region, product, classification, and permitted use, and apply them before retrieval.
Access tags, policy mapping, unauthorised-user tests
- Processing version
Record parser, chunker, rules or model, tokenizer, parameters, time, and manual corrections so results can be rebuilt and compared.
Processing manifest, configuration version, run logs
Fixed-size, sentence, structural, semantic, and parent-child strategies fit different conditions
There is no best strategy apart from the content. Fixed length offers a useful baseline, structure follows policies and manuals more closely, and parent-child designs can balance precise recall with complete reading; actual questions should decide among them.
| Strategy | Use and limitations |
|---|---|
| No chunking | Treats a file or already-small object as one unit, suitable for a short FAQ, independent record, or complete clause. Long documents mix topics and cost more, and some systems lose page-level citation. |
| Fixed characters or tokens | Produces roughly even units with low, predictable implementation cost for weakly structured text and logs. It can split sentences, tables, code, and rule conditions. |
| Sentence or paragraph | Combines natural language units to a target size. It depends on reliable segmentation; abbreviations, lists, Chinese punctuation, code, and OCR can still break boundaries. |
| Heading and layout | Uses sections, clauses, lists, tables, or slide objects and recursively splits oversized units. It suits manuals, policies, and complex PDFs but depends on correct layout and heading hierarchy. |
| Semantic | Finds boundaries from semantic change between neighbouring sentences or passages. It can improve topic cohesion but costs more and varies with model, threshold, language, and corpus distribution. |
| Recursive | Tries section, paragraph, sentence, word, and character separators in order until limits are met. It is a practical fallback, but separator priority still requires document- and language-specific validation. |
| Parent-child or Small-to-Big | Retrieves precise child chunks, then returns a larger parent or neighbour window. It improves completeness but requires mapping, deduplication, and context-budget control. |
| Hierarchical summary tree | Clusters lower-level chunks and recursively creates summaries for document-wide and multi-hop questions. Summaries are derivatives and add generation cost, source validation, and error-propagation risk. |
| Contextual or late chunking | Adds document context before embedding, or lets a long-context embedder process the whole text before pooling chunk representations. It can reduce isolated-fragment ambiguity but is constrained by model context, cost, implementation, and measured results. |
Identify indivisible business units by document type before applying length limits
A contract clause, operating procedure, and price table have different integrity boundaries. First decide which conditions, exceptions, units, or neighbouring steps must stay together; character and token limits come afterwards.
Policies, contracts, and standards
Preserve chapter-clause-subclause hierarchy and keep conditions, duties, exceptions, definitions, cross-references, and schedules together. Index current and historical versions separately and filter by applicable time.
Procedures and manuals
Keep a task, prerequisites, step sequence, warnings, branches, expected result, and recovery coherent. A hit on a middle step should recover the task title and necessary neighbouring steps.
FAQs and knowledge cards
One approved question, answer, applicability, and full source can often be one chunk. Store alternative phrasings as retrieval fields rather than creating drifting official answers.
Reports and research
Organise abstracts, methods, results, limitations, figures, and conclusions. A chart result must retain metric, unit, sample, time, and caveat. Cross-section synthesis may require parent-child or hierarchical retrieval.
Web pages and help centres
Remove navigation, recommendations, footers, and template noise while retaining title, breadcrumb, anchor, and update state. Reconstruct one topic across pagination and dynamic content.
Email, tickets, and conversations
Segment by thread, problem, decision, handling, and final state while retaining speaker, time, and quotation links. Historical cases are not current policy and often contain personal or confidential information.
Structured records
Database rows and API objects are normally retrieved as entities, events, or transactions, not turned into long prose and arbitrarily split. Any generated searchable text must retain the record key, field source, and query time.
Derive size and overlap from questions, content density, models, and the retrieval chain
A few hundred tokens may be an experiment starting point, not an industry standard. Small chunks lose conditions, large ones dilute topics, and overlap can repair boundaries while also adding duplicate results, index cost, and citation ambiguity.
Length unit
State whether the limit uses characters, bytes, words, sentences, or which tokenizer's tokens. Token ratios vary by language and model, so two values of ‘500’ may not be comparable.
Minimum complete unit
First decide whether a rule, procedure, table row, Q&A, or function can stand alone. A maximum length handles oversized units; it must not replace business integrity with a number.
Embedding limit
Chunk text, headings, and supplemental context together must fit the embedding input. Detect and count truncation instead of treating a silently truncated vector as complete.
Generation budget
Larger chunks leave room for fewer distinct pieces of evidence. Reserve context for the query, system instructions, conversation, tool output, citations, and model response.
Purpose of overlap
Overlap protects meaning that crosses a boundary; more is not always better. Duplication raises indexing, storage, and context cost and can crowd other evidence out of Top-K.
Boundary-aware overlap
Prefer repeating a necessary heading, definition, table header, precondition, or adjacent sentence over copying a fixed percentage. Access changes and withdrawal must cover duplicated material too.
Parameter experiment
Compare multiple strategy, size, and overlap combinations on representative questions. Segment recall, precision, completeness, citation, latency, and cost by content type and language before freezing production settings.
Preserve enough structure for understanding without putting the entire document back
A reader who sees ‘approval is required above the limit’ will naturally ask which limit, whose approval, and what exceptions apply. Good chunking retains the smallest structure needed to answer those questions without leaving an orphan fragment or returning the whole file.
Ancestor headings
Store the document title and necessary heading path as separate metadata and selectively use them for embedding, keyword indexing, and display. Identify them as supplemental structure rather than quoted chunk text.
Definitions and preconditions
When a clause depends on a chapter definition, applicable subject, or prior state, link the definition, repeat only essential context, or expand a parent after retrieval instead of copying a whole chapter into every chunk.
Negation, exceptions, and notes
Words such as ‘must not’, ‘unless’, and ‘only applies to’, and footnotes can determine the conclusion. Tests must ensure they remain with the content they constrain.
Cross-references
Resolve references such as ‘section 4.2’ or ‘Appendix B’. If only the referring sentence is returned, retrieve the target too or state that evidence is incomplete.
Neighbour windows
Keep index units precise and expand previous or next chunks after a hit, subject to version, state, access, and budget. Never expand across documents or classification boundaries accidentally.
Parent-child relationships
Children support recall; parents support complete reading. Preserve parentId, order, and range, and deduplicate when several child hits resolve to one parent.
Global questions
A question such as ‘what are the major changes across this policy?’ is not local-chunk retrieval. It may need document summaries, chapter retrieval, change structure, or multi-step queries; making every chunk larger is not a reliable solution.
Tables, images, code, and attachments cannot be treated as ordinary paragraphs
Complex content carries meaning through relationships: a number depends on row and column headers, an image on its caption, code on surrounding scope, and an attachment reference on its target. A length-only split can preserve words while losing their business meaning.
Tables
Preserve caption, column and group headers, units, footnotes, and row relationships. Split large tables by meaningful row groups while repeating necessary headers and linking the full table; recognise continued tables across pages.
Images and charts
Keep image ID, caption, legend, axes, units, surrounding explanation, and source page. OCR or model descriptions are derivatives with a method and review state, not replacements for the image.
Presentations
One slide is not always one topic. Interpret title, bullets, figures, speaker notes, and progressive builds together; template footers and repeated branding normally do not belong in semantic retrieval.
Code and configuration
Prefer module, class, function, configuration-object, or syntax-tree boundaries and preserve path, language, symbol, imports, and version. Character cuts can create invalid or misleading fragments.
Forms and scans
Maintain the relationship among field labels, values, check states, coordinates, and repeated fields across pages. Low recognition confidence, stamps, signatures, or handwriting that affects meaning enters human review.
Attachments and embedded objects
An attachment can contain a required definition, list, or evidence. Register it as a separate source object linked to the parent. A chunk saying ‘see attachment’ is not complete knowledge when the attachment is missing.
Multilingual content
Keep original, official translation, and machine translation as distinct versions with language and alignment. Languages can use different sentence and size settings but must represent the same applicable fact.
Provenance, access, and deletion duties must propagate to every derived chunk
Chunking turns one original into many derivative records; it does not relax the original access boundary. Permissions, version, validity, and deletion instructions must travel with every chunk or ordinary retrieval may bypass source controls.
Provenance
Use relationships such as W3C PROV entities, activities, and agents to connect original, parsing, cleaning, translation, chunking, contextualisation, embedding, indexing, retrieval, and answer and locate the transforming stage.
Precise citation
Page numbers can change after layout and character offsets after editing. Combine document version, heading path, element ID, text quote, and position selectors and verify that an anchor still identifies the same content.
Access inheritance
Chunks inherit source and content-level access, using the stricter scope where necessary. Titles, summaries, and vectors can also leak information; protecting the original download alone is insufficient.
Tenant isolation
A shared index must apply trusted identity and mandatory filters before recall. Chunk IDs, parents, caches, logs, and citation expansion cannot collide across tenants; high-risk uses may justify isolated indexes.
Deletion and withdrawal
When a source is deleted, its licence expires, access is revoked, or content is withdrawn, locate and handle duplicates, parents, children, summaries, embeddings, indexes, caches, and evaluation copies and record uncontrolled external copies.
Content security
Instructions and scripts in a document are data, not system commands. Preserve risk flags during parsing and chunking and restrict agent tools at runtime so a malicious fragment cannot cause unauthorised action.
A chunking approach is a versioned data pipeline, not a one-off script
Changing the parser, chunking rule, or embedding model changes what the retrieval system sees even when the source file is identical. Recording these steps as a rebuildable, comparable pipeline makes it possible to trace a regression to the change that caused it.
1. Build representative documents and questions
Sample by type, length, language, layout, classification, and risk. Register user questions, relevant originals, prohibited results, and expected citations.
2. Validate the parsing baseline
Verify original, text, order, headings, tables, figures, and attachments. Isolate classes that cannot be parsed reliably instead of hiding failure with chunk parameters.
3. Define candidate strategies
Configure structure-first boundaries, maximum and minimum size, overlap, parent-child relations, table and image handling, and exception fallbacks by content type, retaining alternatives for comparison.
4. Generate auditable chunks
Output text, structure, metadata, provenance, access, and processing version and detect empty, oversized, duplicated, truncated, garbled, orphaned, and uncovered content.
5. Evaluate in the retrieval chain
Test with the planned keyword, vector, hybrid, and reranking configuration because chunking cannot be optimised independently of tokenizer, embedding, indexed fields, and result count.
6. Obtain business and technical confirmation
Content owners verify meaning and citations, users verify questions, security verifies access, and technology verifies cost, performance, rebuild, and monitoring before the version combination is frozen.
7. Rebuild incrementally or fully after change
Analyse changes in source, parser, rules, tokenizer, embedding model, or access approach. Do not silently mix incompatible old and new chunk representations in one index.
8. Monitor and regress
Track misses, wrong chunks, truncation, duplicated evidence, broken citations, and cost. Regress affected questions after a fix and retain prior results to explain movement.
Acceptance tests chunks, retrieval, answers, access, and operations
A handful of readable chunks is not enough evidence. Test with real question patterns: required evidence should be found, restricted material should remain hidden, answers should link back to the source, and an update should rebuild predictably.
Structural coverage
Measure whether source content maps to chunks and inspect empty chunks, omitted pages, duplication, overflow, and anomalies. Full character coverage still does not prove correct tables, order, or semantics.
Within-chunk integrity
Sample whether a chunk identifies its topic, subject, action, condition, negation, exception, unit, and time and whether a parent heading or neighbouring passage is required.
Retrieval relevance
Label relevant and irrelevant chunks for representative questions and measure Recall@K, Precision@K, first qualified rank, and no-answer queries, segmented by document type, language, query type, and risk.
Evidence sufficiency
Check that returned chunks collectively contain all facts, rules, and conditions needed for an answer, not merely a matching term. Label the multiple pieces required by multi-hop questions.
Verifiable citation
A user can open the correct version and page or section; quoted text matches the original and retains table headers, legends, and context. Anchor drift is detected.
Final answer
With retrieval, reranking, prompt, and model fixed, compare correctness, relevance, completeness, grounding, refusal, and hallucination. Do not misattribute a generation failure to chunking.
Access and deletion
Test self, delegate, cross-department, cross-tenant, revoked access, withdrawal, and cache so text, headings, parent expansion, and citations do not leak. This hard gate is not offset by average recall.
Operating cost
Measure chunk count, index size, embedding and rebuild time, retrieval latency, duplicated context, model tokens, and review volume, then compare cost among approaches that meet quality requirements.
Common failures cannot be explained only as chunks being too large or too small
Changing chunk size whenever retrieval fails often damages another question type. Start with the observed symptom, then determine whether parsing, structure, access, versioning, deduplication, or context assembly is responsible.
Half a rule is retrieved
A condition is in the prior chunk, an exception in the next, or a definition at chapter start. Fix structure, necessary headings, parent expansion, neighbour windows, or cross-reference retrieval rather than only adding overlap.
A table becomes loose text
After row-column relationships and units disappear, semantic search can still return a wrong number. Improve layout parsing, table modelling, and citation display, or stop indexing a table that cannot be recovered reliably.
Duplicates crowd results
Excess overlap, template headers, and near-identical versions occupy Top-K. Remove templates, filter versions, deduplicate parents, diversify, or rerank while preserving legitimate historical access.
Orphan pronouns and abbreviations
‘This product’, ‘the above limit’, and ‘its owner’ are meaningless alone. Add minimal headings and context, merge passages, or use contextual representations, marking generated additions as derivatives.
Child chunks lose access
A restricted source is indexed without access, or parent expansion enters a higher classification. Stop retrieval, withdraw derivatives, repair inheritance, and regress every identity scenario.
Citations drift after updates
Old page, character, or chunk offsets survive a new layout. Bind location to document version and validate during rebuild; retain text quotations when needed for re-anchoring.
Evaluation asks only easy questions
Title matches hide failures in negation, exceptions, tables, history, insufficiency, and multi-hop evidence. Expand representative questions and counterexamples before changing the chunker.
Long context bypasses governance
A whole document fitting a model does not mean it should be sent. Access, position bias, noise, cost, citation, and withdrawal remain, and research shows that use of information at different long-context positions is not stable.
Example: chunking a procurement approval policy
This example only shows how to define and validate a chunking approach. It is not a client document, real approval policy, default chunk size, measured retrieval result, or technology choice.
Sample documents
Select the current policy, one historical version, a delegation table, procedure, and FAQ, including clauses, a cross-page table, footnotes, attachment references, and different classifications. Confirm original version and access first.
Structural rule
Create candidate chunks by policy clause and retain chapter headings. Keep a condition, threshold, exception, and explanation together. Split the delegation table by meaningful organisation and role row groups with headers, units, and validity.
Oversized content
Recursively split a long clause into subclauses linked to one parent and a procedure into task-level step groups. After a child hit, return the parent clause or necessary neighbouring steps within access and budget.
Source and access
Every chunk stores document ID, version, page or section, heading path, validity, owner, and access tags. A shorter FAQ does not become more authoritative than the policy clause it cites.
Test questions
Cover current and historical thresholds, organisational applicability, exceptions, insufficient material, separation-of-duty conflict, missing attachments, withdrawn policy, unauthorised users, and questions needing both table and prose evidence.
Stop conditions
Do not approve release if table relationships cannot be recovered, negation or exceptions are lost, source locations cannot open, access labels are absent, expired versions enter current retrieval, or key questions lack sufficient evidence.
Handover
Transfer samples and questions, parser and chunker configuration, chunk schema and ID rules, source and access mapping, candidate comparison, frozen version combination, rebuild procedure, monitoring, known gaps, and regression requirements.
In enterprise AI and RAG, chunking defines retrieval units but does not determine answer quality alone
Chunking is easy to blame when a RAG answer is wrong because it determines the passages the system can see. Yet several transformations sit between an original and the final answer; preserving each layer is what turns parameter guessing into diagnosis.
Find the layer where the failure occurred
One wrong answer does not prove that chunking failed. The source may be stale, a table may have parsed incorrectly, a correct chunk may have been removed by filtering or reranking, or the model may have ignored supplied evidence. Inspect the original, candidates, ranking, final context, and answer as one trace before deciding what to change.
Match query and chunk granularity
A definition fits a small chunk, a procedure needs complete steps, and a summary or multi-hop question needs several local facts plus higher structure. Route question types to appropriate indexes, levels, or tools.
Separate retrieval from delivery
The small unit used for similarity does not have to be the final model context. Retrieve children, rerank, then expand parents or neighbours while deduplicating, filtering access, and enforcing the token budget.
Keyword and vector retrieval both change
Chunking alters term density, exact phrases, entity relationships, and embedding meaning. Evaluate it in the planned full-text, vector, or hybrid chain rather than inspecting vector similarity alone.
Distinguish contextual additions
Generated descriptions, summaries, or hypothetical headings can improve recall and can also introduce error. Store them separately with model and prompt versions, and use the original—not the generated addition—as factual citation.
Long context does not remove chunking
Even when a model can accept a whole file, selecting relevant evidence reduces irrelevant and unauthorised content, cost, and citation difficulty. Lost in the Middle also shows input capacity is not reliable use of every position.
Compare advanced methods with a baseline
Semantic chunking, summary trees, contextual retrieval, and late chunking are test candidates, not automatic upgrades. Compare quality, cost, stability, and risk against a simple structural or sentence baseline on the same corpus and queries.
Regress every material change
Changes to parser, chunk parameters, embedding model, source version, access, retrieval, or reranking can change recall. Re-run high-impact, refusal, citation, and access tests for the new combination.
How document chunking differs from adjacent concepts
Chunking is usually discussed beside parsing, indexing, retrieval, and context assembly, but each solves a different problem. Clear boundaries make ownership and layer-specific tests easier, and prevent one parameter from standing in for governance of the whole chain.
| Concept | Boundary from document chunking |
|---|---|
| File splitting | Creates multiple files for transport, archiving, or manual organisation. Document chunking creates retrieval units and need not create separate files. |
| Pagination | Is a layout or display boundary. One page can hold several topics and one table or clause can cross pages; page is useful location metadata but rarely the sole semantic boundary. |
| Parsing and OCR | Acquire text, structure, reading order, and layout objects from files as upstream input. Later chunking cannot automatically repair parsing errors. |
| Word or token segmentation | Encodes text into words, subwords, or tokens for a model. Chunking determines the larger content units indexed and retrieved independently. |
| Summarisation | Compresses and rewrites content and can form a higher retrieval representation. Chunking normally preserves original passages; a summary remains a derivative linked to its source. |
| Embedding | Converts text or other content into a vector. Embeddings are often generated per chunk, but chunks also serve full-text search, citation, and context and are not the same as vectors. |
| Indexing | Organises chunks and fields for query. Chunking defines content units; indexing separately handles fields, filters, ranking, and storage. |
| Retrieval and reranking | Select and rank chunks for a query. Chunking affects candidate quality, but recall, filtering, Top-K, and reranker failure are different stages. |
| Context assembly | Deduplicates, expands, orders, and places results in model input. It can expand a small retrieval chunk to a parent while enforcing access, evidence sufficiency, and token budget. |
| Knowledge base | Governs approved knowledge, structure, version, access, quality, and maintenance. Chunking is one processing step that creates retrieval representations for some unstructured content. |
| RAG | The complete chain includes query, retrieval, context, and generation. Chunking mainly occurs during ingestion and influences runtime by defining retrieval units. |
Sources and scope
This entry explains document chunking in enterprise search, knowledge-base, and AI RAG projects: turning acquired and correctly parsed document content into retrievable, understandable, citable, and maintainable units while preserving source identity, structure, location, context, access, version, and provenance. Document chunking is not synonymous with file splitting, pagination, OCR, layout parsing, word or token segmentation, summarisation, embedding, vectorisation, indexing, retrieval, reranking, context assembly, or generation; those are upstream, downstream, or complementary steps. No chunk size, overlap, or supposedly best algorithm applies to every document, language, query, embedding model, and retrieval method. This entry does not fully define document management, OCR and layout recognition, table and image understanding, sentence segmentation, embedding models, vector databases, full-text or hybrid retrieval, reranking, complete RAG or agent architecture, copyright or privacy law, or vendor configuration. Accountable project roles must validate the production approach against content, questions, access, quality, cost, and risk.
- Microsoft Azure RAG chunking phase — selecting fixed, sentence, layout, and semantic approaches by document structure, comparing engineering cost, and treating the strategy as consequential production design
- Azure AI Search document-chunking guidance — factors for size and overlap including content shape, user queries, and embedding-model constraints, with starting points for experimentation
- Microsoft Azure RAG design and evaluation guide — staged validation of preparation, chunking, enrichment, embedding, indexing, retrieval, context, and generation
- Google Cloud Agent Search parsing and chunking — digital, OCR, and layout parsing; heading, table, and list structure; RAG chunk retrieval; and configuration constraints
- Google Cloud Document AI Layout Parser — extracting text, reading order, headings, tables, lists, images, and context-aware chunks from complex files
- Amazon Bedrock Knowledge Bases chunking — fixed-size and sentence boundaries, hierarchical parent-child, semantic and no-chunk strategies, multimodal behaviour, and limitations
- Amazon Bedrock Knowledge Bases metadata guidance — document and chunk metadata for retrieval filtering, content scope, and access control
- NIST RAG definition — a generative model paired with a separate retrieval system or knowledge base whose relevant information is supplied as generation context
- W3C PROV-O Recommendation — provenance from originals to chunks using entities, activities, agents, generation, use, derivation, revision, and attribution
- W3C Web Annotation Data Model — sources, states, and selectors for precise resource, text-position, text-quote, fragment, and specific-target references
- Liu et al., Lost in the Middle — long-context models do not use relevant information at different input positions consistently, so capacity does not prove reliable use of a whole document
- ICLR 2024 RAPTOR paper — clustering lower-level chunks and recursively summarising them into a retrieval tree for document-wide and multi-step questions
- Günther et al., Late Chunking — a long-context embedder first contextualises document tokens before pooling chunk representations