Enterprise AI Project Wiki · Data, knowledge, and models
Retrieval
Also calledInformation retrieval · IR · Search · Content retrieval · Knowledge retrieval · Search and retrieval · 检索
Retrieval is the process of representing a defined information need as one or more queries, applying mandatory filters and matching against an identified and authorised collection, generating candidates, calculating or combining relevance, ranking, and returning locatable results. A governed retrieval result identifies the query, collection and version, caller access, matching object and location, source, score or ranking basis, filters and refusal state, and execution time.
Retrieval answers which authorised content to find first; it does not generate the final answer
When a user asks a question, the system must first decide where to look, which candidates to find, and which results the user may receive. Retrieval passes possible evidence forward, but relevance is not correctness and does not prove that the evidence is sufficient.
For the question ‘How is a purchase above a certain amount approved?’, retrieval must first decide whether to search the current policy, a delegation table, or a historical record and then find the threshold, organisation, exception, and effective time. Returning candidates does not complete the business decision; the system must still determine whether evidence is sufficient, sources conflict, and the caller may view them.
Information retrieval concerns the relationship between an information need and content objects. A query is only one expression of that need and can contain multiple subquestions, abbreviations, spelling errors, time ranges, and implicit entities. The same term can mean different things in different departments. Quality therefore cannot be reduced to keyword presence or vector distance.
NIST's RAG definition distinguishes the separate retrieval system or knowledge base from the generative model: relevant information is found first and then supplied as context. Retrieval can serve enterprise search without generation or form part of RAG; either way it remains accountable for collection, access, version, ranking, and results.
The retrieval contract defines inputs, collections, constraints, outputs, and failure states
An ‘input question, output results’ description is too thin to assign responsibility when retrieval fails. A testable contract also names eligible collections, time and access constraints, the evidence returned, and explicit stop states.
- Information need
State whether the user needs a fact, definition, rule, procedure, comparison, case, historical state, evidence set, or actionable object.
Question taxonomy, task statement, sample queries
- Query representation
Preserve the original query, language, time, and conversation scope plus rewrites, expansion, entity resolution, subqueries, and filters. Separate user text from system-generated material.
Query log, transformation chain, parsed fields
- Target collection
Identify knowledge base, data source, index, fields, and version, including current, historical, regional, product, and tenant scope. ‘Search everything’ is not a boundary.
Collection catalogue, index version, routing rules
- Identity and access
Use trusted caller or service identity and enforce tenant, object, attribute, purpose, and content-state restrictions before candidate recall.
Identity context, filter policy, denial records
- Candidates and ranking
Record exact, structured, full-text, sparse, dense-vector, graph, or combined methods, candidate count, fusion, reranking, and final result count.
Retrieval configuration, stage outputs, scores
- Result object
Return content ID, title, passage, source version, location, applicable time, access tags, relevance information, and an openable citation—not untraceable prose alone.
Result schema, source links, citation tests
- Failure state
Distinguish no relevant content, access exclusion, stale index, uninterpretable query, source conflict, insufficient evidence, dependency failure, and timeout.
Status codes, user messages, alerts and escalation
Understand the information need before rewriting, decomposing, or expanding a query
A user's wording is not always the right search expression. ‘How much can I still claim this month?’ involves identity, period, limit definition, and spend to date. Preserve the original, then add only the conditions needed rather than freely changing its meaning.
Original expression
Keep the user's wording, language, and channel. Record spelling, synonym, or abbreviation correction and do not silently change amounts, dates, organisations, or negation.
Intent and entity
Determine whether the query seeks a person, product, policy, transaction, state, definition, or procedure and resolve names to stable entity IDs. Retain candidates or ask when identity is uncertain.
Time and scope
Terms such as ‘now’, ‘then’, ‘in my region’, and ‘my request’ require trusted time, identity, and business context as filters rather than similarity-model guesses.
Query expansion
Use governed synonyms, aliases, terminology hierarchies, and spelling variants to improve coverage while retaining the original term and expansion source. Excessive broadening can introduce another business object.
Query decomposition
Split questions containing independent conditions or evidence types—for example, policy rules, live request state, and supplier master data—then combine them under approved logic.
Clarification
When entity, time, region, or purpose materially changes results, ask the user. If a default is safe, expose it in the search; do not hide ambiguity behind a high-scoring result.
Query security
Constrain query languages, fields, regular expressions, scripts, and filters to prevent injection, unauthorised enumeration, resource exhaustion, and incorrect access. Validate model-generated queries deterministically.
Retrieval can only work over content that is ingested, indexed, current, and authorised
A file existing in a company drive does not mean it is in this search scope; being indexed does not mean its version and permissions remain valid. A collection catalogue and synchronisation state explain why an item was or was not available.
Collection coverage
Register content type, source, organisation, region, language, time, and known gaps. If relevant material never entered the collection, tuning a ranker cannot create it.
Retrieval units
Documents, chunks, records, entities, rules, table rows, images, and multimodal objects can require different units. One vector field does not automatically fit every object.
Index fields
Separate searchable text, title, keywords, entities, structure, sparse and dense vectors, filter fields, return fields, rank features, and citation fields. Do not expose sensitive metadata as free search text.
Analysis and language
Tokenisation, case, stemming, stop words, synonyms, and character normalisation affect exact and term matches. Validate Chinese, English, Japanese, and mixed technical vocabulary separately.
Version and currency
Record source, chunking, embedding, field schema, access snapshot, and build version. Current queries exclude expired or not-yet-effective content; historical queries resolve the applicable version.
Rebuildability
Retain approved source inventories, transformation settings, index schema, and processing versions so damage, model replacement, or platform migration can be rebuilt without relying on opaque surviving scores.
Exact, structured, full-text, vector, hybrid, and graph retrieval solve different questions
A policy ID suits exact matching, order status suits a structured query, and conversational paraphrases may benefit from vectors. Newer is not inherently better; the method must handle target questions reliably and complement other channels where needed.
| Method | Suitable questions and limitations |
|---|---|
| Exact identifier lookup | Retrieves a specific order, document, person, or version by stable ID with deterministic behaviour. A missing or wrong ID does not semantically find a near match. |
| Structured query and filtering | Queries records by fields, ranges, times, states, and relationships and supports exact calculations. It depends on a defined schema and semantics and does not replace long-document search. |
| Full-text or lexical retrieval | Uses inverted indexes, phrases, field weights, and methods such as BM25 for names, codes, identifiers, and exact expressions. It can miss paraphrases and cross-language meaning. |
| Sparse semantic representation | Uses high-dimensional sparse terms or learned weights, retaining lexical interpretability while expanding vocabulary. It still depends on term space, model, and training domain. |
| Dense-vector retrieval | Uses embeddings to match semantically similar language and paraphrases. It can miss exact identifiers, negation, and fine policy distinctions and can never own access control. |
| Hybrid retrieval | Runs lexical and vector or other channels and merges them through normalisation or rank fusion such as RRF. It balances exact and semantic matching but adds tuning and diagnostic complexity. |
| Graph or relationship retrieval | Traverses entities and relationships for connections, impact, and multi-hop facts. A path query cannot correct missing or wrong relationships in the graph. |
| Multi-index and tool routing | Routes policy, product, case, live-business, or external questions to different sources. Wrong routing searches the wrong collection, so tool scope and selection accuracy need validation. |
Access, validity, and business scope are hard filters—not relevance boosts
A highly relevant document that the caller cannot access must still be excluded. An expired rule cannot rank first simply because it is semantically closer. Eligibility must apply before relevance and continue through candidates, caches, and final context.
Authorise before retrieval
Use trusted identity and policy to narrow accessible candidates before lexical or vector matching. Post-retrieval removal has already exposed content to search, reranking, logs, or model context.
Document- and content-level access
If one document has sections at different classifications, enforce access at independently retrievable units. Parent or neighbour expansion uses the strictest access among included content.
Tenant isolation
Shared indexes require non-bypassable tenant keys and filters. Caches, pagination, suggestions, counts, logs, and operator tools require the same isolation; high-risk cases can use separate indexes.
Time and state
Filter publication, effective, expiry, withdrawal, draft, and history. No relevance score can make a future or expired policy answer a current question.
Business applicability
Apply organisation, region, product, customer, language, role, and purpose before recall. Cross-scope comparison requires explicit intent and authority.
Coverage after filtering
Filtering can leave insufficient accessible evidence. Measure candidates and Recall before and after filters and explain restriction or insufficiency; never relax access to improve hits.
Audit
Record effective identity, policy version, filter expression, candidate count, and denial reason while minimising logs so queries and matched passages do not leak through monitoring.
Candidate recall, score fusion, ranking, and reranking are connected but distinct stages
Recall deliberately brings back a broader set; reranking then judges its relationship to the particular question. Recording stages separately shows whether the right evidence never entered the pool or entered and was later suppressed by fusion or ranking.
1. Generate candidates
Use one or more channels to retrieve enough plausibly relevant content, primarily avoiding missed evidence. The candidate pool can exceed final results but remains within access and scope.
2. Calibrate or fuse
Raw scores from different fields, lexical methods, vectors, and indexes are usually not directly comparable. Use validated normalisation, weights, rank fusion, or rules and preserve channel contributions.
3. Deduplicate and diversify
Merge overlapping chunks, shared parents, and near-identical versions so Top-K is not occupied by one piece of evidence, while preserving genuinely distinct sources, views, and history.
4. Rerank
Apply a cross-encoder, learning-to-rank model, business rule, or other query-content judgement to a limited candidate set to improve front-rank precision, accepting added latency, cost, and model risk.
5. Apply authority rules
Current, formally approved, or designated-source precedence must be an auditable rule. A relevance model cannot decide which source is authoritative.
6. Form final results
Choose the final N for the task and return source, location, scores, and limits. For generation, also confirm evidence sufficiency and preserve results for different subquestions.
Choose Top-K and final context by evidence need, noise, cost, and risk
Returning dozens of items can look safer while duplicates, old versions, and weak matches bury the decisive clause. The amount of final evidence should follow the question, deduplication, complete conditions, model capacity, and response cost.
Result count
Too small a K misses evidence; too large adds noise, latency, and tokens. Experiment by question type, recall channel, reranking ability, and evidence count rather than using one global number.
Subquestion coverage
Check that every part of a complex question has qualified content. One high-scoring topic must not occupy all results while another has no evidence.
Parents and neighbours
After a small chunk matches, expand its parent clause, neighbouring steps, table header, or definition within the same access and version. Record expanded content separately from the initial hit.
Source diversity
Preserve independent sources for comparison or corroboration. For current official policy, avoid having old versions, training material, and paraphrases dilute the authoritative result.
Citation usability
Results need an openable original location. If the user cannot access the cited source, the system should not present an unverifiable internal reference as public evidence.
Context assembly
Deduplication, order, grouping, truncation, and formatting are a separate post-retrieval stage. Preserve inclusion, exclusion, and truncation reasons to locate retrieval versus assembly failures.
No content, retrieval failure, access exclusion, conflict, and insufficiency require different handling
All may appear as ‘no answer’ in an interface, but their responses differ. An index failure needs repair, access exclusion must not reveal restricted content, insufficiency should identify what is missing, and conflicting sources need preserved differences and responsible review.
No knowledge
No approved source in the collection answers the question. Register the gap and owner and provide a human route; model memory must not masquerade as knowledge-base evidence.
Retrieval failure
Relevant content exists but is not returned because of query, chunking, fields, analyser, embedding, filter, recall, or ranking. Use labelled relevant items to locate the failing stage.
Access exclusion
The system need not reveal the title, count, or detailed reason for restricted content. Return a safe access message to the user and retain a security audit.
Stale index or failed dependency
When a source update is not indexed, a channel times out, or reranking fails, mark results incomplete and choose safe degradation, retry, or stop according to risk.
Source conflict
When versions, scopes, or authorities disagree, preserve the difference instead of letting rank silently choose the fact. Apply approved time, scope, and source rules or escalate.
Insufficient evidence
A relevant hit is not necessarily enough. If conditions, exceptions, attachments, another subquestion, or live facts are missing, state what can and cannot be confirmed and stop high-risk action.
Low confidence is not one threshold
Score distributions vary by query, collection, and model. Calibrate thresholds by task and combine score, count, source, margin, evidence coverage, and failure state rather than letting one similarity number decide answer or refusal.
Retrieval can leak knowledge and become an injection or resource-abuse surface
Risk begins before a final answer. Result counts, title fragments, filtering differences, and timing can expose restricted knowledge, while malicious indexed text can enter model context. Both query ingress and result egress need controls.
Query privacy
Queries can contain names, accounts, health, contracts, or commercial intent. Limit collection, access, retention, and analytics and mask or isolate them in logs, evaluation, and vendor paths.
Result side channels
Counts, titles, autocomplete, scores, timing differences, and ‘exists but forbidden’ messages can leak information. Include them in unauthorised and cross-tenant tests.
Query and filter injection
Parameterise database and search expressions and restrict fields, operators, scripts, regex, and wildcards. Model-generated queries cannot bypass application authorisation or resource limits.
Malicious content
Retrieved pages, documents, and uploads can include prompt injection, scripts, and false commands. Treat them as untrusted data, separate them from system instructions, and constrain agent tools.
Denial of service and abuse
Limit cost, timeout, concurrency, and quota for complex queries, excessive K, many subqueries, fuzzy and regex search, and vector batches. Record abuse without logging sensitive text unnecessarily.
Index and supply chain
Protect indexes, embeddings, synonyms, rankers, and configuration from unauthorised change. Confirm retention, training, region, deletion, and incident terms for managed search, embedding, and reranking providers.
Offline retrieval evaluation needs queries, relevance judgements, and an explicit denominator
‘The answer appeared in the first ten’ is one observation, not a stable metric. Define what counts as relevant, whether multiple pieces are required, and the known relevant pool before comparing recall, precision, and ranking under the same version combination.
Representative query set
Collect definitions, exact IDs, paraphrases, filters, history, exceptions, insufficiency, multi-hop, and unauthorised queries from real tasks and segment by user, language, content, and risk.
Relevance judgement
For each query, label relevant objects, graded relevance, mandatory evidence, and prohibited results and retain annotator, guideline, disagreement, and adjudication. Recall has no defensible denominator without this.
Precision@K
The proportion of relevant items among the first K results, measuring how clean the front is. Define K, relevance grade, and deduplication rules.
Recall@K
The proportion of all known relevant items appearing in the first K, measuring missed evidence. If the relevance pool is incomplete, the value only describes current labels.
MRR and ranking quality
MRR focuses on the rank of the first relevant item. Tasks needing multiple or graded evidence can add nDCG, MAP, success rate, and evidence coverage; first hit alone is insufficient.
Negative cases and refusal
Test no-answer, forbidden, ambiguous, and malicious queries for false hits and leakage. Strong positive metrics do not prove the system stops correctly.
Segmentation and error analysis
Break averages down by language, document type, question, region, access, and risk, and diagnose collection, query, chunking, filtering, recall, fusion, or reranking per failure.
Fixed version combination
Lock content, labels, chunks, index, analyser, embedding, filters, candidate count, fusion, and reranker for comparison, change one main variable at a time, and preserve results.
Production monitoring covers query drift, index freshness, access, quality, and cost
An offline set will not automatically capture new policies, terminology, or changes in how users ask. In production, query distribution, synchronisation delay, no-results, access denials, quality drift, and resource use belong in one operating view.
Task outcome
Observe whether users find and open the right source, complete a task, reformulate, or escalate. Clicks and dwell time are behavioural signals, not relevance truth without review.
Query distribution
Monitor frequent, growing, no-result, long-tail, language, and new terminology while protecting privacy and preventing automated clients from distorting statistics. Use change to extend evaluation and content, not direct self-training.
Freshness and synchronisation
Measure source change to searchable time, failure backlog, deletion propagation, and access synchronisation. Alert, isolate, or stop high-impact answers when a source exceeds its limit.
Quality drift
Use fixed sentinel queries and sampled real queries to detect ranking shifts, no results, duplicates, wrong versions, and access denials. Regress after model or index changes.
Performance and capacity
Measure query parsing, filtering, each recall channel, fusion, reranking, and assembly latency, throughput, errors, timeout, index capacity, and per-query cost separately.
Change governance
Content, synonyms, fields, analysers, embeddings, fusion, weights, thresholds, Top-K, and reranker changes need an owner, rationale, experiment, approval, release, rollback, and impact record.
Rebuild and retirement
Retain sources and configuration for rebuild. On replacement, migrate necessary logs and evaluation, revoke connections and secrets, handle provider copies, and ensure old indexes no longer serve traffic.
Example: retrieval in a procurement knowledge base
This example explains retrieval-stage responsibilities and validation. It is not a client knowledge base, real query, permission model, relevance score, threshold, or technology choice.
Question scope
Employees search required application material, current approval rules, their own request state, and exception handling. General retrieval does not decide legal interpretation, another person's sensitive request, or automatic approval.
Collection routing
Policy questions use current or specified historical indexes, supplier identity uses master data, request state uses an authorised live tool, and cases remain handling references. Do not mix everything into one vector store.
Query processing
Resolve region, organisation, request type, time, and the identity behind ‘my’, while retaining the original question. Split a question that asks both threshold and required material.
Recall and reranking
A lexical channel finds policy codes and terminology; a vector channel handles natural phrasing. Fuse after access and effective-time filtering and rerank by query-clause relevance.
Results and stops
Return current clauses, applicable delegation rows, and precise locations. Expose limitations and stop conclusions or actions on policy-training conflict, missing attachments, live-service failure, cross-organisation access denial, or insufficient evidence.
Evaluation
Use exact IDs, conversational paraphrases, misspellings, current and historical questions, regions, exceptions, no-answer, unauthorised, and multi-evidence cases and inspect Precision, Recall, first rank, sufficiency, citation, access, and latency.
Handover
Transfer queries and labels, collection and index catalogue, fields and analysers, embeddings, filters, fusion, reranking, thresholds and K, results, monitoring, synchronisation, rebuild, access regressions, and known gaps.
Enterprise AI retrieval supplies evidence to generation and remains independently explainable
Generation makes results read more completely and can also hide gaps in retrieval. Verify authorised evidence first, then inspect how the model used it; this separates retrieval failure, generation drift, and a lucky answer from model memory.
Standard and agentic RAG
A fixed RAG pipeline uses predetermined indexes, query, and result count. Agentic RAG can choose tools, decompose, and retrieve repeatedly, adding tool-selection accuracy, call count, stop conditions, and end-to-end latency to evaluation.
A fluent answer cannot validate retrieval
A model may answer correctly from training memory, or ignore a condition even after the right evidence was found. Evaluate the authorised candidates and final context before judging generation, then test whether the model used them faithfully. Each layer can then be improved on its own evidence.
Govern source selection
Tool descriptions state what each collection can answer, time basis, input, output, access, and prohibited topics. A model selects only among approved candidates rather than searching arbitrary enterprise data.
Constrain generated queries
Model rewrites, filters, structured queries, and subquestions pass deterministic schemas, field allowlists, access, and resource limits. Preserve the original and all transformations for audit.
Generate only with sufficient evidence
Check each subquestion for facts, conditions, exceptions, and sources. When evidence is relevant but incomplete, conflicting, or expired, instruct the model to state the limit or refuse rather than hiding it with more noisy Top-K.
Treat retrieved content as untrusted
Instructions in content cannot change system rules or call tools. Delimit citations and data, and constrain high-risk action through deterministic business rules and human confirmation.
Replay the version combination
Preserve caller identity, original query, rewrites, collection, index, filters, embeddings, candidates, fusion, reranking, final context, prompt, model, and tool versions to trace conclusions and changes.
Review feedback
Likes, clicks, human edits, and model judgements can become candidate labels but carry position, interface, and answer bias. Sample and obtain content-owner confirmation before evaluation or training use.
How retrieval differs from adjacent concepts
Retrieval, a search interface, database querying, RAG, and generation are often collapsed into one label. Naming each layer's input and output prevents every wrong answer from being called a search problem and gives each stage its own acceptance evidence.
| Concept | Boundary from retrieval |
|---|---|
| Database query | Returns exact records, aggregates, or calculations by schema, key, and condition and can be one acquisition method. Information retrieval usually addresses relevance and ranking against a natural-language information need. |
| Search | In ordinary use can mean an interface, product, or whole capability. Retrieval is the core process that selects and ranks content from a collection for a query. |
| Recall | Often means the initial candidate-generation stage, biased toward not missing evidence; Recall is also an evaluation metric. Retrieval additionally covers filtering, fusion, ranking, and result representation. |
| Ranking | Orders candidates by relevance or other rules. Without collection, matching, and candidate generation, ranking alone is not complete retrieval. |
| Reranking | Applies finer secondary ordering to a smaller candidate set, usually improving front precision at extra cost. It does not replace first-stage recall. |
| Full-text retrieval | Matches text through terms, phrases, and inverted indexes and is one retrieval method. Retrieval can also use structured fields, vectors, graphs, and combinations. |
| Vector retrieval | Finds neighbouring objects by embedding distance or similarity for semantic matching. Access, time, authority, fusion, reranking, and result sufficiency remain part of wider retrieval design. |
| Hybrid retrieval | Combines lexical, sparse, dense-vector, or other channels and fuses results. It is one multi-channel strategy, not every retrieval system. |
| Recommendation | Predicts potentially interesting content from users, items, and behaviour and normally lacks an explicit current query. Retrieval primarily starts with an information need and query. |
| Context assembly | Selects, deduplicates, expands, and formats retrieved evidence for a model. It can change final context but must not hide original retrieval results. |
| RAG | Calls a generative model over retrieval results to form an answer. Interfaces such as Retrieve and RetrieveAndGenerate likewise distinguish retrieval-only from retrieval-plus-generation. |
Sources and scope
This entry explains retrieval in enterprise search, knowledge-base, and AI RAG projects: identifying, filtering, and ranking candidate objects and their source information from an authorised collection in response to an information need expressed by a person or system. Retrieval is not synonymous with primary-key lookup, structured query, a web-search product, recall as a metric, ranking or reranking, full-text search, vector search, hybrid search, graph query, recommendation, context assembly, generation, or a complete RAG system; these can be methods, stages, or adjacent capabilities. No relevance formula, score threshold, Top-K value, or supposedly best algorithm works independently of content, questions, language, access, currency, and risk. This entry does not fully define search-engine architecture, inverted indexes, BM25 mathematics, embedding models, vector databases, hybrid retrieval, reranking models, query languages, database optimisation, knowledge graphs, recommendation algorithms, complete RAG or agent architecture, privacy and security programmes, or vendor configuration. Accountable project roles must validate the production approach with approved content and representative questions.
- Microsoft Azure RAG information-retrieval phase — query transformation, multi-field and multi-channel search, filtering, candidate recall, fusion, reranking, final result count, and Precision@K, Recall@K, and MRR
- Microsoft Azure RAG preparation phase — collecting representative content, real queries, the exact context answering each query, and valid answers together for stage-level evaluation
- Microsoft Azure RAG design and evaluation guide — fixed standard retrieval versus agentic RAG and separate evaluation of content, chunking, embedding, retrieval, context, and generation
- Google Cloud hybrid-search guidance — sparse lexical representation, dense semantic representation, and their combination as distinct matching channels
- Google Cloud text embeddings — dense vector representations that support semantic passage matching even when query and content do not share wording
- Amazon Bedrock Knowledge Bases Retrieve — returning source chunks, metadata, locations, and relevance scores with configurable filters, result counts, and reranking, separate from generation
- Amazon Bedrock RetrievalFilter API — metadata equality, range, set, containment, and Boolean filters applied before returning knowledge-base results
- Apache Lucene BM25Similarity API — term-frequency saturation, document-length normalisation, inverse document frequency, and implementation parameters showing that lexical scores depend on collection and configuration
- NIST TREC 2021 Overview — comparing information-retrieval systems with tasks, topics, document collections, relevance judgements, and common evaluation methods
- NIST RAG definition — a generative model paired with a separate information-retrieval system or knowledge base that supplies relevant information as generation context
- Original Microsoft MS MARCO paper — a large-scale reading-comprehension and retrieval research dataset based on anonymised real Bing queries and human answers
- Karpukhin et al. DPR paper — dual-encoder dense passage retrieval for open-domain question answering compared with sparse retrieval baselines
- Thakur et al. BEIR paper — evaluating zero-shot generalisation of sparse, dense, late-interaction, and reranking approaches across domains, query types, corpora, and tasks