Enterprise AI Project Wiki · Data, knowledge and models
Knowledge source
Also calledKnowledge origin · Information source · Source of knowledge · Fact source · Authoritative source · Content source · 知识来源
A knowledge source is a traceable origin recognised by an organisation for answering a stated question, interpreting a business rule, reconstructing a fact, or supporting an action. It can be a controlled record, business-system data, approved document, human expertise, or external publication. A governed source identifies its owner, scope, version or effective time, access and use rights, acquisition and transformation path, quality limits, and accountability for resolving conflicts.
A knowledge source is a traceable origin for a conclusion, not a paragraph found by search
The question 'who approves this purchase?' may require the current delegation rules. 'Who approved this request at the time?' needs the transaction record and the rule then in force. 'Why was this threshold adopted?' may also need the approval minutes and policy basis. Similar wording does not make one retrieved document appropriate for all three questions.
ISO 30401 frames knowledge management around organisational objectives, people, processes, and continual improvement. W3C PROV describes provenance through entities, activities, and responsible agents. A project knowledge source therefore includes not just content, but who produced or approved it, through which activity, when it applied, and how it reached the current answer.
Sources may be database records, policies, contracts, manuals, decision minutes, expert confirmation, or external publications. Model-written summaries, chunks, and vector indexes are derived representations. If they cannot lead back to an original version and location, they cannot independently bear responsibility for a fact.
Enterprise knowledge comes through many carriers, not just PDF files
Formal rules may live in policy, current state in a business system, exception authority in a delegation table, and operating knowledge with people. Inventory sources around the evidence a question needs so documents, live records, and accountable experts each provide their proper part.
Policies and normative material
Approved policies, standards, procedures, work instructions, and delegation tables state what must, may, or must not happen. Distinguish formal, explanatory, template, and superseded material.
Business systems and data services
ERP, CRM, case, master-data, contract, stock, order, and status services provide current or historical facts. Screenshots and personal exports are usually copies, not automatic authority.
Official records and evidence
Approved requests, signed contracts, decisions, acceptance confirmations, audit records, and communications can attest to an activity or decision. Content, context, and structure all affect trustworthiness.
Product and technical material
Manuals, API documentation, release notes, configuration guidance, known issues, and supplier support responses explain technical behaviour but must be tied to product, version, region, and support status.
Human expertise
Subject experts, service staff, operations, and field teams hold exceptions and tacit judgement not yet documented. Record the person's role, question, date, confirmed scope, and responsibility for formalisation.
Project artefacts
Approved requirements, rules, design decisions, test evidence, issue closure, and handover material explain the project. Drafts, chats, and personal notes need an explicit status and owner before use.
External public or subscribed material
Government, standards, industry, research, supplier, database, and news sources add external facts. Verify publisher identity, original page, update practice, licence, and geographical scope.
Derived content
Summaries, terminology maps, knowledge graphs, chunks, translations, tags, and AI-authored explanations improve retrieval but must retain source version, transformation activity, and review state.
Authority is specific to a question, object, time, and accountable remit
A senior-issued document does not answer every detail, while a fresh frontline record may have no authority to interpret policy. Ask which questions, objects, and period a source covers and who owns interpretation. Authority is not a permanent overall score.
State the knowledge question
Replace 'understand procurement' with answerable questions: who may approve, what the current threshold is, what state one request had at a stated time, or who handles an exception. Each needs different sources and time semantics.
Set object and grain
Identify whether the source concerns policy, process, entity attributes, one transaction, a decision, a technical version, or advisory experience. A department manual cannot establish the actual state of one case.
Confirm issuing and decision authority
Record who may set rules, confirm facts, maintain master data, sign records, or publish externally. Author, custodian, administrator, and business decision owner may differ.
Limit applicability
Specify organisation, legal entity, department, product, region, customer, channel, language, contract, and scenario. Do not guess precedence between group policy and local procedure from document rank alone.
Limit effective time
Distinguish creation, publication, approval, effective, expiry, transaction, and acquisition times. The latest document may answer current policy while incorrectly rewriting a historical decision.
Separate normative and explanatory
Normative sources prescribe behaviour; guidance, training, FAQs, and examples aid interpretation and may simplify. If an FAQ has formal decision authority for a question, record that grant explicitly.
Confirm the system of record
An organisation may designate a system for a class of records or attributes, but that does not make it the only truth for every question. Register authority at attribute, time, and decision level where needed.
A source register must let people find, assess, and reproduce the source
A saved link may point to changed content months later and cannot show which version supported an earlier answer. Record owner, scope, version, effective period, access, transformation state, and supersession so a decision can be reproduced.
- Stable identity
Assign stable IDs to the source and its version; record formal name, type, original location, and acquisition method. A filename is not sufficient identity.
Source ID, original URI/system object, version or snapshot ID
- Accountable agents
Record content owner, approver, maintainer, technical custodian, and conflict adjudicator separately rather than one ambiguous owner.
Role register, approval, maintenance and escalation path
- Applicable scope
State answerable questions, business objects, organisation and region, language, product or process, inclusion, and exclusions.
Purpose statement, scope tags, known non-applicable cases
- Time and status
Preserve creation, publication, approval, effective, expiry, acquisition, and review dates, plus draft, active, superseded, withdrawn, or archived status.
Version history, state events, effective interval
- Rights and protection
Record ownership, licence, contractual use, personal/confidential class, permitted users, geography, copying, model-processing, and citation restrictions.
Licence/contract reference, classification, access policy
- Structure and language
Record format, encoding, language, pages or anchors, tables and images, attachments, scanning/OCR, and machine-parsing limits.
Format inventory, parsing report, accessibility and OCR result
- Provenance and integrity
Preserve original publisher, acquisition channel, prior version, transformation activity, checksum, and derivative relationships so summaries and chunks return to the original.
Provenance chain, processing log, checksum, derivative map
- Quality and risk
Record accuracy, completeness, currency, consistency, representation, known gaps, conflicts, review findings, accepted risk, and reassessment triggers.
Quality assessment, issue log, approved limitations
Onboarding preserves the original, transformations, and the responsibility chain
Scanning, parsing, table extraction, and conversion can lose headings, footnotes, negation, or cell relationships. Preserve links between original and derivatives, processing versions, and quality checks so an error returns to a transformation step instead of being blamed vaguely on the model.
1. Request from questions and risks
State the answers or actions, intended users, error consequences, and existing gap. Do not onboard an entire folder merely because it exists.
2. Confirm owner and rights
Ask the content or system owner to confirm formal status, applicability, access, copying, AI processing, cross-border, retention, and citation conditions.
3. Acquire a verifiable original
Prefer the formal repository, record system, approval channel, or original publisher. Preserve version, snapshot, acquisition time, and checksum rather than relying on a forwarded copy.
4. Analyse structure and dependencies
Find body, tables, images, attachments, links, terminology, cross-references, local supplements, external systems, and past versions. Missing attachments make completeness uncertain.
5. Transform without replacing the original
OCR, parsing, conversion, translation, cleaning, chunking, and tagging create separate derivatives with tool, parameter, time, error, and human-correction records.
6. Validate content and task fitness
Compare original and derivative, sample tables, negations, exceptions, headers, footers, formulae, times, and links, then use representative questions to test whether the source supports the intended conclusion.
7. Approve scope before publication
Appropriate business, content, data, records, security, or legal roles approve questions, users, duration, and limits. A technically successful import is not approval.
8. Establish monitoring and exit
Subscribe to changes, review periodically, and collect user feedback. Supersession, withdrawal, rights changes, or quality failures stop retrieval and trigger treatment of indexes, caches, and derivatives.
Source quality is fitness for a purpose, not one overall score
A concise handbook may be excellent for learning but insufficient for a compliance decision; primary law is authoritative but difficult for ordinary users. Judge accuracy, completeness, freshness, intelligibility, and risk for the intended use rather than assigning one score to every purpose.
Authenticity and identity
Verify that the claimed publisher, system, expert, or record produced the source and acquisition did not confuse a draft, mirror, retelling, or altered copy with formal content.
Accuracy and reliability
Assess whether content correctly represents the asserted fact, activity, or rule and whether its creation process is dependable. Formal identity does not make every value error-free.
Completeness and context
Include body, attachments, tables, footnotes, exceptions, definitions, and preconditions needed to understand who decided what, when, and for which case. Isolated passages can reverse meaning.
Currency and point-in-time fitness
Use an active version for current questions and retain the then-effective version for historical questions. Expose update delay, snapshot cut-off, and source-system synchronization lag.
Consistency and explainable conflict
Compare terms, values, states, and accountability across sources. Where they differ, show applicability, version, or pending adjudication rather than silently overwriting at ingest.
Coverage and representation
Assess target regions, products, roles, languages, normal and exceptional situations. Repeated common FAQs do not fill rare high-impact gaps.
Availability and locatability
Authorised users must be able to retrieve and interpret content, and systems must locate a page, section, record, or point in time. A summary without access to the original impedes verification.
Parsability without oversimplification
Extraction must preserve table relationships, headings, images, formulae, and negation. Machine convenience cannot change business meaning.
Conflict, staleness, and insufficient material must be visible outcomes
When retrieval finds two answers, silently choosing the more plausible text is the dangerous option. Define precedence, effective time, and decision authority in advance. Showing the difference and stopping a consequential conclusion can be the correct result.
Determine whether it is a real conflict
Compare object, organisation, date, version, terminology, and grain. Group policy and a local procedure, or a current rule and historical record, may both be correct in different contexts.
Execute approved precedence rules
For each question, register the role of applicable law or contract, approved policy, attribute system of record, signed record, and explanatory material. The precedence rule needs its own owner and version.
Never merge silently
Preserve provenance and time when values, states, or responsibilities disagree. Mark the conflict and block high-impact automation rather than letting a model average, guess, or choose a fluent answer.
Create an adjudication record
Record the question, source versions, impact, temporary control, adjudicator, conclusion, rationale, effective time, and derivatives or downstream systems requiring correction.
State insufficiency precisely
When coverage, active version, attachment, record completeness, or permission is missing, say what is absent, what can be established, and who can resolve it. Do not fill a project fact from generic knowledge.
Preserve minority and dissenting views
Where experts or research legitimately disagree, record positions, evidence, and limits. Only a conclusion with decision authority becomes a normative answer.
Visible content is not necessarily licensed or authorised for ingest, retrieval, or output
An employee's access to a contract in its source system does not automatically permit copying it to a global knowledge base, sending it to a supplier model, or citing it to another role. Confirm rights separately for acquisition, storage, indexing, model processing, and output.
Use rights and licences
Confirm rights to crawl, copy, index, embed, translate, summarise, process with models, share internally, and cite externally. Public accessibility is not unlimited reuse permission.
Personal and confidential material
Identify personal data, trade secrets, contracts, credentials, restricted records, and export or regional restrictions, then minimise, mask, isolate, or decline onboarding according to purpose.
Inherit source permissions
Filter the knowledge base and retrieval by user, tenant, object, attribute, and purpose. Revocation from the source must propagate to chunks, indexes, vectors, and caches.
Treat external content as untrusted
Web pages, email, documents, and tool results may contain malicious instructions or deceptive content. Treat them as data, separate instruction layers, restrict tool authority, and test prompt injection.
Secrets and active facts
Do not index passwords, API keys, private keys, or one-time tokens as ordinary knowledge. Query live price, stock, and account state through authorised tools instead of stale text copies.
Supply chain and exit
Record whether connectors, parsers, embedding or model services retain or train on inputs, use subprocessors, delete data, and report incidents, with alternatives for service withdrawal or failure.
Provenance explains how content formed; citation shows what supports this claim
Provenance shows how content moved from an original through transformations; citation lets a reader check one claim. Both matter. Provenance without a readable citation frustrates users, while a link without version and processing history prevents reproduction.
Preserve the provenance chain
W3C PROV models generation, use, derivation, revision, and responsibility among entities, activities, and agents. Connect originals, parsed forms, chunks, translations, summaries, indexes, retrieved results, and answers.
Cite a specific version and location
Include source ID, title, version or effective time, and page, section, table, record, or stable selector. A filename or website home page cannot reproduce the evidence.
Separate source, evidence, and citation
The source is the origin, evidence is the content supporting one claim, and citation is the representation linking claim to evidence. Listing sources does not support every sentence.
Test faithfulness, completeness, and sufficiency
NIST citation-quality work asks whether the source supports the claim, whether omitted context changes the message, and whether evidence carries the required burden. Apply human or automated checks proportionate to risk.
Preserve multi-source attribution
For a synthesis, identify which source supports each fact, rule, and inference. Do not attribute a secondary summary as the original publisher's words or decorate a mixed paragraph with one citation.
Handle restricted sources
Authorised users may see necessary title, version, and location. For others, citations must not leak filenames, people, paths, or passages; provide a controlled source category or human channel.
Knowledge sources change, are superseded, and fail; they need continuing maintenance
Policies are revised, web pages update silently, and APIs change field meaning. Maintenance detects change, assesses impact, refreshes indexes and caches, and retains history. Otherwise the system can continue citing obsolete content fluently.
Change detection
Use official notices, version APIs, repository events, periodic checks, owner confirmation, and user feedback to discover releases, withdrawal, dead links, and permission or licence changes.
Impact analysis
Identify affected questions, rules, chunks, tags, indexes, evaluation samples, response templates, tool actions, and downstream systems. One changed page may alter multiple conclusions.
Coexisting versions
Route current answers to the new effective version while historical answers retain the previous version and interval. Do not overwrite the original and erase the basis of past decisions.
Reprocessing and regression
Parse, chunk, embed, and index again; compare counts, locations, permissions, and retrieval, then test that additions, modifications, deletions, negations, and exceptions take effect.
Withdrawal and quarantine
When authenticity, rights, integrity, or security fails, stop new use, isolate derivatives, assess prior answers and actions, then repair, replace, or notify as required.
Retention and disposal
Retain originals, versions, processing, and approvals according to business, records, contract, privacy, and audit requirements. Delete controllable copies, indexes, vectors, and caches on expiry.
Accountability and service expectations
For high-impact sources, define owner, review interval, change-notice target, incident contact, alternative source, and recovery objective. An unmaintained 'authoritative document' quickly loses authority.
Constructed example: procurement source registration and conflict handling
This customer-independent procurement question shows policy, delegation table, transaction record, and operating instruction carrying rule, value, fact, and procedure evidence. The focus is how to stop on conflict or absence and reach an authorised decision owner.
Current approval threshold
Use the approved delegation table active on the question date and record legal entity, category, currency, tax basis, version, and approver. Training slides explain but do not override it.
Facts about one request
Amount, currency, state, submitter, and approval events come from the recognised procurement record system and append-only history. Screenshots and forwarded email do not replace that record.
Supplier identity
Canonical supplier ID, legal entity, and active state come from attribute-level master-data authority. The name in a contract is transaction-time evidence and must not silently rewrite current master data.
Exceptional process
The active procedure explains over-threshold, insufficient material, separation-of-duty, and unknown ERP-state handling. If a diagram and text conflict, an approved rule stops automatic submission and raises a case.
Expert supplement
For an undocumented operational exception, record the question, expert role, date, scope, and temporary expiry and send it to the policy owner. Unapproved experience does not become a permanent deterministic AI answer.
Answer and citation
Cite the exact delegation clause, request event ID, and supplier-record version, separating facts, rules, and inference. If a source is missing, state the gap and accountable channel rather than inventing a threshold or status.
Enterprise AI selects an eligible source before retrieval and verifies each claim after generation
RAG can place relevant passages before a model but does not decide whether they are authorised, current, or sufficient. Reliable answers narrow eligible sources before retrieval and test claims against citations afterward. Omitting either stage can produce a wrong answer with impressive links.
Define an answerable scope
Register permitted questions, users, organisations, time semantics, and risk for each source. 'We ingested 100,000 documents' is not a capability boundary; unsupported questions require clarification, refusal, or human handoff.
Filter authority and time before retrieval
Filter by tenant, user, object, source state, effective interval, language, and region before keyword, vector, or hybrid retrieval. A model cannot decide permissions after seeing restricted content.
Separate source content from instructions
Documents, web pages, email, and tool results are untrusted content and cannot change system instructions, expand tool authority, or request other sources. Injection detection is an additional control, not the boundary.
Keep context with each chunk
Carry source ID, version, heading path, page/section, effective dates, permission, and adjacent definitions or exceptions. If a hit loses a negation, table heading, or precondition, expand context or stop.
Separate retrieved fact and model inference
Label direct source facts, cross-source synthesis, calculations, and model inference. High-impact rules and tool parameters come from deterministic data or rules, not guessed natural-language snippets.
An answer can carry links without being supported by them
Check each citation for direct support of the current claim, omitted exceptions and conditions, and an existing version and location. A model may attach a footnote merely because a source concerns the same topic. Without an evidence relationship between link and conclusion, the answer is not grounded.
Stop on conflict or weak evidence
When eligible values conflict, only secondary material exists, a source is stale, an attachment is missing, or evidence is insufficient, expose the difference and block high-impact answers or actions for adjudication.
Evaluate real source changes
Test current and historical questions, regional rules, conflicts, withdrawals, inaccessible material, malicious documents, tables, and multi-hop sources; measure retrieval coverage, selection, citations, refusal, leakage, and tool side effects.
Version the complete assembly
Record model, prompt, retrieval, reranker, source inventory, original and derivative versions, permission policy, tools, and evaluation set so a bad answer can be replayed and attributed to source, transformation, retrieval, or generation.
How a knowledge source differs from neighbouring concepts
A source is the origin of content or fact, a knowledge base provides sources under control, a retrieval result is a candidate selected for one query, and a citation connects a claim. Separating them prevents “ingested” or “retrieved” from being mistaken for eligible evidence and proven answer.
| Concept | Boundary with a knowledge source |
|---|---|
| Knowledge base | A governed collection or system that organises, stores, indexes, and serves knowledge. Sources are the recognised origins of its content, and one knowledge base can ingest many sources. |
| Source document | A document-carried source. Database records, APIs, human confirmation, and event streams can also be knowledge sources, so the scopes differ. |
| Data source | Emphasises where data is obtained. A knowledge source also states what understanding, rule, or decision it can support and the associated accountability, scope, and citation; the two can overlap. |
| Authoritative source | A source recognised within a defined scope to decide a class of fact or rule. Not every knowledge source is authoritative; some are explicitly limited supporting, external, or expert sources. |
| System of record | A designated location for a class of records, entities, or attributes. It may be authoritative for some questions but does not automatically own policy interpretation, external facts, or every historical context. |
| Evidence | Content used to support a particular claim or decision. A source can contain much material, but only relevant and sufficient parts are evidence for that claim. |
| Citation | A representation linking a claim to specific evidence with identity, version, and location. Its presence does not prove source authenticity, faithful interpretation, or sufficiency. |
| Provenance or lineage | Describes how entities were generated, transformed, and derived by activities and agents. It explains how knowledge arrived but cannot alone prove correctness or permission. |
| Reference material | Offers background, explanation, or further reading and may have no formal decision accountability. Mark permitted questions so reference material is not silently promoted to a normative source. |
| Training corpus | Adjusts model parameters and normally cannot be retrieved and cited item by item at answer time. Its presence does not ground an answer in project knowledge sources. |
| Model output | Derived content that may be a lead, summary, or draft. Without source verification and accountable confirmation, it cannot loop back and establish itself as fact. |
Sources and scope
This entry explains a knowledge source in enterprise knowledge-base, retrieval, and AI projects: a person, record, data system, document, or external resource that supplies original content, business facts, accountable confirmation, or provenance for an answer, interpretation, decision, or action. A knowledge source is not every searchable item, nor is it identical to a knowledge base, source document, reference material, data source, evidence, citation, authoritative source, system of record, or model-training corpus. No source ranking works for every question; a source may be authoritative for one object, time, or decision and merely informative for another. This entry does not fully define a knowledge-management system, records management, data governance, content management, document chunking, retrieval ranking, copyright licensing, fact-checking, archival appraisal, legal evidence, regulated source hierarchies, or a complete RAG architecture. Accountable project roles must approve what each source may answer, precedence, AI use, retention, and citation according to purpose and risk.
- ISO 30401:2018: requirements for establishing, implementing, maintaining, reviewing, and continually improving an organisational knowledge-management system; revision is under development
- ISO 15489-1:2016: records, systems, metadata, policies, accountability, business context, controls, creation, capture, and management across formats and technologies
- ISO/IEC 5259-2:2024: measurable fitness-for-purpose quality characteristics for structured and unstructured data used in analytics and machine learning
- W3C PROV-O Recommendation: interoperable provenance through entities, activities, agents, generation, use, derivation, revision, and attribution
- W3C DCAT 3 Recommendation: resource-catalogue identity, versions, lifecycle, provenance, quality, citation, data services, and agent relations
- W3C Web Annotation Data Model: linking annotations and citations to resources, precise segments, time states, agents, motivations, and rights
- W3C Data on the Web Best Practices: licences, provenance, quality, versions, coverage, feedback, original-publication citation, and persistent identifiers
- NIST AI RMF Core: AI data and third-party sources, documentation, accountable roles, risks, measurement, and continuous lifecycle governance
- NIST AI RMF Playbook Manage: documenting data sources, origins, transformations, augmentations, labels, dependencies, constraints, metadata, necessity, and obsolescence
- NIST AI 600-1: generative-AI source and content provenance, fact-checking, third-party data, privacy, security, and continuous evaluation
- NIST agentic-AI evaluation probes: strict evaluation of citation faithfulness, completeness, and sufficiency against trusted source material
- UK government reference-data guidance: trusted sources, systems of record, persistent identifiers, owners and stewards, update history, expired versions, and metadata
- Microsoft input, context, and retrieval hygiene: permission-aware indexes, document access, metadata filtering, citations, untrusted external content, and prompt-injection defence