Enterprise AI Project Wiki · Data, knowledge and models
Masked data
Also calledDe-identified data · Pseudonymised data · Anonymised data · Sanitised data · Data masking · Tokenised data · 脱敏数据
Masked data is data whose direct identifiers, linkable attributes, sensitive content, and relationships have undergone documented suppression, replacement, generalisation, perturbation, or another protective treatment for a stated purpose, recipient, environment, and threat model, followed by validation of task fitness and residual disclosure risk. The term describes a controlled treatment and its result; it does not by itself mean the data is anonymous, irreversible, safe for publication, or outside continuing protection.
Masked data names a treatment; the label alone cannot prove safety
Projects often call redacted names, partly hidden phone numbers, substituted customer IDs, generated test copies, and even encrypted storage 'masked'. These controls change different objects, address different attack surfaces, and have different reversibility. ISO/IEC 20889 supplies terminology and a classification for de-identification techniques, while ISO/IEC 27559 addresses re-identification risk across the data lifecycle; neither supports replacing a precise treatment statement with a vague label.
Effectiveness depends on who receives the data, what they may do with it, where it is processed, whether mappings or other datasets are available, how much background knowledge an attacker has, and the consequences of disclosure. A copy acceptable for a tightly controlled test team may be wholly unsuitable for public download.
A deliverable therefore needs more than a field marked 'masked'. Record the source and output dataset identities, purpose, field-level methods and parameters, reversible material, recipient and environment, validation, residual risk, approver, validity period, and disposal method.
Define the protected object, permitted purpose, recipient, and attacker capability first
The same data sent to an internal test team, an outsourced supplier, or public research needs different protection. Name the person or entity protected, retained use, other information available to the recipient, and plausible re-identification methods. Strength is meaningful only in that context.
Subjects and business objects
Identify people, employees, customers, suppliers, patients, devices, accounts, contracts, prices, credentials, trade secrets, or other protected objects. Identifiability risk and enterprise-confidentiality risk can coexist, but their authority and consequences differ.
Approved purpose
State whether the data supports functional or performance testing, analytics, model training, incident reproduction, demonstration, or cross-organisation research, and why each field is necessary. Convenience is not a reason to retain avoidable data.
Recipients and powers
Name teams, suppliers, tenants, service accounts, and automated jobs, together with their query, export, join, redistribution, and original-system access capabilities.
Processing environment
Describe network, storage, backup, administration, and cross-border boundaries as the data moves through development, testing, analytics, cloud services, model platforms, logs, and endpoints. Lower-trust environments need less data or stronger treatment.
Available auxiliary information
Assess public registers, social information, transaction records, internal directories, time and location data, other datasets, and query services reasonably available to the recipient. Innocent-looking attributes may identify a person after linkage.
Threats and consequences
Distinguish accidental viewing, insider misuse, intrusion, mapping compromise, brute force, frequency analysis, differencing queries, and model extraction, then assess identity exposure, discrimination, fraud, commercial loss, and security impact.
Inspect direct and combined identifiers, content, metadata, and relationships
After names and telephone numbers disappear, rare roles, exact times, locations, free text, and relationship graphs may still identify someone. Risk belongs to the whole record combined with external information, so structured fields, attachments, logs, and metadata all matter.
Direct identifiers
Names, government identifiers, account numbers, exact contact details, biometrics, and assigned unique numbers that directly locate an object usually require suppression, substitution, or strict isolation.
Indirect or quasi-identifiers
Age, postcode, role, organisation, dates, locations, and rare events may not identify alone but can narrow a record to one or a few people when combined or linked externally.
Sensitive attributes
Health, finance, salary, appraisal, complaint, transaction, permission, contract-price, or risk findings can cause harm even without identity. Hiding a name cannot be the entire protection objective.
Free text and documents
Notes, email, tickets, CVs, contracts, and OCR text may contain names, addresses, signatures, contextual identity clues, and secrets. Regular expressions find known patterns; they cannot prove that all sensitive content is gone.
Images, audio, and video
Faces, voices, number plates, screens, background locations, embedded thumbnails, and annotations in medical imagery may expose identity or confidential information and need media-specific detection, editing, and sampling.
Metadata and technical traces
Authors, paths, device IDs, timestamps, GPS, database keys, log correlation IDs, version history, and hidden sheets can bypass visible redaction. Export formats and attachments belong in scope.
Relationships, frequency, and rare values
A unique reporting line, unusual purchase, extreme amount, small population, or event sequence can become a fingerprint. Safe-looking fields do not make a row, table, or multi-table join safe.
Derived values and model features
Scores, vectors, clusters, embeddings, and features can encode source information. Removing original columns does not eliminate membership inference, attribute inference, or approximate reconstruction risk.
Select techniques by protection objective and record the utility they remove
Substitution, generalisation, perturbation, truncation, and synthetic generation solve different problems and damage different uses. Stronger is not always better: broken date order, entity relationships, or boundary values can make testing misleading. Record risk reduction and utility loss together.
Suppression and redaction
Remove whole columns, rows, text spans, image regions, or attachments when the purpose does not need them. Check hidden fields, revision history, caches, and backups for residual originals.
Static and dynamic masking
Static masking produces a transformed copy. Dynamic masking substitutes query results while the stored value remains unchanged. The latter limits ordinary display but does not mean the source has been transformed.
Substitution, shuffling, and pseudonyms
Fictitious values, same-domain values, record shuffling, or stable pseudonyms can preserve format or joins. Stability and reversibility improve some test uses while increasing linkage, frequency, and mapping-compromise risk.
Tokenisation
A meaningless token replaces a value and a controlled vault or service can restore it. Strength depends on generation, vault security, permissions, separation, audit, availability, and compromise response; tokenisation is not anonymisation.
Generalisation and aggregation
Convert exact age to a band, location to a region, or detail to group statistics to reduce uniqueness. Bands, groups, and minimum cells must follow the distribution and attack tests rather than a reusable template.
Perturbation and swapping
Add noise, rounding, offsets, or swaps to values, dates, categories, or relationships. Validate business constraints, totals, correlations, tails, and anomaly patterns against the stated task.
Encryption
Protect confidentiality at rest or in transit and restore plaintext with a key. It does not reduce what an authorised processor sees after decryption and is not de-identification; key and access management remain separate controls.
Synthetic data
Rule- or model-generated records can reduce reliance on source examples, but ordinary synthesis may memorise training records or preserve rare combinations. Claim only the privacy guarantee supported by the mechanism and evidence.
Masking, pseudonymisation, de-identification, and anonymisation are not a universal linear scale
These terms vary across standards and organisations. Pseudonymous data with a reversible mapping can still reconnect to a person, while anonymisation depends on available external information and reasonable means. Describe treatment and residual risk instead of relying on a reassuring label.
| Treatment state | Limited conclusion it supports |
|---|---|
| Display masking | One interface or query does not display the whole value; source storage, other queries, administrators, and inference attacks may still reveal it. |
| Static masked copy | Specified fields in an output copy were transformed; omissions, multi-table relationships, external linkage, rare values, and reversibility still need examination. |
| Pseudonymised data | Identifiers were replaced and separate additional information can restore association. Under UK GDPR it remains personal data; other regimes require their own assessment. |
| De-identified data | Association with identity has been reduced toward a stated objective. The recipient, attacker model, metrics, sharing model, and ongoing evaluation must be explicit. |
| Anonymous data | This is a high-threshold, context-specific conclusion about identifiability under the applicable regime, not a transformation button. A party retaining recovery information will commonly be unable to make that claim about its own holding. |
| Public-release data | Uncontrolled recipients, long-lived external information, and unlimited copying require another assessment. Acceptability inside a controlled environment does not imply publishability. |
A reviewable treatment chain connects source data to a usable copy
Masking is not downloading data, running a script, and uploading the result. Source scope, approval, treatment environment, rule version, utility validation, release target, and expiry form one chain, while temporary files and logs must not leak either source or result.
1. Register the request
Record purpose, requested fields and population, recipient, environment, duration, business owner, and why a lower-risk source cannot meet the task.
2. Discover and classify
Inventory tables, fields, relationships, documents, media, metadata, derivatives, and secrets; identify personal, commercial, and security sensitivity and authoritative sources.
3. Minimise scope
Remove unnecessary records, columns, history, attachments, and precision. Do not copy a complete production estate when sampling, aggregation, or generated data is sufficient.
4. Build the threat model
State recipient powers, auxiliary data, query budget, recovery material, attack paths, and acceptable residual risk, and obtain approval from accountable roles.
5. Design field and relationship treatments
Choose suppression, pseudonyms, tokens, generalisation, perturbation, or synthesis for every object, including cross-table consistency, business constraints, time, text, and attachments.
6. Execute in an isolated zone
Restrict source access and fix tool, code, rule, and parameter versions. Intermediate files, failed outputs, logs, and temporary credentials receive equivalent protection and cleanup.
7. Validate utility and risk
Check schema, relationships, and task outcomes, then have an appropriately independent role perform disclosure review, linkage trials, sampling, and attack tests.
8. Approve and package delivery
Release only the approved version with purpose limits, residual risk, access controls, checksum, expiry, no-re-identification and no-redistribution conditions, and accountable contacts.
9. Monitor, reassess, and dispose
Reassess changes in environment, recipient, outside data, fields, model, or purpose. On expiry, revoke access and address controllable copies, caches, indexes, logs, and backups.
Every masked version needs a reproducible treatment record
Two files both labelled masked may use different sources, rules, and keys. The version record lets the team reproduce treatment, know who can reverse a mapping, and decide whether an old copy remains suitable after a source field or purpose changes.
- Dataset identity
Record output ID and version, generation time, source snapshot, selection condition, row count, and integrity checks.
Inventory, snapshot ID, query/export version, checksum
- Purpose and recipient boundary
State approved task, users and systems, permitted environment, query/export powers, region, duration, and prohibited uses.
Request, approval, data-use terms, environment inventory
- Field and content rules
For every field and unstructured object, record retention, suppression, substitution, generalisation, perturbation, parameters, and preserved relationship constraints.
Field map, rule library, text/media policy, data dictionary
- Tool and code baseline
Preserve transformation code, tool and dependency versions, configuration, random seed, or key reference without exposing actual secrets.
Controlled commit, build artefact, configuration summary, key reference
- Reversible material
Identify owners, locations, permissions, purposes, and destruction conditions for token vaults, pseudonym maps, keys, or re-identification services, kept separate from the output.
Key/mapping register, permission and audit record
- Validation and residual risk
Record task fitness, omission scans, uniqueness and linkage analysis, attacks, findings, accepted limits, and reassessment triggers.
Validation report, sample review, risk decision, issue-closure evidence
- Approval and lifecycle
Obtain confirmation from the business, data, privacy, security, and user roles that actually bear responsibility; record release, receipt, change, revocation, and destruction.
Approval record, distribution log, expiry job, destruction evidence
Validation must ask both whether the data still works and whether it can still identify
Checking only that sensitive fields vanished can produce a safe but useless shell; checking only that workflows run may leave identity clues. Test relationships, distributions, and boundaries through real tasks and attempt linkage, inference, and rare-combination attacks.
Structural integrity
Check types, lengths, formats, mandatory fields, uniqueness, foreign keys, hierarchies, and multi-table references so treatment does not create orphans, invalid enumerations, or unloadable data.
Business utility
Run representative approved tasks across state changes, rule combinations, totals, time order, interfaces, and exceptional paths. Averages and non-null rates are not enough.
Statistical fidelity
Examine distributions, correlations, rare categories, extremes, time trends, and group differences. Performance or AI uses also need realistic densities, text lengths, and difficult cases.
Content omissions
Scan text, attachments, media, metadata, hidden fields, export history, and error logs, with manual review of high-risk samples. Measure false negatives in the detector itself.
Uniqueness and small groups
Test whether attribute combinations narrow to one person or a small group, especially rare roles, locations, dates, conditions, events, amounts, and relationships. Thresholds are context-specific.
Linkage and inference attacks
Simulate a recipient joining reasonably available public or internal information and attempting frequency, dictionary, brute-force, differencing, membership, and attribute inference attacks.
Recovery controls
Verify ordinary recipients cannot reach keys, token vaults, or pseudonym mappings and that authorised restoration requires strong identity, least privilege, dual control or a ticket, audit, and alerting.
Independent review and reassessment
Have a suitable role other than the transformation author review high-risk data. New fields, recipients, sources, external information, or attack methods can invalidate an earlier conclusion.
Masking does not replace access, key, environment, and lifecycle controls
Risk-reduced data may remain sensitive and can become identifiable when combined with a mapping table, older version, or external source. It still needs least access, isolation, audit, key control, copy restriction, and expiry—not an open share because its filename says masked.
Source and treatment zones
Expose source data only to approved processing identities and run treatment in an isolated zone. Do not download source snapshots to personal devices or move them through ordinary collaboration tools.
Mapping and key separation
Manage pseudonym maps, token vaults, and decryption keys separately from output data; a general account should not hold both powers. Audit access, recovery, and export.
Least access and purpose limitation
Give recipients only the records, fields, and duration necessary and restrict bulk export, cross-database joins, copying, and redistribution. Pair dynamic masking with row/column permissions, audit, and abuse controls.
Suppliers and cloud services
Determine whether providers retain inputs, train services, generate logs, use subprocessors or cross-region transfer, and can delete data and report incidents. Calling an upload masked is not a substitute for review.
Version and change control
A change to source schema, rule, tool, parameter, or purpose creates a new version. Old approval does not cover new fields or recipients; failed transformations stop delivery and retain evidence.
Disclosure and re-identification incidents
Define stop, quarantine, revocation, investigation, notification, correction, and re-release paths for erroneous distribution, omissions, mapping compromise, abusive queries, or discovered re-identification.
Expiry and retirement
Delete or archive outputs, intermediates, model-input caches, search indexes, and controllable backups on schedule and revoke accounts, tokens, and keys, while retaining audit evidence required by law or contract.
Constructed example: a procurement-process test copy
This customer-independent procurement test copy goes beyond field replacement to time, relationships, free text, and mapping authority. The aim is to preserve business structure needed for tests while stating residual identification risk and permitted use.
Purpose and scope
The objective is to reproduce request, approval, ERP submission, and reconciliation in an isolated test environment. Select only necessary requests from an approved period and state range; do not copy unrelated history, mailboxes, or the entire supplier estate.
People and organisations
Replace employee names and contacts with consistent fictitious values and use stable test pseudonyms for self, delegation, and approval relationships. Generalise organisation hierarchy to the needed level and withhold the mapping from testers.
Suppliers and payment data
Replace legal names and registration numbers and use non-payable test formats for bank accounts. Remove real payment credentials, tax attachments, and high-risk contacts instead of retaining actual numbers with a few masked characters.
Amounts and dates
Apply an approved offset or banding method while preserving currency, tax, and above/below-threshold relationships. Shift related dates consistently to retain ordering and duration, then retest totals, boundaries, and exceptions.
Text and attachments
Use detectors to flag notes, contracts, and invoices, then manually sample high-risk content. Where only document structure matters, replace it with a fictitious same-format attachment and remove author, path, thumbnail, and revision metadata.
Validation and delivery
Validate joins, approval routes, amount reconciliation, ERP idempotency, and recovery while attempting linkage through directories, time, and rare combinations. Deliver the dataset and rule versions, utility results, residual risk, access duration, and cleanup job.
Enterprise-AI masking must span training, retrieval, models, logs, and outputs
AI data travels among training sets, vector stores, prompts, tools, logs, and human feedback. Hiding several fields before a model upload cannot control embedding leakage, free-text secrets, reconstructed output, or downstream copies. Follow the data path and protect each station, not only the entrance.
Map the complete data path
Inventory pre-training or fine-tuning data, retrieval knowledge, embeddings and vector stores, prompts and conversations, tool input/output, human feedback, evaluation sets, operational logs, caches, and monitoring—not only the structured upload.
Treated training material does not prove the model cannot leak clues later
A model may memorise rare passages or combine apparently harmless attributes into identifiable information. Even after source treatment, test membership inference, attribute inference, and training-content extraction and control model, log, and output use. Masking is one risk control, not proof of model safety.
Authorise deterministically before retrieval
Do not retrieve everything and ask a model to remove secrets afterwards. Filter by tenant, user, object, attribute, and purpose before retrieval, return minimum passages, and stop prompt injection from bypassing permissions or driving tool export.
Preserve evidence relationships in documents
After replacing names, numbers, and organisations, questions, answers, and evidence must remain consistent. Random replacement may break cross-paragraph reasoning; stable pseudonyms improve it but increase linkage risk.
An embedding is not an anonymous copy
Vectors can disclose source topic, membership, or sensitive attributes and retrieve exposing passages. Vector stores inherit source permissions, deletion, tenant separation, versioning, and incident response.
Treat prompts and logs separately
Users may type personal data, contracts, secrets, and credentials into free text. Before sending data to a third-party model, telemetry, or human feedback, apply minimisation and verifiable redaction with stop, quarantine, or human handling on failure.
Output detection is not the only defence
Filters catch some formatted identifiers and secrets but cannot prove that a model did not infer or paraphrase sensitive facts. Combine source authorization, least-privilege tools, content policy, extraction tests, and human handoff.
Evaluate protection and utility together
Across languages, document types, rare entities, adversarial prompts, and user roles, test misses, over-redaction, answer quality, evidence fidelity, tool actions, and latency. Regress every change to treatment rules, model, prompt, knowledge, or detector.
Propagate deletion to derivatives
When a source expires or consent is withdrawn, assess datasets, indexes, embeddings, caches, evaluation sets, feedback, and controllable model derivatives. Where exact deletion is unavailable, state that limit, compensating controls, and rebuilding conditions in advance.
How masked data differs from neighbouring concepts
Masking, pseudonymisation, de-identification, anonymisation, and synthetic data can all reduce identification risk but differ in goal, reversibility, and legal meaning. Use masked data as a broad starting label, then state the actual method, attacker assumption, residual risk, and permitted purpose.
| Concept | Boundary with masked data |
|---|---|
| Data masking | A family of techniques that changes displayed or copied values. Masked data is the governed result tied to purpose, treatment, validation, and residual risk; the technique alone proves no threshold. |
| De-identification | Targets reduction of association with identity and requires re-identification-risk assessment. The broader project use of masked data can also cover trade secrets and credentials and is not automatically a legal synonym. |
| Pseudonymisation | Replaces identifiers with pseudonyms while separately held information can restore association. It preserves longitudinal or cross-table relationships but needs mapping separation and continuing controls. |
| Anonymisation | A context- and regime-specific conclusion about non-identifiability, usually a higher bar than controlled internal masking or pseudonymisation. A project team cannot declare it merely from a method name. |
| Tokenisation | Uses tokens and a controlled mapping service to replace originals. It is one reversible protection, and token format, vault, permissions, and linkage behaviour determine risk. |
| Encryption | Uses keys to protect storage or transport and permits exact recovery. It mainly addresses unauthorised reading and does not reduce identifiability after authorised decryption. |
| Redaction | Removes sensitive spans from text, images, or documents. It is a local technique whose contextual clues, metadata, and omissions still require testing. |
| Synthetic data | Is generated by rules or models and need not map record-for-record to a source. It may reduce use of real data but does not automatically provide anonymity or a formal privacy guarantee. |
| Data minimisation | Limits collection, use, and retention at source. It is often more direct than copying and treating excess data; masking is not permission to collect too much. |
| Access control | Determines who may access data under which conditions and complements masking. Weak access cannot be rescued by weak masking, and masking does not cancel least privilege. |
| Test data | Provides inputs, state, and references for a test objective. A masked copy becomes adequate test data only with test traceability, coverage, expected results, and reset procedures. |
Sources and scope
This entry explains masked data in enterprise projects: data produced by applying controlled transformations to a copy, query result, or data flow to reduce the risk that personal, organisational, or commercially sensitive information is improperly identified, read, linked, or misused. 'Masked data' is a practical umbrella term here, not one universal legal or technical state. Masking, redaction, pseudonymisation, de-identification, anonymisation, tokenisation, encryption, generalisation, perturbation, and synthetic data have different purposes, reversibility, and residual risks and are not interchangeable merely because a team calls them sanitisation. This entry does not fully define anonymisation under applicable law, privacy impact assessment, information classification, access control, cryptographic design, data-loss prevention, synthetic-data methods, test data, data governance, or a security programme. Whether a dataset may be used, disclosed, or treated differently under regulation requires a separate assessment of the applicable jurisdiction, sector, contract, recipient, processing environment, and reasonably available external information.
- ISO/IEC 20889:2018: privacy-enhancing de-identification terminology, technique classification, characteristics, and applicability; confirmed current in 2024
- ISO/IEC 27559:2022: framework for identifying and mitigating re-identification risk throughout the lifecycle of de-identified data
- NIST SP 800-188: de-identification goals, direct and quasi-identifiers, sharing models, disclosure review, re-identification studies, and ordinary masking-tool limitations
- NISTIR 8053: de-identification methods and re-identification risk for structured data, free text, multimedia, and medical imagery
- NIST SP 800-122: context- and impact-based identification of PII and selection of confidentiality safeguards
- UK ICO introduction to anonymisation: anonymity, pseudonymisation, and identifiability in the UK GDPR context; the page notes review after legislative change
- UK ICO pseudonymisation guidance: purposes and threat models, hashing/encryption/token techniques, separation of additional information, and ongoing effectiveness
- ENISA pseudonymisation techniques and practices: attacker models, brute-force and dictionary risks, and trade-offs among protection, utility, scale, and recovery
- Microsoft SQL Server dynamic data masking: query-result masking rather than stored-data transformation, to be combined with permissions, audit, encryption, and inference controls
- NIST AI 600-1: generative-AI data-privacy risks including models leaking, generating, or inferring sensitive information and memorising training data
- NIST privacy-preserving federated-learning material: both model updates and trained models can leak or support reconstruction of training data
- NIST differentially private synthetic-data material: ordinary synthesis does not automatically guarantee privacy, and formal guarantees trade off against utility