Enterprise AI Project Wiki · Data, knowledge and models
Test data
Also calledTest data · Testing data · Test dataset · Test input data · Test sample data
Test data is controlled data prepared to execute one or more tests. It includes data that places the test object in its required precondition, inputs supplied to that object, responses supplied by external dependencies, and reference data used to determine the expected result and pass or fail. It should identify its test objective, source or generation method, applicable version, covered condition, sensitivity, and reset method.
Test data makes test conditions executable and decidable; it is not a handful of convenient examples
A test often needs more than values typed into a form. It may require existing accounts, roles, suppliers, stock, historical states, dates, external responses, and configuration to put the system into a specified condition. Afterwards, expected states, amounts, messages, audit records, or forbidden side effects are needed to judge the result. The data supporting all those conditions and decisions belongs to the test-data set.
NASA software-testing material separates test inputs, prerequisites, expected results, evaluation criteria, and test configuration, and calls for allowable ranges, boundaries, invalid inputs, adverse conditions, stress, and recovery to be tested. Test data must therefore serve a named objective. A realistic-looking database snapshot with no coverage rationale is not a substitute for test design.
The ISO/IEC/IEEE 29119 series separates testing concepts, processes, documentation, and design techniques. Test data is an execution asset within that system, but sufficiency still depends on the test basis, risk, coverage model, expected result, and evidence. More rows do not automatically mean better coverage.
What a manageable test-data asset contains
An input record alone is not complete test data. The team also needs its condition, source, expected result, pre-state restoration, permitted environments, and usage period. Versioning these together makes the result reproducible.
Identity and version
Give the dataset, record family, or generation recipe a stable identifier and record the applicable system, software, model, rule, interface, and schema versions so a same-named file cannot change silently.
Objective and traceability
Link requirements, risks, business rules, scenarios, defects, or changes and state whether the data demonstrates normal handling, rejection, a boundary, recovery, or another property.
Precondition state
Record the actors, objects, balances, permissions, workflow state, time, and dependency state that must exist before execution and how that state is established and checked.
Inputs and stimuli
Include form fields, files, messages, API requests, device signals, clock changes, concurrent order, or operator action while preserving unit, format, encoding, and source semantics.
Dependency and test-double responses
Define success, rejection, timeout, duplication, out-of-order, and partial responses from external systems, simulators, stubs, or models. A simulated payload must still conform to the real interface contract.
Expected result and permitted variation
State expected output, state changes, records, notifications, tool actions, and forbidden side effects. Numeric, probabilistic, or human-judged results need tolerances, scoring, and adjudication rules.
Origin, processing, and sensitivity
Mark hand-built, rule-generated, synthetic, de-identified production-derived, or authorised real data, together with transformations, rights, personal or confidential classification, and access restrictions.
Setup, cleanup, and retention
Define loading order, uniqueness strategy, reset or rollback, post-run deletion, failed-run evidence retention, and disposal so one test cannot contaminate another.
Derive test data from requirements, scenarios, and risks
Do not begin with convenient records and ask what they can test. Work backward from requirements and scenario conditions, boundaries, exceptions, and severe consequences to the identities, objects, states, and input combinations needed. Every data group then has a verification purpose.
Fix the test object and formal version
Identify the system, software, model, interface, rule, configuration, and environment versions first. The same data can have a different expected result under another version.
Confirm the test basis
Use approved requirements, business rules, interface contracts, data definitions, risk controls, changes, and prior defects rather than inferring the only correct behaviour from the current implementation.
Expand normal and off-nominal scenarios
Trace actors, objects, states, triggers, dependencies, results, and compensation through the business flow, including insufficient data, insufficient authority, duplicates, concurrency, timeout, and external failure.
Choose coverage techniques
Use equivalence partitions and boundaries for values and classes, decision tables for rule combinations, state transitions for lifecycles, and scenarios, combinations, and error guessing for sequences and interfaces.
Choose the lowest-risk adequate source
Prefer small hand-built data, reproducible generation, synthetic data, and assessed de-identified samples. Use tightly bounded real data only where a lower-risk source cannot satisfy the test purpose.
Establish the test oracle
Derive expected results from business rules, independent calculations, an authoritative system, approved examples, or authorised experts. Where no single truth exists, approve scoring, review, and disagreement handling before execution.
Review coverage and harm
Have business, testing, data, security, privacy, and domain roles check omitted populations, rare states, false merges, sensitive content, irreversible actions, and risks caused by testing itself.
Baseline a reproducible package
Preserve the data version, generation seed, transformation code, schema, checksum, environment preparation, order, and cleanup, then link them to run records and defect evidence.
Test data covers meaningful classes and risks, not a target row count
Ten thousand similar normal records may still omit a missing currency, access conflict, or effective-date boundary. Stratify coverage by business equivalence class, state, population, and risk. Volume supports evidence within those strata; it is not the goal.
Nominal and representative data
Cover primary users, business objects, journeys, formats, and common combinations under intended conditions, without testing only tidy ideal examples.
Equivalence partitions
Divide valid and invalid values expected to receive the same treatment into non-overlapping classes and choose representatives from each. The partition rationale comes from requirements and implementation risk.
Boundary values
Exercise thresholds and adjacent values: minimum and maximum length, monetary limits, effective dates, capacity limits, and timeouts. Inclusion and exclusion boundaries are frequent defect locations.
Rule combinations
Use decision tables for condition combinations, priority, exceptions, and conflicts. Individually valid fields do not guarantee that their combined decision is correct.
State and time
Cover permitted and forbidden transitions, terminal state, reversal, correction, late events, timezone, validity, reordering, and concurrent update with verifiable before-and-after state.
Authority and actor
Prepare self, delegate, expired delegation, cross-organisation, insufficient permission, and conflicting-duty identities to verify what cannot be read, decided, or executed as well as what can.
Failure and recovery
Create missing, malformed, duplicate, timeout, unavailable dependency, partial success, exhausted resource, and retry conditions, then check safe stopping, idempotency, reconciliation, compensation, and recovery.
Scale and distribution
Performance data must match record count, field width, relationship density, hotspots, tails, arrival rate, and concurrency. Duplicating one row a million times does not recreate production complexity.
Hand-built, generated, synthetic, de-identified, and real data have different limits
Hand-built data controls boundaries, synthetic data scales, and real data exposes distributions and dirt, but every source has blind spots and rights. Explain the choice, transformations, and what it cannot represent. Data is not automatically low-risk because it is no longer the original production table.
| Source | Appropriate use and limitation |
|---|---|
| Hand-built | Suited to small, exact rule, boundary, and failure tests with clear expectations; it can omit real-world combinations and tails. |
| Rule- or program-generated | A fixed seed and constraints can create repeatable combinations, states, and volume; an incorrect generator can systematically manufacture false coverage. |
| Synthetic | Creates records that do not represent production subjects and can reduce direct use of personal data; business validity, statistical fidelity, bias, leakage, and preservation of rare risks still require evaluation. |
| De-identified or pseudonymised copy | Can preserve complex structure for migration or integration testing; removing names or masking fields is not anonymity, and linkage, rare values, or a mapping table can still identify subjects. |
| Controlled real sample | Consider only where the purpose cannot be met with lower-risk data, with minimum scope, authority and approval, isolation, audit, expiry, and disposal. Convenience does not justify copying production. |
| Capture and replay | Can reproduce protocols, sequence, or failures, but sensitive content, credentials, freshness, third-party rights, and side effects must be controlled and replay must not reach live objects. |
Without a trustworthy expected result, test data produces output but not a decision
A returned result is not evidence that the result is correct. Expected outcomes come from approved rules, authoritative material, or qualified judgement and include permitted ranges and genuinely indeterminate cases. Otherwise testing only detects change, not satisfaction.
Exact expected value
Deterministic calculations, states, codes, and messages can have complete expected values with rounding, timezone, ordering, encoding, and null semantics stated.
Constraints and invariants
Where full output is too large, check properties that must always hold: amount conservation, monotonic state, uniqueness, referential integrity, non-disclosure, or no duplicate creation on retry.
Independent reference
Use an approved manual calculation, authoritative system, different implementation, or verified golden sample so expected results do not repeat the same error as the system under test.
Ranges and statistical decisions
Performance, retrieval, and probabilistic output need metrics, sample eligibility, tolerance, uncertainty treatment, segment thresholds, and failure rules approved before results are seen.
Experts and multiple annotators
Where no objective single answer exists, record the guide, qualifications, independent decisions, disagreement, adjudicator, and indeterminate label instead of forcing fabricated ground truth.
Forbidden outcomes
Besides desired output, name stop conditions such as unauthorised reading, automatic approval, live-object modification, prompt or personal-data leakage, silent loss, or duplicate charging.
Identity, relationships, and state must be reconstructible or automated tests interfere with each other
Many failures come from an account, approval task, balance, or cache left by an earlier run rather than the current input. The data package creates complete relationships, isolates parallel execution, and cleans up afterward so each case begins from a known state.
- Use dedicated test identities
Separate accounts, organisations, roles, keys, and certificates from production, with traceable ownership, permissions, and lifecycle instead of one shared administrator.
Test-identity register, owner, permission snapshot, and expiry
- Preserve referential relationships
Load master records, transactions, attachments, state history, and external identifiers according to the data model; directly inserting one business row can bypass real constraints.
Relationship map, loading order, and integrity checks
- Give each test ownership
Parallel runs use a unique namespace, tenant, transaction, or tagged record so one test cannot change another test's objects.
Ownership tags, run identifier, and parallel-conflict check
- Start from a known state
Verify balance, state, time, queue, and dependency response before execution so setup failure, product failure, and assertion error can be separated.
Setup result and prerequisite validation log
- Clean without destroying evidence
Successful data may be rolled back or deleted automatically; failed data retains only the evidence needed for reproduction before authorised disposal.
Cleanup result, residue scan, and evidence reference
A test environment is not exempt from data protection, and masking is not deleting a few fields
After a name is replaced, a rare role, time, address, and relationship can still identify a person. Assess re-identification and misuse, control extraction, access, copying, and expiry, and confirm whether external test tools retain content.
Microsoft testing guidance recommends synthetic data as the default and anonymisation of sensitive information where production data is genuinely required, with persistent test data secured separately from production. Product guidance is not a universal legal decision: the project must still determine lawful purpose and controls for the subject, region, contract, and organisation.
NIST SP 800-188 notes that masking tools built to give developers and testers realistic data can offer only limited privacy protection. Direct identifiers are not the whole problem: quasi-identifiers, rare combinations, free text, images, attachments, place and time, and linkable outside data can re-identify a subject.
Pseudonymised data can remain reversible through mappings or additional information, so mappings, keys, and source copies need separation and protection. Test data may also contain trade secrets, security configuration, real credentials, or executable malicious payloads and therefore needs minimisation, approval, isolation, protection, audit, incident handling, and timed deletion.
Test data must be rebuildable, maintainable, and disposable instead of accumulating forever
When rules, interfaces, or data models change, old data may no longer represent current requirements. Retain generation method, version, and retirement reason, and explain whether metrics remain comparable after updates. Expired data and credentials leave on schedule.
Register
Record owner, purpose, source, sensitivity, licence, applicable version, coverage, and permitted environments. Unregistered data should not support formal acceptance evidence.
Generate and transform
Version scripts, rules, seeds, source version, filtering, and de-identification steps and validate record count, schema, relationships, distribution, and privacy objective.
Package and release
Use an unambiguous version, checksum, and change note and deliver data with test-case mapping, expected results, setup, reset, and environment requirements.
Load and preflight
Establish isolated state in order and verify object version, dependencies, clock, permissions, and integrity. A failed preflight is not a product-test failure.
Execute and retain evidence
Save the exact data version, run identifier, time, output, and deviation. Keep enough failure information to reproduce without retaining complete sensitive datasets indefinitely.
Clean and reset
Undo external effects, clear queues and caches, retire identities and secrets, remove temporary objects, and scan residue so the next run starts from a provable state.
Change and retire
Reassess the set when requirements, schemas, rules, production distributions, defects, or risks change; stop using obsolete versions and archive or securely delete them under retention rules.
Constructed example: a reviewable test-data package for purchase requests
This customer-independent procurement setting combines normal, boundary, insufficient-information, duplicate, and access conditions. Each record links to an expected rule, environment state, and cleanup method rather than arriving as an input spreadsheet alone.
Normal submission
Prepare an authorised requester, valid department, available budget, active supplier, amount and currency, required attachment, and successful dependency response. Expect one request in pending approval, not automatic approval.
Boundary and rule
Around approved limit M1, prepare below, equal, immediately above, and inconsistent unit or currency values, with approval route and prohibited action determined by the applicable rule version.
Insufficient material
Separately omit supplier identity, amount or currency, supporting evidence, or budget source. The expected outcome names the missing item and stops formal submission instead of letting a model invent it.
Identity and authority
Prepare self, delegate, expired delegation, cross-organisation, and conflicting-duty actors and verify visibility, submission and approval authority, and denial audit.
Interface and recovery
Simulate ERP timeout, duplicate request, accepted-but-unknown state, reordered acknowledgement, and partial failure and verify idempotency, enquiry, reconciliation, and compensation without blind recreation.
Data origin
Prefer generated fictional people, suppliers, and attachments. If a migration defect requires an approved de-identified structural sample, register origin, processing, access, expiry, and destruction separately.
Handover package
Deliver dataset version, generator, field and relationship notes, case mapping, expectations, load and reset method, sensitivity, known gaps, and approval so the customer can reproduce the conclusion.
An enterprise AI test set must resemble intended use and remain independent from training and tuning
An AI system can gradually learn a familiar question set during development: the team repeatedly inspects failures and adjusts prompts and retrieval, so scores rise without proving performance on new work. Represent real language and risk while retaining evidence not used to choose the solution.
Separate data purposes
Training data fits parameters, validation data selects a model or threshold, and test data evaluates a frozen choice with relative independence. Retrieval knowledge, runtime context, red-team prompts, and production-monitoring samples are separate objects again.
A question set seen too often is no longer an independent exam
If samples, near duplicates, templates, or source documents enter training, knowledge, prompt development, or manual tuning, the score favours familiar material. Record that exposure and keep data unused for solution selection for consequential releases. A set repeatedly used to change the system should be reclassified honestly as development data.
Represent intended use
Stratify by task, language, user role, document type, material quality, region, time, population, and consequence and disclose differences between operational and sample distributions. Do not hide rare or high-risk failures in one average.
Exercise knowledge and capability limits
Include sufficient evidence, missing evidence, conflicting evidence, out-of-scope knowledge, stale material, inaccessible content, and malicious instructions to test answering, refusal, clarification, and human handoff.
Preserve label rationale
For every expected answer, evidence span, tool action, and risk label, record rule, source version, annotation guide, annotator qualification, disagreement, and adjudication. Permit indeterminate labels rather than fabricating ground truth.
Evaluate the full assembly
Freeze prompt, retrieval, knowledge, rule, tool, filter, and human-process versions as well as the model, and measure answer quality, evidence fidelity, authority, tool side effects, latency, cost, and handoff.
Use blind and held-out data
For important releases, keep some samples unavailable to front-line developers or use a relatively independent evaluator. NIST AITE uses sequestered blind data to reduce train-test contamination; this supports the independence principle, not a requirement to copy one platform.
Update continuously while retaining a baseline
Add cases as production topics, knowledge, and attacks change, while retaining a controlled regression core and comparability across versions. Record every replacement, relabel, or deletion and its effect on historical metrics.
How test data differs from neighbouring concepts
Test data supplies execution conditions, a test case supplies steps, an expected result supplies judgement, and the environment hosts execution. Together they form evidence but do not replace one another. Training, validation, and production-monitoring data may look similar yet require different purpose and independence.
| Concept | Difference from test data |
|---|---|
| Test case | States the objective, preconditions, steps, inputs, and expected result; test data is the concrete asset referenced, loaded, supplied, or used to judge it. |
| Test environment | Provides compute, network, configuration, dependency, and permission conditions; test data creates business state inside it, but neither replaces the other. |
| Acceptance sample | A class of test data selected and confirmed by both parties for formal acceptance and tied to the acceptance object and pass criteria; development regression data is not automatically an acceptance sample. |
| Production data | Supports real subjects and activities with real authority, rights, and lifecycle. Copying it to a lower environment does not make those risks disappear under the test-data label. |
| Synthetic data | Describes an artificial origin and may support testing, training, or analytics. Fitness as test data depends on the objective, fidelity, risk, and expected result. |
| Masked or de-identified data | Describes treatment of identifiable or sensitive content, not its use. Testing still requires traceability, coverage, access, re-identification assessment, and lifecycle controls. |
| Training, validation, and test sets | Partition machine-learning data by model-development and evaluation purpose; enterprise system test data also exercises rules, interfaces, authorization, workflows, tool actions, and human processes. |
| Test result | Records actual output, status, variance, and evidence. Test data supplies execution conditions and references; data created during a run needs its own identity. |
| Test oracle | Determines expected or correct behaviour through golden data, rules, invariants, or expert judgement; it is not merely the input dataset. |
| Mock, stub, or simulator | A component replacing dependency behaviour. Its responses are test data, while the component logic, protocol, and runtime remain separately governed. |
Sources and scope
This entry explains test data in software, systems, and enterprise AI projects: data selected, constructed, transformed, or generated to execute a test under specified conditions, establish required state, supply inputs, and judge actual results. Test data is not the test case, test script, test environment, a production-data copy, an acceptance sample, training data, a validation set, an AI evaluation report, a test result, or a defect record. It does not fully define test planning, test techniques, data masking, synthetic data, data quality, privacy or security policy, performance testing, AI red teaming, or sector compliance. Authorised project roles must approve which data may be tested, who provides and confirms it, whether production-derived data is permitted, how it is de-identified, which risks it covers, how expected results are judged, and how long it is retained.
- ISO/IEC/IEEE 29119-1:2022: general software-testing concepts and common terminology for the 29119 series
- ISO/IEC/IEEE 29119-3:2021: software-test documentation templates for different organisations, projects, activities, and lifecycle models
- ISO/IEC/IEEE 29119-4:2021: test-design techniques used during test design and implementation
- ISO/IEC TS 42119-2:2025: risk-based application of the ISO/IEC/IEEE 29119 series to AI-system testing
- ISTQB CTFL 4.0.1: risk-based testing and equivalence partitioning, boundary-value, decision-table, and state-transition techniques
- NASA SWE-066: requirements traceability, approved procedures, annotated test sets covering boundaries, failures, and stress, and complete execution evidence
- Microsoft Azure Well-Architected Framework: realistic isolated data, per-scenario ownership, automated setup and cleanup, synthetic-data preference, and secret protection
- NIST SP 800-188: de-identification purpose, disclosure risk, quasi-identifiers, synthetic data, and the limits of ordinary masking tools
- UK ICO pseudonymisation guidance: identification risk, separation of additional information and keys, technique selection, and ongoing effectiveness testing
- NIST AI RMF Core: documentation of test sets, metrics, and tools; representativeness, suitability, rights, deployment-like conditions, and monitoring
- NIST AI RMF trustworthiness characteristics: accuracy tied to clearly defined, realistic, representative test sets and disaggregated measurement
- NIST AI Technology Evaluation: sequestered testbeds, blind data, common metrics, and scoring to reduce train-test contamination