Enterprise AI Project Wiki · Data, knowledge, and models
Vector retrieval
Also calledVector search · Dense retrieval · Similarity search · Embedding retrieval · Semantic vector search · ANN retrieval · 向量检索
Vector retrieval converts a query into a vector compatible with indexed objects, performs exact or approximate nearest-neighbour search under a defined distance or similarity measure within an authorised collection, version, access scope, and time scope, and returns candidates with their original object identity, provenance, metadata, and retrieval record. A governed version fixes at least the embedding model and task mode, input processing, vector dimension and numeric type, normalisation, metric, index algorithm and parameters, filtering method, candidate count, and source-content version.
Vector retrieval finds neighbours in a representation space; it does not decide facts or answers
Vector retrieval is good at finding content whose wording differs but meaning is close. It measures distance between representations, however; it does not know whether a policy is current, a number is correct, or similarity alone permits an answer.
A query such as “what evidence is needed for travel reimbursement?” and a policy sentence such as “expense claims require an itinerary and valid receipts” share few words but can carry similar meaning. An embedding model maps the query and content to numbers so that objects related under its training objective tend to lie near one another. Vector retrieval finds those neighbours and returns the documents, images, products, or records behind them.
Nearness only exists inside a defined representation contract. Changing the model, task type, truncation, normalisation, or metric can change coordinates and ordering for the same text. If an old document vector and a new query vector do not occupy a compatible space, the service can still compute a number, but that number lacks a defensible relevance meaning.
Nearest neighbours are candidates, not verified facts. Incorrect, expired, inaccessible, or merely on-topic content can be close, while identifiers, amounts, negation, exceptions, and exact names can be weak in a purely semantic space. Enterprise retrieval still needs access and effective-date filters, authoritative sources, lexical signals, reranking, evidence-sufficiency checks, and business stop rules.
Every vector must resolve to its original object and generation contract
A vector is only a sequence of numbers and is difficult to explain without its source and generation method. Production needs each vector to resolve to exact content, version, field, model, and processing step for updates, deletion, and replay.
Object and vector identity
Give the source, document, chunk, image, or record a stable object ID and each vector its own ID. One object can have separate body, title, image, language, or task vectors; array position alone is not an identity relationship.
Original content location
Keep source ID, version, page/section or record key, heading path, character or element range, checksum, and withdrawal state. Results must return to readable, citable content; the vector is not the textual evidence.
Model and task mode
Record provider, exact model ID/version, query/document or other task type, input prefix, language, and modality. A dual encoder can use different query and passage instructions, but the two sides must follow one compatible model contract.
Input processing
Record parsing, cleaning, heading enrichment, summary or contextual text, case and Unicode handling, truncation, tokenizer, input limit, and empty-value handling. Silent API truncation can make stored text differ from what was actually embedded.
Vector format
Record dense, sparse, or binary type, dimension, float32/float16 or quantised representation, normalisation, and zero/invalid-vector rules. The index field dimension must match model output.
Retrieval configuration
Fix metric, exact/ANN mode, index type and parameters, shards and replicas, filter timing, candidate count, threshold, oversampling, full-precision rescoring, and reranking.
Access and temporal metadata
Store tenant, organisation, role, object access, classification, region, content status, effective and expiry times, and deletion marker beside the vector, with the same update accountability as the source.
Processing and availability times
Record when embedding, write, index visibility, deletion propagation, and configuration publication occurred. Distinguish approved content, generated vector, completed index, and authorised retrieval states.
Confirm that queries and candidates genuinely occupy a compatible embedding space
Vectors with the same dimension can still come from different models, task prefixes, or normalisation rules. Comparing incompatible representations still produces results, but their distances have no dependable meaning.
Same model version
Document vectors from model A normally cannot be compared with query vectors from model B. Equal dimensions do not imply equal coordinate meaning. A model upgrade usually needs a new index and regression evidence.
Paired task roles
Some models separately optimise RETRIEVAL_QUERY and RETRIEVAL_DOCUMENT or require query:/passage: prefixes. Following one model contract does not require identical strings on both sides.
Modality and cross-modality
Text-to-text, image-to-image, or text-to-image retrieval works only when the model explicitly creates a shared space. Vectors from unrelated modality models do not become comparable by sharing a field.
Language coverage
A multilingual model may align languages yet degrade on low-resource language, domain abbreviations, or mixed scripts. Test actual language pairs and business terms; “multilingual support” does not prove equal quality.
Dimension and truncation
Reduce dimensions or use truncatable embeddings only as the model permits and with required normalisation. Lower dimension changes storage, speed, and quality and requires its own version and evaluation; it is not lossless.
Distribution change
Changes in content type, length, template, language, entities, or query distribution alter neighbourhoods. Adding many near-duplicate templates can crowd top results without any model change.
Scores do not cross spaces
A score of 0.8 under another model, field, collection, or normalisation does not carry the same meaning. Combine channels with validated rank fusion, calibration, or reranking rather than raw-score comparison.
Cosine, dot product, and Euclidean distance must match the model and index
The distance metric is not a cosmetic option. Model training, vector normalisation, and index ordering must agree; otherwise candidate order changes and an established threshold no longer means what it did.
| Metric | Meaning and boundary |
|---|---|
| Cosine similarity | Compares vector angle and reduces magnitude effects. Its mathematical range is commonly -1 to 1, but a product can transform cosine distance into a different search score, so confirm returned-score semantics before setting thresholds. |
| Dot/inner product | Depends on direction and magnitude. It gives cosine-equivalent ordering for unit-normalised vectors; without normalisation, magnitude can be an intended model signal or an unwanted bias. |
| Euclidean/L2 distance | Measures the length of the coordinate difference, so smaller is nearer. It can give cosine-equivalent ordering for unit vectors, but raw values and direction remain different. |
| Other metrics | Binary vectors can use Hamming or Jaccard, and some systems support L1. A metric must match the data type, training objective, and index operator; product support is not a reason to switch arbitrarily. |
Separate distance from similarity
Similarity commonly increases with nearness; distance commonly decreases. A database can return negative inner product for ordering, while a search service can transform distance monotonically. Record API field and display semantics.
Do not call a score probability
A value of 0.82 is not “82% correct” or “82% relevant.” It is proximity in one space, and its distribution changes with query, collection size, duplicates, and model.
Derive thresholds from error cost
Use labelled positives, hard negatives, no-answer and unauthorised examples to inspect score distributions, then balance false negatives and false positives for the task. Different query or collection classes can require different rules.
Neighbour gaps also matter
A first result tied closely with later candidates can indicate ambiguity, duplicates, or multiple readings. A high absolute score still cannot decide when valid sources conflict.
Exact KNN and ANN differ on whether the true K nearest vectors are guaranteed
At scale, comparing every vector is often too slow, so ANN trades some recall for speed and capacity. Measure that trade-off against exact KNN rather than assuming that lower latency preserved every true neighbour.
| Method | Use and cost |
|---|---|
| Exact KNN / exhaustive search | Computes distance to every vector in the allowed collection and returns the true K nearest. It supports small collections, precision-sensitive work, and ANN ground truth, but computation and bandwidth become costly at high scale, dimension, or query volume. |
| Approximate nearest neighbour / ANN | Uses graphs, partitions, quantisation, or other structures to inspect only part of the space, trading some neighbour loss for lower latency, higher throughput, or less storage. Approximation depends on data, filters, parameters, hardware, and update state. |
ANN recall can mean how many exact-KNN neighbours the index reproduced; business retrieval Recall means how many relevant pieces of evidence entered the result. An ANN index can almost perfectly reproduce nearest neighbours in the embedding space and still miss business evidence because the embedding, chunking, or source coverage is wrong.
Vector count alone does not select exact or ANN. Consider dimension, concurrency, filter selectivity, update rate, K, tail latency, memory, GPU/SSD availability, tolerated misses, and rebuild window. A small heavily filtered subset can favour exact search; a large high-throughput collection usually needs ANN or a tiered design.
Index structures make different trade-offs among recall, latency, memory, build, and updates
The index decides how search is narrowed and also affects memory, incremental updates, and rebuild cost. Selection should follow data scale, filter patterns, and freshness needs—not a single-query speed comparison.
Flat / brute force
Stores original vectors and compares each one without an approximate index. It is simple, update-friendly, and exact, useful for baselines, small collections, or subsets after selective filters, and can be accelerated with batching, SIMD, or GPUs.
HNSW
Builds a multilayer proximity graph and navigates from sparse upper layers to the dense base. m, efConstruction, and query breadth affect graph quality, build time, memory, writes, and recall; deletion-heavy workloads also need tombstone and graph-degradation tests.
IVF / partition indexes
Clusters vectors into lists and probes only nearby lists. List count, training sample, and probes control speed and loss; distribution drift or too little data at index build can reduce quality.
Product Quantization
Splits vectors into low-dimensional subspaces and encodes short quantised codes for lower memory and bandwidth, often with IVF. Compression is lossy; oversample and rescore against original full-precision vectors to determine whether loss is acceptable.
Scalar or binary quantisation
Converts float32 to lower-precision values or bits to reduce memory and disk. Savings depend on whether original vectors remain, and quality depends on model and data; compression ratio is not retrieval evidence.
Disk and tiered indexes
Keep navigation or compressed structures in memory and more vectors on SSD. Capacity improves, but I/O, cache warm-up, and tail latency appear; test real cold and warm states.
Multiple fields and indexes
Title, body, image, or language can have separate vector fields queried in parallel. Their scores are not naturally comparable, so define candidate budgets, fusion, deduplication, and final reranking.
A product name is not an algorithm contract
Implementations called HNSW or IVF differ in sharding, filtering, updates, quantisation, defaults, and hardware. Acceptance must fix product/API version, index definition, and workload, not merely the algorithm name.
Metadata filters define business scope and also change vector recall
Organisation, language, classification, and validity commonly restrict the eligible set first. Highly selective filters leave fewer searchable items and can starve some ANN strategies, so access correctness and recall must be tested together.
Filter fields
Store tenant, subject permission, organisation, region, language, type, state, effective/expiry time, product, and source class as filterable metadata. Embeddings must not encode deterministic constraints on their own.
Prefiltering
Searches only eligible objects during candidate traversal and can ensure the result belongs to the allowed subset. Highly selective filters can expand graph traversal and increase CPU and latency; verify the product's implementation.
Postfiltering
Takes unfiltered top-K and removes ineligible items. Latency is more predictable, but fewer than K can remain and eligible objects outside the original top-K become false negatives. More candidates mitigate but do not create a security guarantee.
Partition or separate index
Stable tenant, region, or broad category partitions reduce search scope and isolate effects, but add indexes, routing, capacity fragmentation, and rebuild responsibility.
Evaluate filter and ANN together
Use common, rare, intersecting, empty, and changed-access conditions to measure result count, ANN recall, business relevance, P95/P99 latency, and resources. Unfiltered ideal queries are insufficient.
Security filters must be non-bypassable
Identity and object authorisation are hard controls enforced on data access and result return. Oversampling, reranking, caches, parent expansion, and debug logs must retain the same scope.
An auditable vector query passes through scope, representation, candidates, and source resolution
Between a user's words and final evidence sit query processing, embedding, filtering, neighbour search, and source resolution. Preserving each input and output explains why an item appeared and reveals the layer where an error began.
Confirm the information need
Keep the original query and identify object, time, language, identity, exact identifiers, negation, and expected evidence. Route identifier and strict-field questions to exact, structured, or lexical methods rather than forcing a vector.
Select collection and field
Choose the knowledge base, index version, and vector field by purpose, then apply tenant, access, effective-date, and deletion scope. Searching the wrong collection is more fundamental than a distance setting.
Generate the query vector
Use fixed model, task type, prefix, language, and input limits; validate dimension, normalisation, empty value, truncation, and service version. Keep a query hash, response ID, and time without logging sensitive raw text unnecessarily.
Run nearest-neighbour search
Use the matching metric and exact/ANN index, recording filters, K, search breadth, probes, shard, timeout, and fallback. A timeout-driven algorithm or collection switch must be visible.
Expand and recompute candidates
A quantised or subvector index can oversample and recompute distances with original full-precision vectors. This corrects approximation loss, not the embedding model's business error.
Fuse and rerank
Add lexical, rule, or other vector fields where required, combine with rank fusion or validated calibration, then apply a cross-encoder, business rule, or human review. Retain ranks at every stage.
Resolve original objects
Deduplicate, expand parents, neighbours, or structured records, recheck access and validity, and return source, version, location, matched unit, and limitations. Internal arrays and graph nodes are not user evidence.
Judge sufficiency and stop
Check required facts, conditions, exceptions, and sources. When nearby content does not support the conclusion, sources conflict, evidence is insufficient, access is denied, or the chain fails, limit the answer and external action.
Top-K and similarity thresholds control candidate volume; they do not define answerability
High similarity means representations are close, not that facts are complete; a result above threshold may cover only half the question. Candidate volume and answer conditions are separate controls, with the latter also checking source, time, conflicts, and coverage.
Two meanings of K
The index can search or oversample a large candidate set while only a smaller final top-K is delivered. These separately affect ANN recall, reranking cost, context noise, and latency and need separate records.
Threshold and rank
Top-K guarantees count but can include irrelevant items; a threshold imposes minimum proximity but can return fewer than K. Combine when appropriate and define an explicit no-candidate state.
Deduplication and diversity
Overlapping chunks or template copies can occupy every top slot. Deduplicate by source, parent, or similarity cluster and control source/topic coverage for multi-evidence work.
Parent-child expansion
A small matching chunk can expand to a parent clause or neighbour, but expanded content must retain source, version, and access and be checked for irrelevant or unauthorised additions.
Limits on comparability
Scores from different fields, models, collections, metrics, quantisation, or product transformations cannot share one raw threshold. Prefer ranks or labelled calibration for multi-route fusion.
Business stop rules remain separate
Even above a vector threshold, amounts, authority, compliance, identity, and external operations need structured facts, rules, or human confirmation. Evidence and risk—not one cosine value—determine answerability.
Evaluate representation quality, ANN approximation, business relevance, and final task separately
A wrong final answer does not reveal whether the embedding missed meaning, ANN missed a neighbour, or reranking and generation failed. Layered evaluation gives each change a matching measure and keeps one end-to-end score from hiding offsetting errors.
Evaluation snapshot
Fix source content, chunks, metadata, access, embedding model/task, dimension, normalisation, metric, index, and parameters. A changed model or corpus is a different experiment.
ANN recall
Use exact-KNN top-K under the same snapshot and metric as truth, and measure how many ANN results reproduce it. Report K, filters, parameters, hardware, concurrency, and data scale.
Business relevance
Authorised domain reviewers label candidates relevant, partially relevant, irrelevant, expired, unauthorised, or insufficient. Use Precision@K, Recall@K, MRR, nDCG, and evidence coverage; exact neighbours can still be irrelevant.
Difficult and negative cases
Include paraphrases, languages, domain terms, typos, identifiers, negation, numbers, time, exceptions, close-but-wrong topics, duplicate templates, no answer, unauthorised, and malicious queries.
Filtering and isolation
Measure recall, result count, and access violations under common and selective tenant, role, region, state, and time filters. Access leakage has a zero target and cannot be averaged away by relevance.
End-to-end task
Check correct-source citation, answer completeness and faithfulness, insufficient-evidence stopping, human handoff, and business task completion; attribute errors to embedding, ANN, filter, rerank, context, or generation.
External benchmark boundary
ANN-Benchmarks and billion-scale challenges help compare recall–QPS trade-offs, but their vectors, hardware, and query distributions differ from the project and cannot become production promises.
Vector-retrieval performance joins recall, latency, throughput, memory, build, and freshness
Optimising only for milliseconds can sacrifice recall or leave updates queued; maximising recall can exceed cost and concurrency limits. Test quality, resources, and data freshness together under expected load.
Query latency
Measure P50/P95/P99 for query embedding, routing, filters, ANN, rescoring, fusion, reranking, source fetch, and network. Separate cold start, cache hit, single-query, and batch behaviour.
Throughput and concurrency
Report successful QPS, queues, timeouts, and degradation under the target query mix. High batch throughput is not interactive latency. Test writes and rebuilds alongside queries.
Quality–speed curve
Vary efSearch, probes, visited nodes, candidate count, and oversampling and plot ANN recall against latency or QPS. Parameters are valid only for the stated data, K, and filter distribution.
Memory and storage
Count original and compressed vectors, graph edges, centroids, metadata, object copies, WAL/logs, and double space during rebuild. Dimension × float bytes is only a lower bound.
Index construction
Measure embedding generation, load, training, graph build, compression, shard replication, validation, and failure recovery. A model migration can require old and new indexes to coexist.
Updates and freshness
Set separate SLIs from source change to searchable vector and from delete/access change to invisibility across all replicas. High writes, tombstones, or drift can change index quality and resources.
Cost per successful task
Combine embedding calls, vector storage, CPU/GPU/memory, query, reranking, network, and operations with task success and quality. A price per million vectors is not total cost.
Vectors are not anonymised data; production must protect content, queries, indexes, and versions
Turning text into numbers does not remove semantic, membership, or attribute leakage, and query logs expose user intent. Treat vectors, metadata, source mappings, credentials, and backups as real data assets.
Classify like source content
Embeddings retain useful semantics, and research has reconstructed substantial text under particular conditions. Do not treat a vector as masking; inherit source classification, region, retention, and access controls.
Protect queries
Queries can reveal health, customer, transaction, or investigation intent. Minimise logs, separate debugging and audit purposes, and constrain embedding, search, and analytics services' access to text and vectors.
Tenant and object authorisation
Resolve identity server-side and generate filters from trusted policy. Indexes, caches, parent content, backups, and exports retain isolation. Shared ANN graphs can also let one tenant's distribution affect another's recall and speed; assess partitions or separate indexes.
Content integrity
Malicious or wrong documents remain retrievable after embedding. Keep source checksums and approval state, treat content as untrusted, and never let retrieved instructions alter system rules or invoke tools.
Secrets and credentials
Do not index passwords, tokens, private keys, or unnecessary sensitive fields. Rotate least-privilege embedding, storage, and search credentials and never expose administrative keys to clients.
Deletion and withdrawal
Delete source chunks, vectors, graph nodes, quantised codes, caches, backups, and exports. Record propagation deadlines and run negative queries; a successful primary-table delete is not full-chain erasure.
Versions and change
Monitor content volume, vector norms, languages, duplicates, filter selectivity, ANN recall, no-result rate, latency, and access denial. Model, input process, index, parameter, or product changes require a regressed, rollback-capable combination.
Rebuild and retirement
Retain approved sources, processing recipe, model licence/version, index definition, parameters, evaluation set, and release record so indexes can be rebuilt. On retirement, stop traffic, revoke connections, dispose of provider copies, and hand over known limits.
Vector retrieval example for a procurement knowledge base
This constructed example explains the retrieval contract and validation. It is not a client knowledge base, real policy, selected model, threshold, parameter, performance result, or architecture.
Task scope
Employees use natural language to find current procurement policy, procedures, and approved FAQs. Supplier IDs, the requester's live application state, and approval authority come from authorised structured systems, not vector proximity.
Vector objects
Policies use clause parent-child chunks, procedures use complete steps, and FAQs use approved Q&A units for document vectors, each retaining source, version, location, validity, organisation, role, language, and processing version.
Query processing
“What attachment is needed to buy equipment on an out-of-town trip?” retains the original text, resolves region, purchase type, and attachment intent, and uses the query task type; “out-of-town” and “equipment” also remain lexical signals.
Scope filter
Limit to the user's organisation, visible classification, current content, and English collection before ANN. Compare highly selective access searches with exact filtered KNN to detect post-filter starvation.
Candidates and reranking
Take V vector and L lexical candidates, use rank fusion and deduplication, rerank query-to-clause relevance, and expand definitions, prerequisites, and exceptions from the parent. V, L, and final K remain experiment variables.
Stop conditions
If only obsolete training content matches, policy conflicts with FAQ, an attachment table is missing, the source expired, access crosses an organisation, or evidence covers only part of the question, state the limit and stop a definitive answer or submission action.
Evaluation
Use colloquial, technical, misspelled, exact policy-number, negation, historical, exception, no-answer, unauthorised, and close-but-wrong queries to measure embedding relevance, ANN recall, filters, rerank, citation, answer, latency, and cost.
Handover
Transfer source/chunk/vector catalogues, model/task, input process, dimension/metric, index/filter, K/threshold, exact baseline, relevance labels, performance, security, rebuild, deletion, and known coverage gaps.
Enterprise AI and RAG use vector retrieval for candidates, not similarity as evidence
Vector search is most useful for widening the candidate pool so different phrasings can meet. Generation must still return to authorised source text for conditions and exceptions: a distance score helps ranking but cannot carry factual accountability.
Route before embedding
Definitions and paraphrases suit vector channels; identifiers, amounts, current state, calculations, and authority suit exact, lexical, structured, or live tools. An agent selects from approved tools rather than pushing every question into one vector store.
Retain a hybrid baseline
Names, abbreviations, clause numbers, negation, and rare entities often need lexical signals. Compare vector, text, sparse, and rule combinations on the same queries and labels; vector-only is not an automatic advanced default.
Diagnose by layer
First check source presence, then parsing/chunking, embedding exact KNN, ANN approximation, filters, fusion/rerank, context, and model generation. A wrong answer is not automatically a model or vector-store defect.
Context is not raw top-K
Deduplicate candidates, expand parents, add conditions and exceptions, select by question, and retain precise citation. Concatenating K chunks adds repetition, noise, unauthorised content, and position bias.
A close match is a reason to verify, not to conclude
A passage can be highly similar while omitting the applicable region, effective date, or decisive exception. Before generation, verify facts, conditions, time, and source against the original. Partial coverage, conflict, or expiry requires an explicit limit and a stop for high-risk conclusions or tool actions.
Prompt injection remains
Embedding does not remove hostile document instructions. Isolate retrieved content from system/tool instructions, and constrain tool parameters with schemas, access, deterministic business rules, and human confirmation.
Preserve the complete combination
Replay requires original query, rewrite, identity, collection, source/chunk, embedding, vector field, metric, index/filter, candidates, fusion/rerank, context, prompt, model, and tool versions.
Continuously evaluate change
Changes in content, chunking, embeddings, indexes, access, reranker, model, or prompt can alter answers. Regress relevance, ANN recall, citation, refusal, security, latency, and task results with independent test data.
Vector retrieval compared with adjacent concepts
An embedding creates a representation, a vector database stores and searches neighbours, and RAG adds context and generation. Separating the terms makes it possible to locate a defect in the model, index, or answer rather than treating them as one system.
| Concept | Boundary from vector retrieval |
|---|---|
| Embedding | The model and process that converts an object into a vector. Vector retrieval additionally needs collection, metric, index, filters, candidates, and source resolution. |
| Vector database | A product category for storing, indexing, querying, and operating vectors and metadata. Search engines, database extensions, or specialist libraries can also perform vector retrieval. |
| Semantic search | A user capability that searches by meaning and can use vectors, sparse representations, knowledge graphs, query rewriting, or rerankers. It need not equal vector retrieval. |
| KNN | The problem or exact method returning the K nearest objects to a query vector. Vector retrieval also defines object contract, access, result, and business use. |
| ANN | A class that explores part of the space and trades neighbour recall for speed or resources. Not every vector retrieval must be approximate. |
| HNSW | One multilayer proximity-graph ANN index. It does not define embeddings, business relevance, access, fusion, reranking, or RAG. |
| Vector index | The data structure and configuration enabling similarity queries. Source content, vectors, and index are separately managed; the index should normally be rebuildable. |
| Hybrid retrieval | Combines vector, lexical, sparse, structured, or other channels and fuses results. Vector retrieval is one candidate channel. |
| Reranking | Applies a more precise model or rule to a smaller candidate set after ANN. It cannot recover content missing from candidate generation. |
| Similarity threshold | A candidate filter inside one fixed space and task, not a correctness, confidence, authorisation, or evidence-sufficiency threshold. |
| RAG | A complete pattern combining retrieval, context assembly, and a generative model. Vector retrieval is common but neither necessary nor sufficient for RAG. |
Sources and scope
This entry explains vector retrieval in enterprise search, knowledge-base, and AI RAG projects: representing a query and candidate objects as numeric vectors in a shared space and finding nearby candidates under a specified distance or similarity measure. It is not synonymous with an embedding model, vector database, semantic search, approximate nearest neighbour search, HNSW, hybrid retrieval, reranking, document chunking, knowledge base, or a complete RAG system. Those are respectively a representation method, product category, user capability, algorithm class, index structure, combination strategy, later stage, content process, governed collection, or complete application. A nearby vector only establishes proximity under a particular model, input process, metric, and index version. It does not prove that content is true, authoritative, authorised, current, sufficient, or fit to decide. No similarity threshold, dimension, Top-K, index parameter, or “best vector database” works universally across models, collections, languages, and tasks. This entry does not fully define embedding training, vector-database architecture, lexical or hybrid retrieval, reranking models, CPU/GPU kernel optimisation, database tuning, security programmes, or product configuration. An authorised project role must validate the actual design on approved data and representative tasks.
- Google Cloud text embeddings: dense vectors, semantic matching, normalised vectors, task types, input-token limits, and controllable automatic truncation
- Microsoft Azure AI Search vector relevance: exact KNN, HNSW ANN, shared embedding spaces, cosine/dot-product/Euclidean metrics, and score transformation
- Microsoft Azure AI Search vector filters: prefilter, postfilter, sharded top-K, and the effect of selectivity on false negatives, latency, and throughput
- Microsoft Azure AI Search quantisation: scalar/binary compression, oversampling, original vectors, and rescoring to mitigate lossy compression
- Amazon OpenSearch Service vector search: embedding similarity, exact and approximate k-NN, HNSW, and large-scale semantic retrieval
- Official pgvector documentation: exact neighbours, HNSW, IVFFlat, distance operators, filters, iterative scans, multitenancy, quantisation, and full-vector reranking
- Malkov and Yashunin's original HNSW paper: multilayer proximity graphs, hierarchical navigation, candidate connections, and high-recall ANN
- Jégou, Douze, and Schmid's original Product Quantization paper: subspace quantisation, short codes, approximate distance, inverted files, and billion-scale vectors
- Johnson, Douze, and Jégou's original Faiss/GPU paper: exact, approximate, and product-quantised similarity search with GPU k-selection
- Aumüller, Bernhardsson, and Faithfull's ANN-Benchmarks paper: common data, parameter sweeps, and quality–performance evaluation
- NeurIPS 2021 billion-scale ANN challenge results: recall under query-throughput constraints across billion-vector, filtered, and streaming tracks
- Thakur et al.'s BEIR paper: zero-shot generalisation of sparse, dense, late-interaction, and reranking methods across domains, queries, corpora, and tasks
- Morris, Kuleshov, Shmatikov, and Rush's text-embedding inversion paper: embeddings can expose and reconstruct source text and personal information and are not anonymisation