Vector Index Performance across Synthetic Distributions for Disease Surveillance and Global Health Retrieval

Abstract

Dukhik Hovsepyan

Vector similarity search is increasingly relevant to information retrieval systems used in disease surveillance, biomedical knowledge management, and global-health information retrieval, where efficient access to semantically related records can support the analysis of large collections of multidimensional representations. However, selecting an appropriate vector index remains dependent on workload characteristics, including recall, query cost, construction cost, memory requirements, and vector distribution. This paper presents a reproducible, controlled comparison of four indexing approaches: exhaustive flat search, inverted-file search with exact reranking (IVF-Flat), random-hyperplane locality- sensitive hashing (LSH), and product quantization (PQ). Synthetic vector corpora are generated from isotropic Gaussian, clustered Gaussian-mixture, and anisotropic Gaussian distributions at three corpus sizes using a fixed random seed. For each size, 72 queries are evaluated against exhaustive squared-Euclidean top-10 ground truth. The experiments vary search breadth or representation size and measure Recall@10, query latency, candidate work, build time, and estimated index payload. At the largest corpus size of 1,800 vectors with 32 dimensions, IVF-Flat achieves exact recall on clustered data while evaluating approximately 596 candidates per query, whereas the best tested LSH configuration also achieves exact recall while evaluating approximately 347 candidates. On isotropic data, the tested LSH configuration reaches 0.742 Recall@10, while PQ remains between 0.14 and 0.18 across the evaluated distributions despite substantially reducing the estimated code payload. Exhaustive flat search is the fastest measured query path in this small-scale Python/ NumPy implementation, demonstrating that approximate indexing does not necessarily provide a latency advantage for small collections and dimensionalities. The results are descriptive rather than inferential and should not be interpreted as production-scale rankings. Nevertheless, the study provides transparent evidence for distribution-conditioned index selection and establishes an experimental basis for future evaluation on larger biomedical and global-health datasets, including disease-surveillance records, health-related documents, and real embedding workloads.

PDF