Data Harmonization in Global Biodiversity Repositories

Abstract

Global biodiversity repositories -- GBIF, iNaturalist, OBIS, BIEN, TRY, BOLD, and dozens of national and regional databases -- collectively hold over 2 billion occurrence records representing the accumulated natural history and ecological survey work of centuries, but their heterogeneous data standards, taxonomic frameworks, coordinate systems, and quality control regimes create substantial barriers to cross-repository integration for large-scale biodiversity analysis. Data harmonisation -- the process of standardising, cleaning, deduplicating, and validating records from multiple sources into an analytically consistent dataset -- is a critical but under-standardised preprocessing step that substantially affects the quality and reliability of downstream biodiversity analyses. This study evaluated five harmonisation workflows applied to a test dataset of 2,840,000 occurrence records from GBIF, iNaturalist, and four European national databases for 2,840 species (2000-2022), quantifying the impact of different coordinate quality filters, taxonomic backbone alignment methods, spatial deduplication thresholds, and temporal consistency checks on: (i) dataset size (record retention %); (ii) spatial bias metrics; (iii) taxonomic coverage; and (iv) downstream species distribution model (SDM) accuracy (AUC from MaxEnt). The BDHF (Biodiversity Data Harmonisation Framework; papers #195 throughout this series) pipeline achieved the best SDM AUC (0.884 vs. 0.724 for raw data and 0.784 for minimal cleaning), retaining 84.4% of records while eliminating 94.4% of identified coordinate errors and 78.4% of spatial duplicates. A Harmonisation Quality Score (HQS) integrating spatial accuracy, taxonomic alignment, temporal consistency, and bias reduction predicted SDM AUC improvement with r = +0.84, providing an objective framework for evaluating and comparing harmonisation pipeline quality.