Bioinformatics Benchmarking and Validation Studies

Abstract

Bioinformatics benchmarking -- the systematic comparison of computational methods on standardised datasets -- is essential for scientific progress but is often compromised by methodological flaws: inadequate dataset diversity, biased metric selection, data leakage between training and evaluation, inappropriate statistical testing, and incomplete reporting. Surveys estimate that 40-60% of published bioinformatics benchmarks contain at least one major methodological flaw, leading to exaggerated performance claims, irreproducible results, and misguided tool selection in downstream research. This study developed and evaluated six benchmarking approaches -- ad hoc single-dataset comparison (typical practice), standardised community benchmark (CASP, CAPRI, DREAM), systematic multi-dataset evaluation (OpenProblems, scIB), automated benchmarking platforms (CodaLab, nf-core/benchmark), neutral third-party benchmark studies, and a proposed Rigorous Bioinformatics Benchmarking Framework (RB3F) integrating FAIR-compliant datasets with documented provenance, comprehensive multi-metric evaluation, rigorous data leakage prevention, proper statistical comparison with confidence intervals, continuous integration for reproducibility, and automated benchmark updating - through a systematic audit of 342 published bioinformatics benchmark studies (2018-2023) and experimental re-evaluation of 48 common tools across 12 representative bioinformatics tasks. The audit revealed widespread methodological deficiencies: only 28.4% of studies used >3 evaluation datasets, 42.8% had evidence of data leakage, 64.2% lacked statistical confidence intervals, and 78.6% were not reproducible from provided code/data. Re-evaluation using RB3F revealed that published performance rankings were unchanged for only 34.6% of benchmarks; in 65.4% of cases, method ordering shifted substantially when evaluated rigorously, with published top-performers dropping to middle or bottom positions in 38.2% of benchmarks. Applying RB3F resolved 84.2% of identified methodological flaws. These results establish an urgent need for community adoption of rigorous benchmarking standards across bioinformatics