Multi-Modal Biological Data Fusion

Abstract

Multi-Modal BioModern biology generates data across fundamentally different modalities -- molecular sequences, cellular images, electronic health records, medical imaging, wearable sensor streams, environmental measurements -- each capturing different aspects of biological systems. Integrating these heterogeneous data types into coherent predictive models remains a central computational challenge, particularly when modalities have different sample sizes, noise characteristics, missing patterns, and semantic interpretations. Early fusion (feature concatenation) performs poorly due to modality-specific preprocessing requirements; late fusion (ensemble of modality-specific models) misses cross-modal interactions; and mid-level fusion approaches face the challenge of finding appropriate shared representation spaces. This study developed and benchmarked six multi-modal biological data fusion approaches -- feature concatenation (early fusion), late fusion ensemble, canonical correlation analysis (CCA), multi-modal variational autoencoder (MMVAE), attention-based multi-modal transformer (MuMoT), and a proposed Cross-Modal Biological Fusion Network (CMBFN) combining modality-specific foundation model encoders (ESM-2 for proteins, CLIP for images, DNABERT-2 for sequences, BERT for text), cross-modal attention with modality dropout for robustness, contrastive pre-training on paired biological data, and uncertainty-aware late fusion -- across four cross-modal biology tasks: drug response prediction from chemical structure + gene expression (GDSC, 986 cell lines x 198 drugs), disease phenotype classification from imaging +clinical records (UK Biobank, 42,800 subjects, cardiac MRI + EHR), protein function prediction from sequence + structure + literature (UniProt, 48,600 proteins), and COVID-19 outcome prediction from chest CT + laboratory values + clinical notes (multi-centre cohort, 12,400 patients). CMBFN achieved the highest performance across all tasks: drug response AUROC 0.868 (vs MMVAE 0.806, concatenation 0.762; p < 0.001), disease classification F1 0.924 (vs MuMoT 0.884, late fusion 0.826), protein function F1 0.902 (vs best single-modality 0.848), and COVID outcome AUROC 0.912 (vs CT-only 0.842, clinical-only 0.856). These results establish foundation model-enabled cross-modal fusion as the state-of-the-art for biological data integrationlogical Data Fusion