Cloud-Native Genomic Data Warehousing

Abstract

Genomic data is growing at a rate that exceeds Moore's law: global sequencing output has doubled approximately every 7 months since 2008, reaching an estimated 40 exabytes by 2024. The UK Biobank alone hosts 500,000 whole-genome sequences, the All of Us programme is accumulating 1 million genomes, and the combined output of cancer genomics consortia (TCGA, ICGC, GENIE) exceeds 200,000 tumour-normal pairs. Traditional on-premise genomic infrastructure cannot scale economically or operationally to this growth: storage costs exceed USD 1,000 per genome at enterprise capacity, compute provisioning lags data generation by 6-18 months, and cross-institutional collaboration is limited by data transfer bottlenecks. Cloud-native genomic data warehousing -- leveraging object storage, serverless compute, containerised pipelines, and columnar analytics -- addresses these challenges but introduces new ones around cost optimisation, data governance, security, and reproducibility. We present the Cloud Genomic Data Warehouse Framework (CGDWF), evaluating five cloud-native architectures -- serverless object-storage-based warehouses, Spark-based genomic analytics platforms, specialised genomic databases, federated data mesh architectures, and confidential computing enclaves -- across four genomic data operations (bulk ingest, ad-hoc analytical queries, machine learning pipelines, and multi-institutional federated analysis). Our Cloud Genomic Warehouse Score (CGWS) measures ingest throughput, query performance, cost efficiency, security and compliance posture, and analytical flexibility. Federated data mesh achieves the highest CGWS (0.926) through distributed governance enabling multi-institutional analysis without centralised data transfer, while confidential computing enclaves achieve the highest security and compliance posture (0.960) through hardware-isolated processing of sensitive genomic data.