Federated Optimization Algorithms

Abstract

Federated learning enables collaborative model training across distributed data holders without centralising raw data, but the optimisation landscape of federated settings -- characterised by non-IID data distributions, heterogeneous compute capabilities, intermittent client availability, and communication bottlenecks -- differs fundamentally from centralised training. This study presents a controlled evaluation of six federated optimisation algorithms -- FedAvg, FedProx, SCAFFOLD, FedNova, FedAdam (server-side adaptive), and personalised federated learning (Per-FedAvg) -- across four non-IID partitioning schemes and three tasks: image classification (CIFAR-100 across 100 clients), next-word prediction (Stack Overflow, 342K users), and clinical mortality prediction (eICU, 208 hospitals). A total of 2,280 experiments were conducted varying heterogeneity level, client participation rate, and communication budget. SCAFFOLD achieved the fastest convergence under high non-IID conditions (2.4x fewer communication rounds than FedAvg to reach target accuracy) through variance reduction of client gradient drift. FedAdam achieved the highest final accuracy (68.4 +- 0.6% on CIFAR-100 with Dirichlet alpha=0.1 non-IID vs. 64.2 +- 0.8% for FedAvg) by applying adaptive server-side momentum that stabilises aggregation of heterogeneous client updates. Per-FedAvg improved per-client accuracy by 4.8 +- 1.2 points over FedAvg by learning personalised model initialisations via MAML-style meta-learning. On clinical data with natural non-IID (each hospital has different patient populations), personalised approaches outperformed global models by 6.2 points AUC, confirming that forcing a single global model on heterogeneous clinical sites sacrifices performance. Communication compression (top-k sparsification + quantisation) reduced bandwidth by 16x with only 1.8% accuracy loss. A practical federated optimisation selection guide mapping heterogeneity type, communication budget, and personalisation requirements to recommended algorithms is proposed