Optimization Techniques in Large-Scale Machine Learning

Abstract

Training modern machine learning models at scale -- language models with billions of parameters, vision models on hundreds of millions of images, recommendation systems serving billions of users -- demands optimisation algorithms that converge reliably within practical computational budgets. This study presents a controlled empirical comparison of eight optimisation methods -- SGD with momentum, Adam, AdamW, LAMB, Adafactor, SAM (Sharpness-Aware Minimisation), LARS, and Lion -- across five large-scale training scenarios: language model pre-training (GPT-style, 1.3B parameters), image classification (ImageNet-1K with ResNet-152 and ViT-B/16), tabular learning (Criteo click-through-rate prediction, 45M samples), federated learning (CIFAR-100 across 100 simulated clients), and fine-tuning pre-trained models (BERT-base on GLUE). A total of 1,920 training runs were conducted on standardised multi-GPU hardware with fixed per-experiment compute budgets. AdamW achieved the lowest final loss on language model pre-training (validation perplexity = 14.82 +- 0.34) and image classification with ViT (top-1 = 79.4 +- 0.3%). LAMB demonstrated the strongest scaling to large batch sizes, maintaining convergence quality at batch size 32,768 where Adam diverged. SAM improved generalisation by a mean of 1.4 percentage points across vision tasks but at 2x the per-step computational cost. SGD with momentum remained competitive on image classification with CNNs (top-1 = 78.6 +- 0.2% with ResNet-152) at substantially lower memory cost than adaptive methods. A practical optimiser selection framework mapping training scale, model architecture, and resource constraints to recommended configurations is proposed.