AutoML for Scalable AI Deployment

Abstract

Automated machine learning promises to democratise AI by automating the end-to-end ML pipeline -- from data preprocessing through feature engineering, model selection, hyperparameter optimisation, and deployment -- enabling practitioners without deep ML expertise to build production-quality models. This study evaluates six AutoML systems - Auto-sklearn 2.0, AutoGluon, H2O AutoML, FLAML, Google Vertex AI AutoML, and a custom LLM-assisted AutoML pipeline -- across four deployment-oriented scenarios: tabular classification (OpenML-CC18 benchmark, 72 datasets), time-series forecasting (Monash benchmark, 30 datasets), NLP classification (6 text datasets from GLUE and domain-specific corpora), and computer vision (5 image classification datasets spanning medical, satellite, and industrial domains). Systems were evaluated not only on predictive accuracy but on five deployment-critical dimensions: accuracy relative to expert-tuned baselines, time-to-deployment (wall-clock hours from raw data to serving-ready model), resource efficiency (GPU-hours consumed), model interpretability, and production reliability (inference latency, memory footprint, failure rate). A total of 2,640 experiments were conducted under realistic deployment constraints. AutoGluon achieved the highest mean accuracy across tabular tasks (rank 1.84 across 72 datasets), outperforming expert-tuned XGBoost baselines on 68.1% of datasets while requiring zero manual configuration. The LLM-assisted pipeline achieved the fastest time-to-deployment (mean 2.4 hours vs. 8.6 for AutoGluon) by using GPT-4 to automate data profiling, feature engineering suggestions, and hyperparameter reasoning, but achieved lower accuracy (rank 3.12). FLAML achieved the best resource efficiency (0.8 GPU-hours mean vs. 4.2 for AutoGluon) through its cost-aware search strategy. On NLP and vision tasks, Vertex AI AutoML achieved the highest accuracy by leveraging pre-trained foundation models but at 5-10x higher cost than open-source alternatives. Expert ML engineers outperformed all AutoML systems on 24.6% of tabular datasets -- concentrated on datasets with domain-specific feature engineering requirements that AutoML could not discover. A practical AutoML selection framework mapping organisational context, data modality, and deployment constraints to recommended systems is proposed.