Reinforcement Learning in Autonomous Systems

Abstract

Reinforcement learning offers a principled framework for training autonomous agents to make sequential decisions in complex, uncertain environments, yet bridging the gap between simulated training and reliable real-world deployment remains a central challenge. This study presents a systematic empirical comparison of five reinforcement learning algorithms -- Proximal Policy Optimisation (PPO), Soft Actor-Critic (SAC), Twin Delayed DDPG (TD3), model-based Dreamer-v2, and offline Conservative Q-Learning (CQL) -- across four autonomous system domains: robotic manipulation (simulated 7-DOF arm), autonomous navigation (urban driving in CARLA simulator), quadrotor control (attitude stabilisation and waypoint tracking), and multi-agent coordination (warehouse logistics). A total of 2,400 training runs were conducted under standardised simulation budgets (10 million environment steps per run) with fixed computational resources to isolate algorithmic effects from implementation variance. SAC achieved the highest mean success rate on continuous-control tasks (manipulation success = 87.4 +- 2.8%; quadrotor tracking error = 0.12 +- 0.03 m) while PPO demonstrated superior robustness to hyperparameter sensitivity (coefficient of variation across 60 configurations = 0.08 vs. 0.18 for SAC). Dreamer-v2 achieved competitive final performance with 4.2x fewer environment interactions than model-free methods, confirming the sample efficiency advantage of model-based approaches. The sim-to-real transfer gap, measured by deploying trained policies on physical hardware, ranged from 8.4% (quadrotor) to 34.7% (manipulation), with domain randomisation reducing the gap by a mean of 41%. A practical algorithm selection framework mapping task characteristics to recommended RL configurations is proposed.