Multi-Agent Reinforcement Learning

Abstract

Multi-agent reinforcement learning extends RL to settings where multiple agents interact within a shared environment, producing emergent coordination, competition, and communication behaviours that no single agent could achieve alone. This study presents a controlled evaluation of five MARL paradigms -- independent learners (IQL), centralised training with decentralised execution (QMIX), communication-enabled agents (CommNet), opponent modelling (LOLA), and population-based training (PBT) -- across four multi-agent domains: cooperative navigation (MPE Spread, 3-6 agents), competitive two-player games (Pommerman, StarCraft micro-management), mixed cooperative-competitive (MPE Predator-Prey), and multi-agent traffic control (SUMO intersection management, 4-16 intersections). A total of 2,040 experiments were conducted under standardised training budgets. QMIX achieved the highest cooperative task performance (mean reward 18.4 +- 0.8 on 6-agent Spread vs. 14.2 for IQL), confirming that centralised value decomposition substantially outperforms independent learning on tasks requiring tight coordination. CommNet improved cooperative performance by a further 8.4% over QMIX by learning explicit inter-agent communication protocols, with analysis revealing emergent message semantics (spatial location, intention signalling). LOLA opponent modelling improved competitive game performance by 12.8% over self-play by anticipating opponent adaptation. PBT maintained the highest diversity of strategies (Shannon entropy = 2.84 vs. 1.42 for self-play), preventing the strategy collapse that degrades competitive robustness. Scaling from 3 to 16 agents degraded IQL performance by 42.6% but QMIX by only 18.4%, confirming that value decomposition scales more gracefully. A practical MARL method selection guide mapping interaction structure, agent count, and communication availability to recommended approaches is proposed.