Generalist AI Models and Capabilities

Abstract

Generalist AI models -- foundation models capable of performing competently across a wide range of tasks without task-specific fine-tuning -- represent a qualitative shift in AI capability. Rather than training separate specialised models for each task, a single generalist model can reason, code, write, analyse data, generate images, and assist with domain-specific problems, with performance approaching or exceeding specialised models on many tasks. This study provides a systematic evaluation of eight leading generalist AI models across twelve capability dimensions: language understanding, mathematical reasoning, code generation, scientific knowledge, commonsense reasoning, long-context processing, multimodal understanding, instruction following, tool use and function calling, creative writing, factual accuracy, and safety alignment. Models evaluated include GPT-4o, Claude 3.5 Sonnet, Gemini 1.5 Pro, LLaMA-3.1-405B, Mistral-Large-2, Qwen2.5-72B, Command R+, and Phi-3.5-MoE. Evaluation uses 24 standardised benchmarks plus a novel Generalist Capability Index (GCI) that aggregates across all twelve dimensions. GPT-4o and Claude 3.5 Sonnet achieve the highest GCI scores (8.42 and 8.38/10 respectively). LLaMA-3.1-405B achieves the best open-weight GCI (7.84). Phi-3.5-MoE achieves a remarkable GCI of 7.24 at 42B active parameters -- the best efficiency-performance ratio. Mathematical reasoning and long-context processing show the largest inter-model variation. All models show residual safety alignment gaps on targeted adversarial evaluation. A generalist capability taxonomy and benchmark protocol are proposed.