Explainability in Foundation Models

Abstract

Foundation models -- large pre-trained models adapted to diverse downstream tasks -- have achieved remarkable performance across language, vision, and multimodal domains, yet their internal decision-making processes remain opaque, creating barriers to trust, accountability, and regulatory compliance in high-stakes deployments. This study presents a systematic evaluation of six explainability methods -- attention visualisation, gradient-based attribution (Integrated Gradients), perturbation-based attribution (SHAP), probing classifiers for internal representations, mechanistic interpretability (circuit analysis), and natural language explanations generated by the model itself (chain-of-thought) -- applied to three foundation model families: BERT-large (NLP classification), ViT-L/16 (image classification), and GPT-2 XL (text generation). Explainability was evaluated across four dimensions: faithfulness (do explanations reflect true model reasoning?), plausibility (do explanations match human intuition?), consistency (do similar inputs receive similar explanations?), and actionability (can explanations guide model improvement?). A total of 2,400 explanation evaluations were conducted with 24 human annotators across three expertise levels. Integrated Gradients achieved the highest faithfulness on NLP tasks (sufficiency = 0.82 +- 0.04; comprehensiveness = 0.76 +- 0.05), confirming gradient-based methods as the most reliable for identifying causally relevant input features. Attention maps showed high plausibility (human agreement = 78.4 +- 3.2%) but poor faithfulness (sufficiency = 0.54 +- 0.06), confirming the well-documented disconnect between attention patterns and actual model reasoning. Chain-of-thought explanations from GPT-2 XL were rated most understandable by non-expert users (comprehension score = 4.2/5) but were faithful to the model's actual computation in only 42.6% of cases. Mechanistic circuit analysis provided the deepest causal understanding but scaled poorly, requiring 40+ researcher-hours per circuit discovered. No single method satisfied all four dimensions simultaneously, confirming that explainability requires a multi-method approach tailored to the stakeholder and deployment context. A practical explainability selection framework mapping stakeholder type, regulatory requirements, and model architecture to recommended methods is proposed