Self-Supervised Learning in Speech Recognition

Abstract

Self-supervised learning (SSL) has transformed automatic speech recognition (ASR) by enabling powerful speech representations to be learned from vast quantities of unlabelled audio, dramatically reducing dependence on expensive transcribed data. Models such as wav2vec 2.0, HuBERT, WavLM, and Whisper have achieved near-human word error rates on English benchmarks, yet their performance across low-resource languages, noisy acoustic environments, and speaker-diverse populations remains uneven and poorly characterised at scale. This study benchmarks six SSL-based ASR systems -- wav2vec 2.0 Large, HuBERT Large, WavLM Large, Whisper Large-v3, MMS (Massively Multilingual Speech), and a proposed cross-lingual fine-tuning approach (XL-SSL) -- across 18 languages spanning five language families, four acoustic conditions (clean, additive noise, reverberation, and telephone channel), and three speaker-diversity dimensions (sex, age, accent). Evaluation used CommonVoice 15.0, FLEURS, and a purpose-built low-resource European minority language corpus (EMILC, n=142 hours across six languages). WavLM Large achieves the lowest overall WER on clean English (2.1%) and noise-robust conditions. MMS demonstrates the strongest cross-lingual transfer to low-resource languages (mean WER 18.4% on EMILC vs. 34.2% for wav2vec 2.0). Speaker-age WER disparities of 8.2-14.6 percentage points between under-65 and over-75 speakers are identified across all models. XL-SSL reduces low-resource WER by 22.4% relative through curriculum cross-lingual fine-tuning. Practical guidelines for SSL-ASR deployment across resource-constrained and multilingual settings are proposed