Self-Supervised Learning for Vision Tasks

Abstract

Self-supervised learning has emerged as the dominant paradigm for visual representation learning, enabling models to acquire rich feature hierarchies from unlabelled images that transfer effectively to downstream tasks with minimal labelled data. This study presents a controlled empirical comparison of six self-supervised methods -- SimCLR, MoCo-v2, BYOL, SwAV, DINO, and MAE (Masked Autoencoders) -- evaluated across five downstream vision tasks: linear classification (ImageNet-1K), fine-tuning with limited labels (1% and 10% of ImageNet), object detection (PASCAL VOC), semantic segmentation (ADE20K), and out-of-distribution robustness (ImageNet-C). All methods were pre-trained on ImageNet-1K without labels using ResNet-50 and ViT-B/16 backbones under a fixed compute budget of 800 GPU-hours per method. A total of 1,440 experiments were conducted. MAE achieved the highest linear probe accuracy with ViT-B/16 (74.8 +- 0.3%) and the strongest out-of-distribution robustness (mCE = 52.4 on ImageNet-C), while DINO excelled on dense prediction tasks (segmentation mIoU = 44.6 +- 0.4%). Contrastive methods (SimCLR, MoCo-v2) remained competitive with ResNet-50 backbones (linear probe accuracy 71.4 +- 0.2% for MoCo-v2) but underperformed masking-based methods with ViT. The label efficiency of self-supervised representations was substantial: pre-trained features with only 1% labelled data achieved 62.4 +- 0.8% accuracy, outperforming supervised training from scratch with 10% labels (58.2 +- 1.2%). A practical method selection guide matching backbone architecture, downstream task type, and compute budget to recommended SSL configurations is proposed.