Continual Learning in Neural Networks

Abstract

Continual learning -- the ability of neural networks to learn new tasks sequentially without forgetting previously acquired knowledge -- addresses one of the most fundamental limitations of current deep learning systems. Standard neural networks, trained on fixed datasets, suffer from catastrophic forgetting when fine-tuned on new tasks: previously learned representations are overwritten by gradients optimised for the new objective. Biological neural systems accomplish continual learning naturally through complementary learning systems, synaptic consolidation, and memory replay - mechanisms that AI researchers have sought to translate into computationally tractable algorithms. This study provides a comprehensive evaluation of twelve continual learning methods across five benchmark protocols: Split-CIFAR-100 (task-incremental and class-incremental), Split-MNIST (permuted and split variants), CORe50 (object recognition in changing environments), Split-TinyImageNet, and a proposed Sequential NLP Benchmark (SNB) covering sentiment, NER, and question-answering tasks sequentially. Methods include regularisation-based (EWC, SI, LwF, MAS), architecture-based (PackNet, PNN, HAT), rehearsal-based (ER-ACE, DER++, GDumb), and hybrid approaches (A-GEM, CLOC -- proposed). CLOC achieves the best class-incremental accuracy on Split-CIFAR-100 (48.4%) through combined online distillation and sparse task coding. DER++ leads on task-incremental (84.2%). EWC remains competitive on low-memory settings. The plasticity-stability trade-off is quantified across all methods, and a continual learning method selection framework is proposed