Deep Learning Acceleration Hardware

Abstract

The compute demands of deep learning have outpaced general-purpose CPU scaling for over a decade, driving a proliferation of specialised acceleration hardware -- GPUs, TPUs, dataflow architectures, neuromorphic chips, and in-memory compute arrays. Choosing the right accelerator for a given workload is increasingly non-trivial: different architectures excel at different points in the training-inference-edge continuum, and the total cost of ownership (TCO) picture is complicated by memory bandwidth bottlenecks, interconnect topology, and software stack maturity. This study benchmarks eight deep learning accelerator platforms -- NVIDIA H100, AMD MI300X, Google TPUv4, Intel Gaudi2, Graphcore IPU-M2000, Cerebras CS-2, SambaNova SN40L, and a representative mobile NPU (Apple M3 Neural Engine) -- across six workload categories: large language model training, vision model training, inference serving (batch and real-time), sparse model inference, and edge deployment. Performance is evaluated on throughput (tokens/s, images/s), energy efficiency (FLOPS/W), memory bandwidth utilisation, and TCO at 3-year horizon. We find that the H100 achieves the highest LLM training throughput (1.84x TPUv4) but the MI300X delivers superior memory-bandwidth-bound inference performance (2.1x H100 on 70B-parameter models) due to its 192GB HBM3 capacity. Cerebras CS-2 achieves the highest energy efficiency on sparse workloads (3.8x H100 at 90% sparsity). A workload-to-accelerator mapping framework is proposed, covering training, dense inference, sparse inference, and edge deployment.