Release
PyTorch 2.15 tackles the 'one bad GPU' problem
The Register · By Joab Jackson ·
The Brief
- PyTorch 2.15 includes a rewritten backend for Nvidia's Collective Communications Library (NCCL).
- The rewrite is designed to improve fault diagnosis in distributed training.
- According to The Register, one bad GPU may no longer force an entire training job to restart from scratch.
Why it matters
Teams running multi-GPU PyTorch training could lose less compute time when a single GPU fails, if the new NCCL backend behaves as described.
Who’s affected
- ML teams running distributed PyTorch training on Nvidia GPUs
- PyTorch 2.15
- #pytorch
- #nccl
- #gpu
- #distributed-training
- #machine-learning
ML admins will be happy to learn that one bad GPU may no longer mean starting an entire training job from scratch. The latest 2.15 release of PyTorch includes a rewritten backend for Nvidia's Collective Communications Library (NCCL), designed to improve fault diagnosis and…