International
The Morning Dev

Only the news worth knowing

‹
 
Software Engineering Radio cover

🎧 Software Engineering Radio · In-depth engineering topics

SE Radio 747: Sabina Anja on How AI Workloads Shape GPU Network Design

· 54 min

Listen now · Software Engineering Radio

SE Radio 747: Sabina Anja on How AI Workloads Shape GPU Network Design

0:00−53:46
Speed

In this episode

Sabina Anja , Chief Technologist and executive advisor at VMware Cloud Foundation, talks with host Amey Ambade about how AI workloads reshape GPU network design, and why the network moves into the critical path of computation once a model is too large for a single GPU. Using Llama 3's 405 billion parameters as a starting point, Sabina explains how sharding spreads model weights across many GPUs. She then walks through the difference between scaling up inside a server over NVLink and scaling out across servers over the cluster network. She describes collective traffic as synchronous: no GPU moves to the next step until every GPU finishes, so one slow GPU or congested link, called a straggler,…

Full show notes ↗