Release
SageMaker Inference now supports Multi-Instance GPU to maximize GPU utilization for model endpoints
AWS · By Shantanu Tripathi ·
The Brief
- Amazon SageMaker Inference now supports NVIDIA Multi-Instance GPU (MIG) on model endpoints.
- Previously, the smallest GPU unit a GPU endpoint could request was one whole accelerator.
- The feature aims to raise GPU utilization by allowing more models to run on one instance.
Why it matters
Developers hosting models on SageMaker GPU endpoints may be able to fit more models on each instance instead of dedicating a whole accelerator to each one.
Who’s affected
- Teams hosting models on SageMaker GPU endpoints
- #sagemaker
- #aws
- #gpu
- #inference
- #machine-learning
When you deploy a model to a GPU endpoint, the smallest unit of GPU you can request is one whole accelerator. This constraint can cost you twice. First, you can’t host more models on an instance than it has accelerators. An ml.p4d.24xlarge has eight A100s, so it holds eight…
Timeline
SageMaker Inference Multi-Instance GPU support announced on the AWS Machine Learning blog
Artificial Intelligence
SageMaker Inference now supports Multi-Instance GPU to maximize GPU utilization for model endpoints
