Release
Introducing Amazon SageMaker HyperPod Global Inference Gateway: One endpoint across Regions and clusters
AWS · By Dmitry Soldatkin ·
The Brief
- Amazon SageMaker HyperPod Global Inference Gateway provides one endpoint across multiple HyperPod clusters and AWS Regions.
- It routes inference requests to clusters with available GPU capacity instead of leaving requests queued on a saturated cluster.
- The source describes the problem it targets: idle GPUs on another cluster, sometimes in another Region, cannot be reached during traffic peaks.
Why it matters
Teams running inference on multiple HyperPod clusters can use spare capacity during traffic peaks without managing routing across endpoints themselves.
Who’s affected
- Teams running inference on Amazon SageMaker HyperPod
- Teams operating multiple HyperPod clusters across AWS Regions
- #aws
- #sagemaker
- #inference
- #gpu
- #multi-region
- #machine-learning
During a traffic peak, a request can sit in a queue on a saturated HyperPod cluster while GPUs on another cluster, sometimes in another AWS Region, sit idle. You have already paid for that capacity, but your requests can’t reach it. In Introducing Amazon SageMaker HyperPod…