Amazon launches Global Inference Gateway for cross-region LLM load balancing
Amazon has introduced the Global Inference Gateway for SageMaker HyperPod, enabling a single endpoint to route requests across multiple clusters and AWS Regions. The gateway manages traffic based on real-time health and load signals like KV cache utilization and queue depth, automating cross-region failover.