AWS AI Updates: September 25, 2026

1. HyperPod Inference Gateway Routes on KV Cache State Instead of Round Robin

AWS shipped an Inference Gateway for SageMaker HyperPod that replaces round-robin load balancing with routing driven by live inference signals. Its Endpoint Picker scores six metrics per request, including KV cache utilization, queue depth, LoRA adapter residency, prefix cache hit rate, predicted latency, and running request count, then picks the pod. AWS reports first-token latency reductions of up to 82 percent and p99 TTFT reductions of 97 to 98 percent, and the gateway installs as a single EKS managed add-on with no application changes against OpenAI-compatible servers such as vLLM and SGLang. Source

2. A Reference Architecture Keeps Line-of-Business Data in Its Own Account While an Agent Queries It

AWS published a build for a central agent that reasons across data spread over multiple accounts without centralizing any of it. Each line of business deploys its own MCP server on AgentCore Runtime, AgentCore Gateway aggregates them behind a single MCP endpoint with semantic tool discovery, and Policy in AgentCore evaluates Cedar rules against the caller’s Okta JWT claims before a tool call is routed. The Gateway exchanges the user JWT for OAuth 2.0 machine-to-machine credentials from AgentCore Identity, and allowedWorkloadConfiguration blocks Runtime invocations that do not arrive through the Gateway. Source

3. The WhisperX Container on SageMaker Needs One Pinned AMI Version or It Fails Silently

AWS documented a speaker-labeled transcription pipeline combining Whisper for ASR, wav2vec2 for forced alignment, and diarization, packaged as a WhisperX Deep Learning Container. The walkthrough recommends ml.g4dn.xlarge for cost or ml.g5.2xlarge for headroom, and flags that InferenceAmiVersion must be pinned to al2-ami-sagemaker-inference-gpu-3-1 or the endpoint dies with a zero-log CannotStartContainerError. Real-time endpoints must finish inside the 60-second response cap, so long audio belongs on asynchronous endpoints, which can also scale to zero. Source

4. Aderant Triages Support Tickets on Nova Lite for Under $1 a Month in Inference

Aderant built ticket triage for a 38-person team covering 268 client environments on Amazon Nova Lite through the Bedrock Converse API. A Lambda workflow fires hourly from EventBridge, enriches Jira tickets with context from Athena, Confluence, SharePoint, and ticket history, then classifies and routes them. Routing accuracy was roughly 96 percent across 109 tickets with 4 misroutes, recovering 8 to 14 engineering hours a week at under $30 a month total and under $1 of that in Bedrock inference. Source

5. A New Billing API Exists Because Agents Kept Guessing at Account Hierarchy

AWS added ListBillingViewSegments to Billing and Cost Management, returning billing context rather than cost and usage data: where an account sits in the hierarchy, who manages the billing relationship, whether rates are billable or pro forma, and time segments covering mid-period configuration changes. The announcement states the API can be called directly or through an AI agent, though it does not mention MCP support. It is available in all commercial regions at no additional charge, scoped to primary billing views. Source