Architecture AI Updates: October 21, 2026

1. Taking Agents From FastAPI Prototypes to Ray on Kubernetes, With an Agent Control Plane and Eval-Gated Releases

InfoQ (Deepak Mohanakumar Chandramouli and Bhumik Vinodkumar Thakkar, Apple). In a QCon AI talk published by InfoQ, two Apple engineers argue that packaging an agent as a FastAPI service in a standard Kubernetes deployment hits scaling limits once agents are multi-turn, stateful and making many parallel tool calls, long-running or streaming. They describe running agents on Ray Serve, with heterogeneous CPU and GPU RayClusters managed by the KubeRay operator, Ray actors for parallel stateful or stateless tasks, and autoscaling driven by in-flight requests per replica (for example, target_ongoing_requests adding a replica past five) rather than CPU alone. Once several agents run in production, they propose a central control plane that handles quotas, queues and region placement for multi-region fault tolerance, plus a shared data plane for per-request operations such as guardrails. Releases are gated on evals: any prompt or model change triggers an LLM-as-a-judge run scoring safety and instruction adherence before deployment. Source