AWS Nitro Sandboxes on EKS: Hands-On with Isolated and Confidential Pods
By Fabio Douek
Human-led, AI-assisted. Research and implementation done with AI tools, led, tested and edited by me. How posts are labeled
Jump to section
- Overview
- Architecture
- Setup
- Step 1: Tools
- Step 2: A VPC and an Auto Mode cluster with Terraform
- Step 3: The RuntimeClasses, before any node
- Step 4: A Graviton NodePool, capped at one node
- Step 5: The one-line change
- Testing
- Test 1: The first pod, and the node it brings
- Test 2: A sandbox pod and a plain pod on the same node
- Test 3: Start latency on a warm node
- Test 4: Host-level settings inside a sandbox
- Test 5: How many sandbox pods fit on a node
- Test 6: A confidential pod, end to end
- Test 7: Confidential, isolated and tampered
- Results summary
- Operating and Monitoring
- Pricing
- Cleanup
- Verdict
- What works
- What doesn’t, yet
- Who should use it
Explain (TLDR) like I am...
Imagine a big toy shelf where any toy can now get its own little box with a lid. You write one word on the toy's tag, and the shelf boxes it for you, so a messy toy can't spill onto the others.
There is also a special box with a seal that says exactly what is inside. A grown-up only opens the cookie jar for a box whose seal matches. The shelf says it has room for 20 boxes, but on some days only 18 or 19 of them fit, and I haven't worked out why yet.
On Amazon EKS, hardware isolation is now a property of the pod. One field in the pod spec runs the pod in its own virtual machine on a shared node. The pod spec itself remains trusted input, so whoever may submit pods controls what runs inside, and that permission deserves restriction.
Confidential mode adds evidence. The pod presents a signed measurement of its own policy, and an AWS KMS key released a secret only on a match. The same pod in isolated mode, without that policy, was refused, interactive access was denied, and a copy with an altered command was stopped before its container started. Two documented behaviours did not match what I observed, so verify before relying on them.
The presenting problem is pods on one node sharing a single kernel. The intervention is a dedicated virtual machine per pod, prescribed with one line in the pod spec. In my run the sandboxed pod had its own kernel, one vCPU and memory sized from its request, next to a regular pod that saw the whole node.
Side effects: about three seconds longer to start, 256 MiB reserved per pod (512 MiB when confidential), and a ceiling of five sandboxes per vCPU: my node advertised 20, and 18 to 20 were usable. For confidential pods the evidence is strong: KMS released a key only to the pod whose policy measurement matched.
Teams that run code they didn't write often carry a quiet worry about what a bad image could reach. Here the answer is one line in the pod spec, and on Auto Mode nothing else needs installing, which takes some of the dread out of saying yes.
The new weight is bookkeeping. Confidential pods need a generated policy, and a change to the image, the command, the environment or even the pod's name means regenerating it and updating the key policy. My 4-vCPU node advertised 20 sandbox slots, five per vCPU, and I couldn't always fill all of them, so someone will be watching capacity more closely than before, and Container Insights won't help them: it reported nothing for the sandboxed pods.
Picture one rehearsal room where every act shared the same walls. Now each act can get its own soundproof booth inside that room, sized to what it asked for: one vCPU and the memory on its request, nothing borrowed from the house.
The booths take about three seconds longer to set up than an open corner, and a room this size has 20 spots, though some nights only 18 or 19 acts fit. The confidential booth plays from a sealed chart, and if anyone edits a bar, the act never takes the stage.
The story for Kubernetes teams is simple: per-pod virtual machine isolation on EKS Auto Mode, switched on with one field in the pod spec, at no additional charge. The runtime classes were there before the cluster had a single node.
The proof point is confidential mode. A KMS key that opens only for a pod whose policy measurement matches, with interactive access denied, is a demo regulated buyers grasp at once. Be honest about density: five sandboxes per vCPU and 256 MiB reserved per pod suit real workloads better than swarms of tiny ones.
Overview
On 2026-11-09 AWS launched Nitro Sandboxes: lightweight VMs that the Nitro hypervisor carves out of an EC2 instance, each one a first-level guest next to its parent. On Amazon EKS the feature is sandboxing with isolated and confidential pods: each pod runs in its own Nitro Sandbox, “a first-level guest with direct hardware-assisted virtualization, not through emulation or nested virtualization”, and many of them share one node.
This post, part 4 of 4, covers EKS. Part 1 explains the architecture and compares the three ways in, part 2 drives the raw primitive on EC2 with Firecracker, and part 3 covers ECS.
On EKS, the announcement’s “one-line change” is runtimeClassName, and on EKS Auto Mode the docs promise the rest: “Auto Mode includes Nitro Sandbox support with no additional setup. You do not build or manage an AMI.” I wanted to know what that sentence leaves out: what Auto Mode actually installs, whether a Graviton node works when the built-in node pool is amd64 only, what a sandboxed pod sees next to a plain one, how many fit on a node, and whether confidential mode’s attestation works end to end through Pod Identity. I ran it twice on an Auto Mode cluster on Kubernetes 1.37 in us-east-1 on 2026-11-10, with one m8g.xlarge node at a time (Graviton4, 4 vCPU, 16 GiB), kubectl 1.37.2 and AWS CLI 2.39.7: an exploration run from 00:12 UTC, and the recorded run the video shows, on a fresh cluster from 02:08 to about 06:30. Where the recorded run shows a number, I use it and give the first run’s alongside. A third run later that day, for the video’s operations section, built the cluster in the console and looked at monitoring and CloudTrail (Operating and Monitoring).
Three results stood out:
- On Auto Mode, the one-line change really is the whole setup. Both RuntimeClasses existed before the cluster had a node, and a NodePool with no sandbox setting produced a sandbox-capable Graviton node. Next to a plain pod on that node, the sandboxed pod had its own kernel, one vCPU and memory sized from its request, and on a warm node it was Ready in 4.4 to 5.4 seconds, against 1.7 to 2.1 for a plain pod.
- Confidential mode gates a KMS key on the pod’s policy, through Pod Identity. The policy generator predicted PCR17 exactly, in both runs. KMS released a secret to the confidential pod and refused the same pod running isolated;
kubectl execinto the confidential pod was denied, and a copy with only its command changed was refused atCreateContainer, as on ECS. - An
m8g.xlargeadvertises 20 sandbox slots (5 × its 4 vCPUs), and I couldn’t always use all of them. That’s the same cap I hit on EC2. In four runs of the same 22-replica Deployment, 18 to 20 pods ran and the rest stayedPendingfor lack of a slot, and I didn’t establish what held the missing ones.
Architecture
The pod path is the usual Kubernetes one with a different runtime at the end. The kubelet calls containerd, containerd calls containerd-shim-nitro-sandbox-v2, and the shim starts a Firecracker process that asks Nitro for a sandbox. Inside the guest, nitro-sandbox-agent runs its own containerd and starts the pod’s containers. On self-managed nodes the docs wire the same shim binary between the kubelet and containerd as a cri-proxy; on Auto Mode it’s there too, because that’s the component that refused my hostNetwork pod in Test 4. Logs, exec and metrics travel back over vsock.
The RuntimeClass does the scheduling work. Its overhead adds memory and one eks.amazonaws.com/nitro-sandbox to every pod’s requests, and its nodeSelector sends the pod to a node labelled eks.amazonaws.com/nitro-sandbox=true; the extended-resource request means that node must also have a free sandbox slot. In the guest, the agent evaluates a Cedar policy for every gated operation: the pod’s sandbox.amazonaws.com/policy annotation if it has one, otherwise, in isolated mode, “a permissive built-in default that allows all containers and operations”. The policy is measured into PCR17 of the sandbox’s attestation document, which is what a KMS key policy can pin.
EKS has two modes, picked by the RuntimeClass. In isolated-sandbox “the parent instance is trusted and the guest is untrusted”, and the policy is optional. In confidential-sandbox the parent is untrusted, and the policy “pins every admission field (image digests, args, environment variables, mounts, capabilities, and security settings)”. Pods without runtimeClassName use the node’s default runtime, on the same node. Test 2 shows which EC2 memory mode sits under the isolated class.
Setup
Terraform creates the VPC and the cluster; everything after that is kubectl and the AWS CLI.
Step 1: Tools
You need kubectl 1.36 or later (I used 1.37.2), Terraform, and the AWS CLI. I used AWS CLI 2.39.7, as in part 2 and part 3, but nothing on EKS needs a new field: the EKS CreateCluster, ComputeConfigRequest and UpdateClusterConfig API pages don’t mention sandboxes or enclaves at all. You don’t need eksctl. The one Linux-only tool is the policy generator for confidential pods. For this demo I ran it in a pod on the cluster (Test 6), so I didn’t need a Linux machine, but that’s a shortcut: in production, run it in the CI/CD pipeline that produces your pod manifests or on a Linux workstation or container, for the reason in Test 6.
Step 2: A VPC and an Auto Mode cluster with Terraform
The hard requirements are the version and the Region. “Nitro Sandboxes requires Kubernetes version 1.36 or later. It is supported on the two latest versions,” says the versions page. The overview, in a line added after launch, says isolated and confidential pods “are available in the US commercial AWS Regions”, while the announcement still puts Nitro Sandboxes “in all commercial regions worldwide”. My runs were in us-east-1, so I can’t say what happens elsewhere. Neither the terraform-aws-modules/eks module (21.29.0) nor the Terraform AWS provider has a sandbox argument, because there’s nothing to switch on at the cluster level. The Terraform builds a VPC (10.63.0.0/16, two private and two public subnets, one NAT gateway) and this cluster:
# EKS Auto Mode, Kubernetes 1.37. Nitro Sandboxes need 1.36+; there is no
# sandbox-specific argument in the module, the provider or the EKS API.
# Only the "system" built-in pool is enabled: it keeps the default NodeClass
# and does not launch a node on its own. The general-purpose pool is amd64-only,
# so the Graviton (m8g.xlarge) NodePool is created with kubectl.
module "eks" {
source = "terraform-aws-modules/eks/aws"
version = "~> 21.0"
name = local.name # my2cents-nitro-eks
kubernetes_version = "1.37"
endpoint_public_access = true
enable_cluster_creator_admin_permissions = true
compute_config = {
enabled = true
node_pools = ["system"]
}
vpc_id = module.vpc.vpc_id
subnet_ids = module.vpc.private_subnets
}
node_pools = ["system"] is the line that needs explaining. Auto Mode’s built-in general-purpose pool “Uses only amd64 architecture”, per the built-in pools page, so it can’t give me a Graviton node. But “Enabling at least one built-in NodePool is required for EKS to provision the ‘default’ NodeClass”, and my own NodePool in Step 4 needs that NodeClass. The system pool supports arm64 and carries a CriticalAddonsOnly taint, and it didn’t launch a node on its own: after terraform apply, which took 13 minutes 42 seconds in the first run and about 13 minutes in the recorded one, the cluster had zero nodes.
terraform init && terraform apply
aws eks update-kubeconfig --region us-east-1 --name my2cents-nitro-eks
kubectl get nodes # No resources found
The control plane reported v1.37.1-eks-577a51e. The docs’ one-command alternative is eksctl create cluster --name sandbox-demo --region us-west-2 --version 1.37 --enable-auto-mode; keep the explicit version, because the docs warn that eksctl defaults to an older one.
The console works too. In the operations run I created the same cluster with Create cluster > Custom configuration: remove general-purpose from the built-in node pools so only system is left, and note that the add-ons step pre-selects the Metrics Server community add-on. I unticked it at creation, so the sandbox node would be the only node, and added it later. The cluster was Active about 12 minutes after Create, and the cluster creator, my console role, became its admin; my CLI user needed an access entry with AmazonEKSClusterAdminPolicy.
Step 3: The RuntimeClasses, before any node
On the fresh cluster, still with zero nodes, kubectl get runtimeclass already listed confidential-sandbox and isolated-sandbox:
NAME HANDLER AGE
confidential-sandbox confidential-sandbox 2m57s
isolated-sandbox isolated-sandbox 2m57s
That’s the recorded run; in the first they were about four minutes old. The user guide doesn’t publish the Auto Mode objects, so here is the isolated one, identical in both runs:
kubectl get runtimeclass isolated-sandbox -o yaml
apiVersion: node.k8s.io/v1
handler: isolated-sandbox
kind: RuntimeClass
metadata:
name: isolated-sandbox
overhead:
podFixed:
eks.amazonaws.com/nitro-sandbox: "1"
memory: 256Mi
scheduling:
nodeSelector:
eks.amazonaws.com/nitro-sandbox: "true"
confidential-sandbox is the same with memory: 512Mi. Both match what the AMI page publishes for self-managed nodes. The overhead is how the scheduler pays for the VM: every pod with this RuntimeClass gets 256Mi (or 512Mi) and one eks.amazonaws.com/nitro-sandbox added to its requests.
The default NodeClass has no sandbox field either: ephemeralStorage 80Gi, networkPolicy: DefaultAllow, snatPolicy: Random.
Step 4: A Graviton NodePool, capped at one node
No AWS page has a sandbox NodePool, so I wrote an ordinary one from the documented sample, limited to m8g.xlarge on demand:
apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
name: sandbox-m8g
spec:
template:
spec:
nodeClassRef:
group: eks.amazonaws.com
kind: NodeClass
name: default
requirements:
- key: node.kubernetes.io/instance-type
operator: In
values: ["m8g.xlarge"]
- key: karpenter.sh/capacity-type
operator: In
values: ["on-demand"]
limits:
cpu: "4" # one m8g.xlarge, never two
kubectl apply -f nodepool.yaml
There’s no sandbox requirement in it, and it needed none: when a sandbox pod was pending, Auto Mode launched an m8g.xlarge with the sandbox label and capacity (Test 1). limits.cpu: "4" held in both runs. Pods beyond what one node could take stayed Pending with all available instance types exceed limits for nodepool, instead of buying a second instance the way ECS did in part 3.
Step 5: The one-line change
The test pod is the docs’ example with a probe and a nodeSelector on Auto Mode’s node.kubernetes.io/instance-type label, which keeps it on the Graviton pool:
apiVersion: v1
kind: Pod
metadata:
name: hello-sandbox
spec:
runtimeClassName: isolated-sandbox # the one-line change
nodeSelector:
node.kubernetes.io/instance-type: m8g.xlarge
containers:
- name: probe
image: public.ecr.aws/amazonlinux/amazonlinux:2023
command: ["bash", "-c"]
args:
- |
echo "kernel: $(uname -rv)" # the full probe is in the block below
sleep infinity
resources:
requests: {cpu: 500m, memory: 512Mi}
limits: {cpu: "1", memory: 1Gi}
The plain pod, hello-plain, is the same file without the runtimeClassName line.
The probe both pods run
echo "kernel: $(uname -rv)"; echo "boot_id: $(cat /proc/sys/kernel/random/boot_id)"; echo "uptime: $(cut -d' ' -f1 /proc/uptime)"; echo "nproc: $(nproc)"; grep MemTotal /proc/meminfo; echo "disks: $(ls /sys/block | tr '\n' ' ')"; ls -la /dev/nsm 2>&1 | head -1
echo "pci: $(for d in /sys/bus/pci/devices/*; do printf '%s ' $(cat $d/vendor):$(cat $d/device); done | tr ' ' '\n' | sort | uniq -c | tr '\n' ' ')"
T=$(curl -s -m3 -X PUT http://169.254.169.254/latest/api/token -H 'X-aws-ec2-metadata-token-ttl-seconds: 60'); echo "imdsv2 token: ${#T} chars"
for p in placement/region placement/availability-zone instance-id iam/security-credentials/; do echo "imds $p: $(curl -s -m3 -w ' [HTTP %{http_code}]' -H "X-aws-ec2-metadata-token: $T" http://169.254.169.254/latest/meta-data/$p | tr '\n' ' ' | head -c 120)"; done
echo "sa token: $(ls /var/run/secrets/kubernetes.io/serviceaccount/token 2>&1)"
echo "procs: $(ls /proc | grep -c '^[0-9]')"
For a Deployment, runtimeClassName goes in spec.template.spec (Test 5 has one). Sandbox annotations go in spec.template.metadata; the setup page warns that an annotation on the controller’s own metadata “is accepted and then ignored”.
Testing
Test 1: The first pod, and the node it brings
I applied hello-sandbox to the cluster with no nodes and waited for it:
kubectl apply -f pod-sandbox.yaml && kubectl wait pod/hello-sandbox --for=condition=Ready --timeout=600s
In the recorded run the pod was Ready 50 seconds after kubectl apply, including launching the node, which was 14 seconds old when I listed it:
NAME STATUS ROLES AGE VERSION INSTANCE-TYPE NITRO-SANDBOX
i-056659827b67f2792 Ready <none> 14s v1.37.0-eks-29d7970 m8g.xlarge true
In the first run it took 85 seconds. That pod’s events:
| Time (UTC) | Event |
|---|---|
| 00:29:16 | Created; FailedScheduling, no nodes |
| 00:30:11 | Nominated to nodeclaim/sandbox-m8g-… |
| 00:30:39 | Pulled, in 1.466 s |
| 00:30:41 | Started |
For comparison, the first isolated task on ECS in part 3 was running about 34 seconds after creation, including its instance.
kubectl describe node on the node Auto Mode launched in the first run (the recorded run’s node also had no taints and 20 allocatable eks.amazonaws.com/nitro-sandbox):
| Resource | Capacity | Allocatable |
|---|---|---|
cpu | 4 | 3770m |
memory | 16048204Ki | 13513Mi |
eks.amazonaws.com/nitro-sandbox | 20 | 20 |
Its kubelet was v1.37.0-eks-29d7970, its labels included eks.amazonaws.com/nitro-sandbox=true and kubernetes.io/arch=arm64, it had capacity for 56 pods, and it had no taints. Twenty is five times the node’s vCPUs, the same per-instance cap I hit on EC2 in part 2 (pcie_slots_total); Test 5 fills it.
Auto Mode treats a sandbox node like any other when it consolidates. After I deleted the sandbox pods, leaving at most a plain pod, it replaced the node, twice in the first run: i-023f6942008372e7d, then i-054c09a2cf3dca87f, then i-0956c7c6b4bd96b04. In the recorded run it did the same between steps: Test 1’s i-056659827b67f2792 was gone by Test 5, which ran on i-08330145d8695dd72. In the operations run, a NodePool with disruption: {consolidationPolicy: WhenEmpty, consolidateAfter: 5m} kept its node for the whole 22- to 32-minute session, though add-on pods also ran on it, so it was never empty.
Test 2: A sandbox pod and a plain pod on the same node
Same image, same probe, same requests (500m, 512Mi) and limits (1 CPU, 1Gi), same node. The only difference is runtimeClassName: isolated-sandbox. The recorded run printed the kernel, uptime, CPU, memory, disk, NSM and two metadata lines, and every one matched the first run except the uptimes; the other rows are from the first run.
isolated-sandbox | Plain | |
|---|---|---|
| Kernel | 6.18.54 #1 SMP Thu Oct 22 17:48:46 UTC 2026 | 6.18.56 #2 SMP Fri Oct 30 22:32:43 UTC 2026 (the node’s) |
| Uptime at probe | 5.43 s (first run: 5.33 s) | 28.12 s (first run: 37.97 s), the node’s, freshly booted |
nproc | 1 | 4 |
| MemTotal | 666,384 kB | 16,048,204 kB |
| Disks | dm-0 vda vdb vdc | dm-0 loop0-7 nvme0n1 nvme1n1 zram0 |
| PCI | 42 Amazon bridges, 1af4:1041 (net), three 1af4:1042 (block), 1af4:1053 (vsock), 1af4:1061 (NSM), 1d0f:8250 | 1af4:1053, two Amazon NVMe 1d0f:8061, ENA 1d0f:ec20, 1d0f:e4c1 |
/dev/nsm | present (crw-rw-rw-, 10, 258) | absent |
| IMDSv2 token | 48 characters | none |
IMDS placement/region | us-east-1 (HTTP 200) | HTTP 401 |
IMDS placement/availability-zone | us-east-1a (HTTP 200) | HTTP 401 |
IMDS instance-id | Not Found (HTTP 404) | HTTP 401 |
IMDS iam/security-credentials/ | Not Found (HTTP 404) | HTTP 401 |
| Service account token | present | present |
Pod spec.overhead | {"eks.amazonaws.com/nitro-sandbox":"1","memory":"256Mi"} | none |
The sandboxed pod is a separate machine, sized to the pod. Its kernel isn’t even the same version as the node’s on arm64: 6.18.54 against 6.18.56. AWS’s x86 examples show matching versions with different builds, so don’t use uname -r alone as evidence; the build string, the uptime and the virtio disks are the reliable tells. It sees one CPU, its 500m request rounded up, and 666,384 kB of memory, sized from the 512Mi request; the 1Gi limit plays no part. The plain pod sees the node: its kernel, all four CPUs, all 16 GiB and the NVMe disks, whatever its cgroup limits say. The admission controller added the RuntimeClass overhead to the sandboxed pod’s spec and nothing to the plain one’s.
The vCPU isn’t reserved. In the overview’s words, “a sandbox does not get dedicated physical cores. vCPUs are time-shared”, and the VM’s size is fixed at boot, so in-place resize isn’t supported.
The NSM device is there in isolated mode, as it was on ECS in part 3. On EC2 in part 2, a direct-mode sandbox had no NSM and a confidential-mode one did, so isolated-sandbox appears to run on EC2’s confidential memory mode underneath. That’s my inference from the device list; no EKS page maps the two.
Two rows disagree with the docs or with ECS:
- The metadata surrogate answered, as documented. The credentials page describes an IMDSv2-only surrogate at
169.254.169.254that serves five placement and service paths and returns 404 for everything else. That’s what I got: a token, the Region and Availability Zone, and 404 for the instance ID and for credentials. On ECS in part 3 the isolated task got no answer at all. The plain pod got no token, and every request without one came back 401, which fits Auto Mode’s “Restricted access to instance metadata using IMDSv2 with controlled hop limits” from the Auto Mode security guide. Neither pod could reach the node’s credentials. - The service account token was mounted. For isolated mode the overview, revised since launch, still says “The service account token is not replicated into the sandbox.” In an isolated pod with no policy,
/var/run/secrets/kubernetes.io/serviceaccount/tokenwas there. If you were counting on that sentence to keep a Kubernetes API credential out of an untrusted image, turn off token automounting in the pod spec yourself, and check.
Test 3: Start latency on a warm node
With the node up and the image cached, I created pods one at a time (alternating sandboxed and plain in the recorded run) and timed from kubectl apply to the return of kubectl wait --for=condition=Ready:
| Pod | Recorded run | First run |
|---|---|---|
isolated-sandbox | 5.32 s, 4.69 s | 5.37 s, 4.38 s, 4.44 s |
| Plain | 1.97 s, 1.69 s | 2.14 s, 1.73 s; a third stayed Pending: the node was out of CPU, and the NodePool cap held |
The sandbox adds about three seconds to a pod start, in both runs. AWS’s blueprints quote “about 5 to 6 seconds” for a new sandbox pod once the node exists, so my numbers are on the fast side of that. For scale, a bare sandbox on EC2 booted to SSH in about two seconds in part 2; the rest is the agent, the image and the container start inside the guest.
Test 4: Host-level settings inside a sandbox
I applied three variants of the isolated probe pod: one with hostNetwork: true, one with hostPID: true, and one with the network-isolation annotation from the network isolation page:
metadata:
name: sandbox-network-isolated
annotations:
sandbox.amazonaws.com/network-isolated: "true"
spec:
runtimeClassName: isolated-sandbox
| Variant | Result |
|---|---|
hostNetwork: true | Scheduled, then stuck in ContainerCreating, with FailedCreatePodSandBox repeating. Not rejected at admission |
hostPID: true | Runs. PID 1 is systemd (/sbin/init), and 167 processes are visible (170 in the first run): the sandbox VM’s, not the node’s |
sandbox.amazonaws.com/network-isolated: "true" | Network devices lo and dummy0, no routes, curl: (6) Could not resolve host, IMDS unreachable (HTTP 000) |
kubectl exec into an isolated pod | Works (uname -r returns 6.18.54) |
All four reproduced in the recorded run (the isolated kubectl exec is in Test 7). Two of them, from that run:
$ kubectl exec sandbox-hostpid -- bash -c 'echo pid1: $(cat /proc/1/comm), processes: $(ls /proc | grep -c ^[0-9])'
pid1: systemd, processes: 167
$ kubectl exec sandbox-network-isolated -- bash -c '...; curl -sS -m 5 https://aws.amazon.com 2>&1 | head -1'
interfaces: lo dummy0
curl: (6) Could not resolve host: aws.amazon.com
The hostNetwork pod’s event says exactly what’s wrong:
FailedCreatePodSandBox: ... cri-proxy: hostNetwork is not supported for nitro-sandbox pods; remove `hostNetwork: true` from the pod spec
The considerations page lists hostNetwork as unsupported, so the refusal isn’t a surprise; where it happens is. The API server accepts the pod, the scheduler places it, and it then retries forever in ContainerCreating instead of failing. On ECS in part 3, the same idea, networkMode: host, was rejected when I registered the task definition. If you want it refused at the door on EKS, that’s an admission policy you write yourself. Inside the guest the question doesn’t arise: every container shares the VM’s network stack, and the policy generator in Test 6 even pins host_network == true with the comment “the VM owns the network stack, so no network namespace exists in the OCI spec.”
hostPID behaves like ECS’s pidMode: host, which showed 172 processes in part 3: “host” is the sandbox VM, and the systemd at PID 1 is the guest’s own init. Nothing on the node is visible.
Network isolation is real, with the caveat the docs spell out: the pod has no network interface, DNS, NTP, IMDS or Pod Identity, but logs and exec still work over vsock, and the vsock channel stays reachable from the parent, “so this is not absolute isolation”.
The flip side of all this is in the security model: the feature is built for untrusted images, not untrusted pod specs. “Treat pod spec submission as a privileged operation.”
Test 5: How many sandbox pods fit on a node
Every sandbox pod requests one eks.amazonaws.com/nitro-sandbox through its RuntimeClass overhead, and the node advertised 20. I filled it with a Deployment of 22 replicas that do nothing:
apiVersion: apps/v1
kind: Deployment
metadata:
name: density
spec:
replicas: 22
selector:
matchLabels: {app: density}
template:
metadata:
labels: {app: density}
spec:
runtimeClassName: isolated-sandbox
nodeSelector:
node.kubernetes.io/instance-type: m8g.xlarge
containers:
- name: app
image: public.ecr.aws/amazonlinux/amazonlinux:2023
command: ["sleep", "infinity"]
resources:
requests: {cpu: 50m, memory: 64Mi}
kubectl apply -f density.yaml
kubectl get pods -l app=density --no-headers | awk '{print $3}' | sort | uniq -c
kubectl describe node <node-name> | grep nitro-sandbox
The video shows the recorded run, counting pod states about every 13 seconds:
t= 14s running 0 creating 0 pending 22 rejected 0
t= 27s running 0 creating 19 pending 3 rejected 0
t= 40s running 16 creating 3 pending 3 rejected 0
t= 54s running 19 creating 0 pending 3 rejected 0
t= 67s running 19 creating 0 pending 3 rejected 0
...
t=162s running 19 creating 0 pending 3 rejected 0
From 54 seconds on it held at 19 running and 3 pending, and the pending pods said why:
0/1 nodes are available: 1 Insufficient eks.amazonaws.com/nitro-sandbox
The node still reported 20 allocatable. I ran the same test five times in all:
| Run | Starting point | Requests | Result |
|---|---|---|---|
| First run, A | The node had already run and deleted six sandbox pods; one other sandbox pod was running | 100m, 128Mi | 13 density pods ran, 14 sandbox pods in all. The rest stayed Pending with Insufficient eks.amazonaws.com/nitro-sandbox, while kubectl describe node showed 14 of 20 allocated. The 13 took 257 s to reach Running |
| First run, B | A fresh node | 50m, 64Mi | 18 Running within about 66 s, 2 Failed, 4 Pending |
| First run, C | A node whose three sandbox pods I’d deleted about three minutes earlier | 50m, 64Mi | 20 Running, 2 Pending, after 100 s |
| Recorded, off camera | A node with recent sandbox churn, after about three quiet minutes | 50m, 64Mi | 18 Running, 4 Pending |
| Recorded, on camera | Right after the off-camera run, on i-08330145d8695dd72 | 50m, 64Mi | 19 Running, 3 Pending, steady from 54 s to 162 s |
The two failures in first run B were rejected by the kubelet after the scheduler had placed them:
OutOfeks.amazonaws.com/nitro-sandbox: Pod was rejected: Node didn't have enough resource: eks.amazonaws.com/nitro-sandbox, requested: 1, used: 20, capacity: 20
The ceiling is 20 sandbox pods per m8g.xlarge (5 × its 4 vCPUs): the node advertised 20 every time, the kubelet’s rejection says capacity: 20, and first run C got all of them. Each density pod still saw one CPU (nproc 1, MemTotal: 280848 kB), so 20 sandboxes are also 20 vCPUs, five times the node’s four. The docs cap both the count and the total vCPUs at five times the node’s vCPUs, and with 1-vCPU sandboxes the two limits land on the same number, so this run can’t tell them apart. On EC2 in part 2, where I could make each sandbox 4 vCPUs, the count was the binding one.
What I didn’t expect is that I couldn’t count on all 20. With the same 50m / 64Mi Deployment, the four runs got 18, 20, 18 and 19. Two more observations from the first run:
- In run A, the scheduler reported
Insufficient eks.amazonaws.com/nitro-sandboxfor the remaining pods whilekubectl describe nodeshowed only 14 of 20 allocated. When I deleted one more running density pod, its replacement didn’t schedule either, with 13 of 20 allocated. - In run B, I deleted three running pods; their replacements hadn’t run 60 seconds later, leaving 15. And even on that fresh node, the burst got two pods rejected by the kubelet as over capacity while only 18 were running.
After the first run I read this as freed slots coming back after a delay, because run C, after a few quiet minutes, got all 20. The recorded run doesn’t support that reading: after about three quiet minutes the off-camera run got 18, and the run right after it 19. I didn’t establish which component held the missing slots or why the number varied, so all I can report is what I saw: the node advertises 20, I got 18 to 20 with the same Deployment, and fewer right after deletions. The practical effect, for CI runners or agent sessions coming and going, is not to plan on all 20, to watch for Pending pods with Insufficient eks.amazonaws.com/nitro-sandbox, and to expect the occasional Failed pod from a burst.
Test 6: A confidential pod, end to end
A confidential pod needs a Cedar policy in its sandbox.amazonaws.com/policy annotation, generated from its manifest by eks-nitro-sandbox-genpolicy. The guest’s agent enforces it and measures it into PCR17, and a KMS key policy can require that PCR17. On EKS the pod gets its AWS credentials through Pod Identity.
Identity. An IAM role that Pod Identity can assume, a service account, and an association between them. Auto Mode needed no Pod Identity Agent add-on.
{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"Service":"pods.eks.amazonaws.com"},"Action":["sts:AssumeRole","sts:TagSession"]}]}
aws iam create-role --role-name my2cents-nitro-eks-attest --assume-role-policy-document file://pod-trust.json
kubectl apply -f sa.yaml # ServiceAccount attest in namespace default
aws eks create-pod-identity-association --cluster-name my2cents-nitro-eks \
--namespace default --service-account attest \
--role-arn arn:aws:iam::<ACCOUNT_ID>:role/my2cents-nitro-eks-attest
The secret. A new KMS key, one per run, whose policy at first held only an account-administration statement. I encrypted the secret on my machine and put the ciphertext in the pod’s environment as SECRET_CT:
KEY_ID=$(aws kms create-key --policy file://kms-eks-admin.json --query KeyMetadata.KeyId --output text) # only the AccountAdmin statement shown below
printf '%s' 'eks: only a measured pod can read this' > secret.txt
SECRET_CT=$(aws kms encrypt --key-id "$KEY_ID" --plaintext fileb://secret.txt --query CiphertextBlob --output text)
The workload is the one from ECS in part 3. The container (amazonlinux:2023) installs openssl, awscli-2 and python3, writes the stdlib NSM client from part 2 (nsm.py) from an environment variable, makes an RSA key pair, gets an attestation document from /dev/nsm, prints the PCRs, asks KMS to decrypt the secret for that document with the Pod Identity credentials, and opens the answer inside the sandbox. Unrolled, without its error handling:
dnf install -y -q openssl awscli-2 python3
echo "$NSM_PY_B64" | base64 -d > /tmp/nsm.py; cd /tmp
openssl genrsa -out key.pem 2048 && openssl rsa -in key.pem -pubout -outform DER -out pub.der
python3 nsm.py attest pub.der att.doc && python3 nsm.py pcrs att.doc | grep -E "module_id|PCR0:|PCR8:|PCR17:"
echo "$SECRET_CT" | base64 -d > ct.bin
aws kms decrypt --region us-east-1 --ciphertext-blob fileb://ct.bin \
--recipient "KeyEncryptionAlgorithm=RSAES_OAEP_SHA_256,AttestationDocument=$(base64 -w0 att.doc)" \
--query CiphertextForRecipient --output text | base64 -d > cfr.der
echo "SECRET: $(openssl cms -decrypt -inform DER -in cfr.der -inkey key.pem)"
The key pair sits in /tmp, on the container’s writable layer. That’s acceptable here because a confidential sandbox’s overlay is encrypted (the generated policy below requires it), while for isolated mode the docs say “The overlay filesystem is unencrypted to allow faster pod startup.” Part 2 explains why that distinction matters.
The manifest, as it ran, minus two long base64 values:
{
"apiVersion": "v1",
"kind": "Pod",
"metadata": { "name": "attest-confidential", "namespace": "default" },
"spec": {
"serviceAccountName": "attest",
"runtimeClassName": "confidential-sandbox",
"nodeSelector": { "node.kubernetes.io/instance-type": "m8g.xlarge" },
"containers": [
{
"name": "attest",
"image": "public.ecr.aws/amazonlinux/amazonlinux:2023",
"command": ["bash", "-c", "dnf install -y -q openssl awscli-2 python3 >/dev/null 2>&1; ... echo ATTEST_DONE; sleep 3600"],
"env": [
{ "name": "NSM_PY_B64", "value": "<...>" },
{ "name": "SECRET_CT", "value": "<...>" },
{ "name": "AWS_REGION", "value": "us-east-1" },
{ "name": "AWS_DEFAULT_REGION", "value": "us-east-1" }
],
"resources": {
"requests": { "cpu": "500m", "memory": "512Mi" },
"limits": { "cpu": "1", "memory": "1Gi" }
}
}
],
"enableServiceLinks": false
}
}
Three details in it come from gotchas. The command is a single line, joined with ; : on ECS in part 3 the generator rejected a multi-line command, and this way I never hit that. Both AWS_REGION and AWS_DEFAULT_REGION are set, following AWS’s blueprints: if only AWS_REGION is set, the Pod Identity webhook leaves out AWS_DEFAULT_REGION, the policy still expects it, and the guest refuses the container. I set both from the start, so I didn’t test that failure. And enableServiceLinks: false, which I learned the hard way, below.
Generating the policy, in the cluster. AWS ships eks-nitro-sandbox-genpolicy for Linux only (x86_64 and aarch64, glibc 2.34 or later). As on ECS, I ran it on the cluster instead of finding a Linux machine: a plain pod on the arm64 node downloads the signed runtime release, verifies it with the public key printed on the download page, and generates the policy from the manifest, which it reads from a ConfigMap.
m8g.xlarge that later hosted the confidential pod, and its output came back through kubectl logs, which that node's kubelet serves. A compromised node could hand back a permissive policy together with its matching PCR17, and you'd paste that PCR17 straight into the key policy. The confidential setup page says to build regeneration into "the CI/CD pipeline that produces your pod manifests", and the security model says "The pod spec and the Cedar policy generated from it are trusted inputs". In production, generate the policy there or on a Linux workstation or container (glibc 2.34 or later), pin a release in the download URL instead of latest, and review the policy before you trust its PCR17.
kubectl create configmap confidential-manifest --from-file=manifest.json=pod-confidential-nopolicy.json
kubectl apply -f pod-genpolicy.yaml
kubectl logs genpolicy
apiVersion: v1
kind: Pod
metadata:
name: genpolicy
spec:
nodeSelector:
node.kubernetes.io/instance-type: m8g.xlarge
restartPolicy: Never
containers:
- name: genpolicy
image: public.ecr.aws/amazonlinux/amazonlinux:2023
command: ["bash", "-c"]
args:
- |
dnf install -y -q tar gzip openssl findutils >/dev/null 2>&1; cd /tmp
U=https://d1uvmbvklg8rt.cloudfront.net/latest/aarch64
curl -fsSLO $U/nitro-sandboxes-runtime.tar.gz && curl -fsSLO $U/SHA256SUM && curl -fsSLO $U/SHA256SUM.sig
cat > nitro-sandboxes-runtime-v1.pub <<'PEM'
-----BEGIN PUBLIC KEY-----
MHYwEAYHKoZIzj0CAQYFK4EEACIDYgAESqdVXO82n+Zh5+IFLh1xjmxZLkjjZWvf
DCS927SdVYjP0AZ92XCxS+ucrs9cjuNlTaDARTHov2XVfMVgau8TEPFKGDRmVZXJ
c25CVihKHLFK8T8hNMIL+fbGcvE2fwIO
-----END PUBLIC KEY-----
PEM
openssl dgst -sha384 -verify nitro-sandboxes-runtime-v1.pub -signature SHA256SUM.sig SHA256SUM && sha256sum -c SHA256SUM --ignore-missing
tar -xzf nitro-sandboxes-runtime.tar.gz 2>/dev/null; GP=$(find /tmp -name eks-nitro-sandbox-genpolicy -type f | head -1); chmod +x $GP; $GP --version 2>&1 | head -1
$GP --profile confidential --with-pod-identity --image-architecture arm64 /work/manifest.json > policy.b64 2> gp.err; echo "exit=$? bytes=$(wc -c < policy.b64)"; cat gp.err | tail -15
echo POLICY_BEGIN; fold -w 4000 policy.b64; echo; echo POLICY_END
sleep infinity
resources:
requests: {cpu: 500m, memory: 512Mi}
volumeMounts:
- {name: manifest, mountPath: /work}
volumes:
- name: manifest
configMap:
name: confidential-manifest
--with-pod-identity matters: without it, the docs say, the token file Pod Identity needs isn’t there in a pod that carries a policy. --image-architecture arm64 matches the node.
My first attempt, in the exploration run and before I’d added enableServiceLinks: false, failed:
error: policy generation failed: manifest does not set enableServiceLinks: false. Kubelet will inject unpredictable service discovery env vars that break exact env matching.
The tool offered three ways out: set enableServiceLinks: false, pass --allow-additional-env, or use a profile with pin_env: false. The last two loosen what the policy pins, so I took the first. With it, the recorded run’s generator pod printed:
Verified OK
nitro-sandboxes-runtime.tar.gz: OK
eks-nitro-sandbox-genpolicy 0.1.4274.0
exit=0 bytes=28151
info: fetching image configs from the registry to resolve pin_image, pin_working_dir, pin_identity, pin_args, pin_env (pulling layers for pin_identity)
info: fetching image config for public.ecr.aws/amazonlinux/amazonlinux:2023
What the policy says. Unlike on ECS, where I ran the tool twice and decoded the gzip myself, one run printed everything to stdout: 28,151 bytes (28,155 in the first run) in five sections, --- Admission Policy --- and --- Runtime Policy --- as readable Cedar, --- Base64 Encoded Policy Envelope --- (8,240 characters, the annotation value; 8,244 in the first run), --- Predicted PCR17 --- and --- KMS Key Policy Condition ---. Apart from the ciphertext in SECRET_CT, the readable Cedar was the same in both runs.
- Admission has one permit for container
attest. It pins the image digest (amazonlinux@sha256:5e98730d…), the working directory,run_as_user 0,run_as_group 0, the extra group IDs,read_only_rootfs, the effective capability list, every mount (including the service account and Pod Identity tokens), the hostname, the exact command, every declared environment value, plus the exact list of variable names, including those the kubelet and the Pod Identity webhook add. The eightKUBERNETES_*service variables are pinned by name only, andAWS_CONTAINER_CREDENTIALS_FULL_URIto one of two endpoints, IPv4 or IPv6. One EKS-specificforbidrefuses the ECS exec annotation. - HARDEN hints sit at the top as comments, one per setting I could tighten in the pod spec, for example “set securityContext.readOnlyRootFilesystem: true to enforce read-only root filesystem in the policy”. My pod ran as root with the default capabilities, and the hints say so.
- Runtime has no permit for exec or stdin (“Blocks: kubectl exec, kubectl cp”), denies
SetupVolumebecause there are no CSI volumes, and permits logs, stats, OOM events and file writes to/etc/hosts, the termination log and the two token volumes. ItsSetupSandboxpermit pins the hostname and requires the encryptedintegrity-memoryoverlay.
The generated policy, abridged
--- Admission Policy ---
// Container: attest
// HARDEN: set securityContext.readOnlyRootFilesystem: true to enforce read-only root filesystem in the policy
// HARDEN: set securityContext.allowPrivilegeEscalation: false to enforce no_new_privileges in the policy
// HARDEN: set securityContext.runAsNonRoot: true to enforce a non-root user in the policy
// HARDEN: set securityContext.capabilities.drop: ["ALL"] to restrict capabilities in the policy
permit(
principal,
action == Agent::Action::"CreateContainer",
resource == Workload::Container::"attest"
)
when {
context.image_ref == "public.ecr.aws/amazonlinux/amazonlinux@sha256:5e98730d73190367a4d29c086f2ed2911cc5867eff5aa8b2a19cbb884e9d601f" &&
context.working_dir == "/" &&
context.run_as_user == 0 &&
context.run_as_group == 0 &&
context.additional_gids == [0] &&
context.read_only_rootfs == false &&
// … effective, bounding and permitted capabilities: the 14 container defaults
// … masked and read-only paths, and every mount, including
// /var/run/secrets/kubernetes.io/serviceaccount and
// /var/run/secrets/pods.eks.amazonaws.com/serviceaccount
context.hostname == "attest-confidential" &&
// … OS flags, hooks, devices, seccomp, SELinux and AppArmor fields
context.process_args == "[\"bash\",\"-c\",\"dnf install -y -q openssl awscli-2 python3 …\"]" &&
// Guest networking: the VM owns the network stack, so no network namespace
// exists in the OCI spec. The pod hostNetwork setting has no effect.
context.host_network == true &&
context.host_pid == false &&
context.host_ipc == false &&
context.shared_pid == false &&
context.linux_namespace_types == ["pid", "ipc", "uts", "mnt", "cgroup"] &&
context.sysctls.isEmpty() &&
context.env.containsAll([
{"name": "PATH",
"value": "/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"},
{"name": "NSM_PY_B64", "value": "<...>"},
{"name": "SECRET_CT", "value": "<...>"},
{"name": "AWS_REGION", "value": "us-east-1"},
{"name": "AWS_DEFAULT_REGION", "value": "us-east-1"},
{"name": "AWS_STS_REGIONAL_ENDPOINTS",
"value": "regional"},
{"name": "AWS_CONTAINER_AUTHORIZATION_TOKEN_FILE",
"value": "/var/run/secrets/pods.eks.amazonaws.com/serviceaccount/eks-pod-identity-token"}]) &&
context.env_names == ["PATH", "NSM_PY_B64", "SECRET_CT", "AWS_REGION",
"AWS_DEFAULT_REGION", "AWS_STS_REGIONAL_ENDPOINTS",
"AWS_CONTAINER_AUTHORIZATION_TOKEN_FILE",
"KUBERNETES_SERVICE_HOST", "KUBERNETES_SERVICE_PORT",
"KUBERNETES_SERVICE_PORT_HTTPS", "KUBERNETES_PORT",
"KUBERNETES_PORT_443_TCP",
"KUBERNETES_PORT_443_TCP_PROTO",
"KUBERNETES_PORT_443_TCP_PORT",
"KUBERNETES_PORT_443_TCP_ADDR",
"AWS_CONTAINER_CREDENTIALS_FULL_URI"] &&
(context.env.contains({"name": "AWS_CONTAINER_CREDENTIALS_FULL_URI", "value": "http://169.254.170.23/v1/credentials"}) ||
context.env.contains({"name": "AWS_CONTAINER_CREDENTIALS_FULL_URI", "value": "http://[fd00:ec2::23]/v1/credentials"}))
};
// Admission: the ECS execute-command annotation (sandbox.amazonaws.com/ecs-exec-enabled) is
// forbidden on EKS. It drives SSM binary mount injection in the
// guest agent, which EKS never uses.
forbid(
principal,
action == Agent::Action::"CreateContainer",
resource
)
when {
context.ecs_exec_enabled
};
// PrepareImage: public.ecr.aws/amazonlinux/amazonlinux@sha256:5e98730d73190367a4d29c086f2ed2911cc5867eff5aa8b2a19cbb884e9d601f
// … one permit pinning the same digest
// Admission: host image mounting is disabled (no permit policy)
--- Runtime Policy ---
// Runtime: stdin writes are denied (no permit policy)
// Blocks: kubectl attach -i, kubectl exec -i, kubectl cp (to pod)
// Runtime: exec is denied (no permit policy)
// Blocks: kubectl exec, kubectl cp
// Runtime: SetupVolume is denied (no CSI volumes)
// … permits for ReadLogs, ReadTerminationLog, Stats, InitGuestMetadata,
// UpdateGuestMetadata, and WriteFile to /etc/hosts, the termination log
// and the two projected token volumes
// Runtime: allow sandbox setup
permit(
principal,
action == Agent::Action::"SetupSandbox",
resource
)
when {
context.sysctls.isEmpty() &&
context.fsgroup == -1 &&
context.host_entries.isEmpty() &&
context.deny_all_egress == false &&
context.hostname == "attest-confidential" &&
// the ephemeral overlay must use the required encryption mode
context.ephemeral_layer == Workload::EphemeralLayer::"integrity-memory" &&
// the agent creates exactly the pod's mounted emptyDir volumes
context.volume_paths.isEmpty()
};
// … permits for WatchOomEvents, RemoveContainer, StreamAgentLogs,
// and CreateTask with no binary logger
--- Base64 Encoded Policy Envelope ---
H4sIAAAAAAAC/+08aXfiSJJ/Res3r8c1XZjL2Iaa6rfcyBiwuaHdq6cjkdLo… (8,240 characters)
--- Predicted PCR17 ---
5ccb2d706da3b61f6ecbb85799cc24c54fddc841b2b259d0a67f732dc9ace02212cba02e6a3ad441c284eb4b1da3b5da
--- KMS Key Policy Condition ---
"Condition": {
"StringEqualsIgnoreCase": {
"kms:RecipientAttestation:PCR17": "5ccb2d706da3b61f6ecbb85799cc24c54fddc841b2b259d0a67f732dc9ace02212cba02e6a3ad441c284eb4b1da3b5da",
"kms:RecipientAttestation:PCR8:ServiceName": "eks.amazonaws.com"
}
}
Pinning PCR17. The key policy gives the Pod Identity role kms:Decrypt only when the attestation document’s PCR17 is the predicted one and the guest image was signed for EKS. PCR8 is “the measurement of the certificate that signed the guest image (the EIF)”, and the confidential setup page says to compare its ServiceName rather than the raw value, and that “Gating on PCR17 alone is not sufficient.” The PCR17 below is the recorded run’s; the first run’s was 11d02790…2de1313, because each run had its own key, so its own ciphertext in the pinned environment.
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "AccountAdmin",
"Effect": "Allow",
"Principal": { "AWS": "arn:aws:iam::<ACCOUNT_ID>:root" },
"Action": "kms:*",
"Resource": "*"
},
{
"Sid": "DecryptOnlyFromAttestedConfidentialPod",
"Effect": "Allow",
"Principal": { "AWS": "arn:aws:iam::<ACCOUNT_ID>:role/my2cents-nitro-eks-attest" },
"Action": "kms:Decrypt",
"Resource": "*",
"Condition": {
"StringEqualsIgnoreCase": {
"kms:RecipientAttestation:PCR17": "5ccb2d706da3b61f6ecbb85799cc24c54fddc841b2b259d0a67f732dc9ace02212cba02e6a3ad441c284eb4b1da3b5da",
"kms:RecipientAttestation:PCR8:ServiceName": "eks.amazonaws.com"
}
}
}
]
}
This is enough for the test, which is about the pod’s role. It isn’t a production key policy: the AccountAdmin statement delegates to IAM, so any IAM principal in the account with an identity policy allowing it can call a plain kms:Decrypt, with no attestation. AWS’s blueprints replace it with a narrower administration statement and add two explicit Deny statements on Principal: "*", one per register, so nothing decrypts without a matching attestation.
Then I put the key policy in place, copied the envelope section from the logs into eks-policy.b64, and spliced it into the manifest as the annotation:
aws kms put-key-policy --key-id "$KEY_ID" --policy-name default --policy file://kms-eks-final.json
jq --rawfile p eks-policy.b64 '.metadata.annotations["sandbox.amazonaws.com/policy"] = $p' \
pod-confidential-nopolicy.json > pod-confidential.json
kubectl apply -f pod-confidential.json
The policy has to come from the final spec. AWS’s blueprints note that the annotation is immutable on a running pod, so a change means deleting and recreating the pod.
Test 7: Confidential, isolated and tampered
I applied the confidential pod with its policy and the same manifest as attest-isolated, with runtimeClassName: isolated-sandbox and no annotation, together; both landed on the same node:
kubectl apply -f pod-confidential.json -f pod-attest-isolated.json
kubectl logs attest-confidential
kubectl logs attest-isolated
| Confidential, with its policy | Same pod, isolated | |
|---|---|---|
| Status | Running | Running |
/dev/nsm and attestation document | yes, 4,809 bytes | yes, 4,808 bytes |
| PCR0 (guest image) | 2be43455…68f83c4 | same |
| PCR8 (signer) | fdb0f814…6ad3698 | same |
| PCR17 (policy) | 5ccb2d70…1da3b5da, the predicted value (first run: 11d02790…2de1313, also the predicted value) | 8d833c2f…aa08dc, in both runs |
| KMS decrypt via Pod Identity | SECRET: eks: only a measured pod can read this | AccessDeniedException ... not authorized to perform: kms:Decrypt |
kubectl exec ... uname -r | denied by the policy | 6.18.54 |
The recorded run printed only PCR17 and the KMS result, so the document sizes, PCR0 and PCR8 are from the first run. The recorded logs, cut to the lines that matter:
# kubectl logs attest-confidential
PCR17: 5ccb2d706da3b61f…
SECRET: eks: only a measured pod can read this
# kubectl logs attest-isolated
PCR17: 8d833c2f8409829a…
aws: [ERROR]: An error occurred (AccessDeniedException) when calling the Decrypt operation: is not authorized to perform
And kubectl exec into each:
$ kubectl exec attest-isolated -- uname -r
6.18.54
$ kubectl exec attest-confidential -- uname -r
… policy denied Agent::Action::"Exec" on container "attest" …
The recorded run printed the denial from policy denied on; in the first run the message began error executing command in container: failed to create exec "…".
Tampering, two ways. In the recorded run I deleted attest-confidential and created a pod with the same name and the same policy annotation, changing only the command: echo tampered; in front of it. In the first run the tampered pod had also been renamed, to attest-tampered.
kubectl delete pod attest-confidential --wait=true
jq '.spec.containers[0].command[2] |= "echo tampered; " + .' pod-confidential.json > pod-tampered.json
kubectl apply -f pod-tampered.json # metadata.name is still attest-confidential
kubectl get events --field-selector involvedObject.name=attest-confidential
| Tampered pod | Status | Event |
|---|---|---|
| Same name and policy, command changed (recorded run) | CrashLoopBackOff | policy denied Agent::Action::"CreateContainer" |
Renamed to attest-tampered, command changed (first run) | ContainerCreating | FailedCreatePodSandBox ... guest sandbox setup failed: rpc status: Status { code: PERMISSION_DENIED, message: "policy denied SetupSandbox" |
What the tables show:
- PCR17 is predictable. In both runs, the value the generator printed before the pod existed is the value the running pod reported, so the key policy can be written before the first deployment, as on ECS.
- Isolated pods can attest too, and KMS can tell them apart. The isolated pod had an NSM, the same guest image (PCR0) and the same signer (PCR8). Of the PCRs I printed (PCR0, PCR8, PCR17), only PCR17 differed, and that’s what KMS refused on. Its PCR17,
8d833c2f…aa08dc, was the same in both runs and identical to the ECS isolated task’s in part 3, while PCR0 and PCR8 differ between EKS and ECS (separately signed guest images). So that value looks like the measurement of the built-in permissive policy, the same on both services. - Exec is a policy decision.
kubectl execworked into the isolated pod and was refused by the guest’s agent in the confidential one, as the runtime policy said. ECS draws the line earlier: in part 3, ECS Exec worked into an isolated task, but a confidential task started with ECS Exec turned on, against a policy generated without--allow-exec-command, never started. Here the confidential pod ran, and only the exec call was refused. - A changed command is refused at
CreateContainer, as on ECS. With the name unchanged, the agent set up the sandbox and then refused to create the container. The first run’sSetupSandboxdenial, one step earlier, came from the rename: the generatedSetupSandboxpermit pinscontext.hostname == "attest-confidential", and of the fields it checks, the hostname is the only one the rename changed. Neither tampered pod ran, and neither failed outright: like thehostNetworkpod, both kept retrying, so watch for the event, not for aFailedphase.
The cost is churn. The policy pins the environment values, including the ciphertext, the image digest and the pod’s name, so re-encrypting the secret, editing the NSM client, moving to a new image or renaming the pod each means a new policy, a new PCR17 and a key policy update. The two runs show it: a new key and ciphertext, nothing else, gave a new PCR17.
As on EC2 and ECS, the attestation document’s module_id starts with the parent instance’s ID, here the first run’s node: i-0956c7c6b4bd96b04-enc01a1c2f63a2c8aa2.
What I didn’t test: the default throttles, which the docs put at 100 Mbps and 1,500 packets per second each way, 100 DNS queries per second and 4,096 tracked connections, and say to raise before you benchmark; EBS and EFS volumes; encrypted images; and AWS Signer verification.
Results summary
| Question | Answer from my runs |
|---|---|
| Cluster-level switch | None: Kubernetes 1.36 or later on Auto Mode, in a US commercial Region per the overview; no module, provider or API argument |
| What Auto Mode ships | Both RuntimeClasses before any node: 256Mi or 512Mi overhead plus one eks.amazonaws.com/nitro-sandbox, a nodeSelector, no toleration |
| Graviton | Works, with your own NodePool, because general-purpose is amd64 only |
| Pod opt-in | runtimeClassName: isolated-sandbox |
| First pod, including the node | 50 s to Ready in the recorded run, 85 s in the first |
| Warm start | 4.4 to 5.4 s, against 1.7 to 2.1 s for a plain pod, across both runs |
| What a sandboxed pod sees | Its own kernel (6.18.54), 1 vCPU and memory from its requests, virtio disks, an NSM device |
| Metadata | Surrogate answered Region and AZ; instance ID and credentials 404 |
| Service account token in isolated mode | Mounted, although the docs say it isn’t |
| Host-level settings | hostNetwork stuck in ContainerCreating; hostPID sees the VM’s systemd |
| Density | The node advertises 20 sandbox slots (5 × its 4 vCPUs); 18 to 20 pods ran in four runs of the same Deployment, the rest Pending on Insufficient eks.amazonaws.com/nitro-sandbox |
| Confidential mode | PCR17 matched the prediction in both runs; KMS released the secret through Pod Identity only to that pod; exec denied; a changed command refused at CreateContainer (CrashLoopBackOff), a renamed pod at SetupSandbox (ContainerCreating) |
| Policy generation | Ran in a plain pod on the cluster as a demo shortcut; in production, generate it in CI or on a Linux workstation, off the node. Needs enableServiceLinks: false (or one of the two looser alternatives the tool offers) |
| Monitoring | kubectl top pod and EC2 CPUUtilization see a busy sandbox; kubectl top node and Container Insights don’t, and Container Insights has no records for sandboxed pods |
| CloudTrail | The attested Decrypt carries PCR17 in base64 and PCR8ServiceName: eks.amazonaws.com |
Operating and Monitoring
Watch this part: the video’s operations section, from 5:24, walks through all of it in the console, the CLI and kubectl.
On 2026-11-10 I ran a third pass, for the video’s operations section, from scratch in the console with the CLI and kubectl alongside: a VPC from the console’s VPC wizard, the cluster (see Step 2), the same m8g.xlarge NodePool with a two-replica Deployment of isolated-sandbox pods, the CloudWatch Observability and Metrics Server add-ons, Test 6’s confidential pod again, and the teardown. Both pods were running 59 to 71 seconds after kubectl apply, node launch included.
Where the sandbox shows. The cluster’s Compute tab lists the node and both node pools, system as “Built-in” and sandbox-m8g as “Custom”. The node page’s capacity allocation counts each pod’s memory with its 256Mi overhead (1 GiB reserved for two pods requesting 256Mi), but not the sandbox slots. The pod page’s structured view doesn’t mention the sandbox at all; only its Raw view shows runtimeClassName, the overhead and the injected nodeSelector.
Managing them is plain kubectl. Logs and exec work, and exec lands in the guest (kernel 6.18.54, one CPU). Scaling to four booted two more VMs, rollout restart replaced every pod and its VM, and scaling to zero removed them; there’s no pause. One gotcha: a bash loop as PID 1 ignored SIGTERM, so pods waited out the 30-second grace period until I added trap 'exit 0' TERM.
While one sandboxed pod burned a full vCPU:
| Signal | What it showed |
|---|---|
kubectl top pod (Metrics Server) | 984m: it sees the sandbox |
kubectl top node | 29m |
| Container Insights, node CPU | 0.8% |
| Container Insights, pod and container records | None for the sandboxed pods, only for the add-ons’ five containers, over 10 minutes and more; the node’s running-pod count did include them |
EC2 CPUUtilization of the node | 26%, one of four vCPUs |
| Application logs | Each sandboxed pod’s stdout reached /aws/containerinsights/<cluster>/application through Fluent Bit |
The overview says the parent instance “cannot observe the CPU or memory state of the sandbox”, and the node, with everything that reads it, didn’t. kubectl top did, as the observability page promises, once Metrics Server was installed; Auto Mode doesn’t install it, though Custom configuration pre-selects it. And the EC2 metric did, because it’s “the sum of guest CPUUtilization and hypervisor CPUUtilization”. The same observability page says the CloudWatch agent “can collect container logs and metrics from sandboxed pods”: logs, yes; metrics, not in Container Insights’ pod records on this setup.
CloudTrail. The CreateCluster record shows the console session and computeConfig.nodePools: ["system"]. The confidential pod’s KMS Decrypt comes from its Pod Identity role, with a recipient block holding the module ID (the node’s instance ID plus -enc…), the image digest, PCR1 to PCR4, PCR8, PCR17 and attestationDocumentEnclavePCR8ServiceName: "eks.amazonaws.com". PCR17 is in base64, which the KMS CloudTrail page doesn’t say; decoded, it equalled the generator’s prediction in both runs. The record was in lookup-events within about 40 seconds to 2 minutes.
Pricing
Nitro Sandboxes have no additional charge. On EKS you pay what an Auto Mode cluster costs anyway. In us-east-1:
| Item | Price |
|---|---|
| EKS control plane | $0.10 per cluster-hour |
m8g.xlarge on demand | $0.17952 per hour |
Auto Mode management fee for m8g.xlarge | $0.02154 per hour |
| One sandbox node, control plane included | $0.30106 per hour |
The NAT gateway in this VPC adds about $0.045 an hour plus data, and the KMS key $1 a month until you schedule its deletion. The node costs the same whether it holds zero sandboxes or twenty, so the real cost shows up as capacity:
- Fixed memory overhead per pod. The RuntimeClass adds 256Mi to every isolated pod’s requests and 512Mi to every confidential one’s. On a 512Mi pod that’s half again in isolated mode and double in confidential mode, as far as the scheduler is concerned. On ECS in part 3 I measured an actual overhead that grew with task size, from 262 to 509 MiB; on EKS the number is fixed, and I didn’t measure what the node actually spends.
- The 20-sandbox cap on an
m8g.xlarge. For small pods, the count binds long before memory. Twenty of my 50m / 64Mi density pods would reserve 20 × (64 + 256) = 6,400 Mi of the node’s 13,513 Mi allocatable, and the remaining memory can only go to plain pods. And in my runs only 18 to 20 of the 20 slots were usable (Test 5). - Per pod at 1 vCPU. By the node’s allocatable 3,770m of CPU, at most three pods requesting 1 vCPU fit on an
m8g.xlarge: about $0.067 an hour each with the Auto Mode fee, before the control plane. A Lambda MicroVM with 1 vCPU and 2 GB costs $0.1261 an hour, but only while it runs; a node bills every hour it’s up. Auto Mode does consolidate sandbox nodes like any other (Test 1), so an emptied node needn’t sit idle.
My exploration ran from 00:12 to about 01:10 UTC, and 13 minutes 42 seconds of that was terraform apply. The recorded run kept a second cluster up from 02:08 to about 06:30 UTC, after another apply of about 13 minutes.
Cleanup
Delete the workloads and the NodePool first, so Auto Mode terminates the node, then the Pod Identity association, the key and the infrastructure:
kubectl delete deployment density --ignore-not-found
kubectl delete pod --all -n default
kubectl delete nodepool sandbox-m8g
ASSOC=$(aws eks list-pod-identity-associations --cluster-name my2cents-nitro-eks \
--query 'associations[0].associationId' --output text)
aws eks delete-pod-identity-association --cluster-name my2cents-nitro-eks --association-id "$ASSOC"
aws kms schedule-key-deletion --key-id <key-id> --pending-window-in-days 7 # key ID, not alias
terraform destroy
aws iam delete-role --role-name my2cents-nitro-eks-attest
In the recorded run, the key went to PendingDeletion and terraform destroy ended with Destroy complete! Resources: 54 destroyed.; I deleted the role separately. As in parts 2 and 3, schedule-key-deletion wants the key ID, not the alias, and seven days is the shortest waiting period. The IAM role was created with the CLI, not Terraform, so terraform destroy leaves it behind; that’s what the last line is for.
Verdict
What works
On Auto Mode, the opt-in is as small as promised. Both RuntimeClasses were there before the first node, an ordinary NodePool with no sandbox setting produced a sandbox-capable Graviton node, and the pod needed one field. A sandboxed pod is visibly its own machine, with its own kernel, a VM sized from its requests, virtio disks and an NSM device, while a plain pod on the same node sees all of it. The metadata surrogate behaved exactly as documented, which is more than I could say on ECS. hostPID is confined to the VM, network isolation removes the network, and exec, logs and Pod Identity work unchanged in isolated mode. Day to day, sandboxed pods scaled, restarted and logged to CloudWatch like any others, and CloudTrail recorded each attested decrypt.
Confidential mode is as complete as it was on ECS, with Kubernetes ergonomics. The generator predicted PCR17 exactly and printed the policy as readable Cedar with hints for tightening it. For this demo it ran in a pod on the cluster, but in production it belongs in the CI/CD pipeline that produces your manifests, off the node that confidential mode treats as untrusted. KMS released the secret only to the pod carrying that policy, through Pod Identity with no add-on, refused the same pod in isolated mode, the guest refused kubectl exec, and a copy with only its command changed was refused at CreateContainer, the same stage as on ECS. Both runs agreed on all of it.
What doesn’t, yet
The docs and the launch blog trail the product in two places that matter for security reviews. The Auto Mode RuntimeClass has no toleration and the sandbox node no taint, contradicting the launch blog, so plain pods share sandbox nodes unless you arrange otherwise. And an isolated pod got the service account token that the overview says it doesn’t.
Monitoring needs a second look too. Container Insights had no pod or container records for the sandboxed pods, and the node’s CPU stayed flat while one of them burned a full vCPU. What saw it was kubectl top, with a Metrics Server that Auto Mode doesn’t install by default, and the EC2 metric for the node.
Failure modes are quiet. A hostNetwork pod is accepted and then sits in ContainerCreating, and a confidential pod with a changed command goes into CrashLoopBackOff (a renamed one, into ContainerCreating); all of them retry instead of failing. ECS refused the host-network task definition at registration, and its tampered task registered, then stopped with TaskFailedToStart instead of retrying. Density tops out at the 20 slots an m8g.xlarge advertises (5 × its 4 vCPUs), and I couldn’t count on all of them: 18 to 20 pods ran in four runs of the same Deployment, fewer right after deletions, and a burst on a fresh node got two pods rejected by the kubelet. I didn’t find out why. The fixed 256Mi overhead, 512Mi in confidential mode, makes small pods expensive in memory. The built-in general-purpose pool is amd64 only, so Graviton needs your own NodePool. And confidential mode adds a policy loop: a regenerated policy and key policy for every change to the image, the command, the environment or the pod’s name, from a generator that should run off the node. The generator needs enableServiceLinks: false (or one of the two looser alternatives the tool offers). A single-line command and both Region variables are requirements I carried over from ECS and AWS’s blueprints; I didn’t test either on EKS.
Who should use it
Kubernetes teams that run container images they didn’t write, per pod: coding agents, CI runners, per-tenant workers. On Auto Mode it’s close to zero setup, and pods keep everything Kubernetes gives them, including Pod Identity, the VPC CNI, logs and exec. Confidential mode is for pods that must prove what they run before they get a key, and like ECS, the measurement comes from a documented tool rather than from inside a running guest. It fits best for pods of 1 GiB and up, where the 256Mi overhead is a quarter or less, with headroom below the 20 slots a node advertises.
Skip it for pods that need hostNetwork, GPUs or in-place resize, and for packing hundreds of tiny pods per node. If you’re on ECS, part 3 is the same feature with a task definition field. If you run your own orchestrator, the raw primitive on EC2 (part 2) gives you more control and more work. If you’d rather not run nodes at all, AWS Lambda MicroVMs bill per second while running and suspend with memory intact, but document no attestation.