OpenShell on Amazon EKS: Hands-On with a Self-Hosted Install

Overview

OpenShell, accepted as a CNCF Sandbox project on 2026-09-26, runs AI agents in sandboxes split into an untrusted workload and a trusted supervisor. Part 1 explained the pieces and ran them on a Mac. This post, part 2 of 6, installs OpenShell 0.1.6 on Amazon EKS 1.37, which is what part 3, part 4, part 5 and part 6 run on.

Upfront: OpenShell doesn’t test EKS. Its CI runs pull requests on kind with Kubernetes 1.35 and releases on k3s 1.29, and its release policy’s build matrix names GKE and OpenShift as supported targets. So everything below, including each workaround, is my own evidence, not a vendor statement.

Before building anything on top of the install, I set three go/no-go checks:

1. Landlock active in the node kernel. OpenShell’s sandbox runtime refuses to start without it.
2. The control path: each sandbox’s workload Pod has to reach its supervisor Pod through the VPC CNI’s NetworkPolicy enforcement.
3. VM isolation: an OpenShell sandbox running in a Kata Containers VM on ordinary EC2 instances with nested virtualization.

The first two passed. The third didn’t, and that shaped the rest of the series.

Architecture

What EKS needs that the defaults don’t give it:

What	Default	What I did
NetworkPolicy enforcement	VPC CNI accepts NetworkPolicies but doesn’t enforce them	enableNetworkPolicy: "true" in the vpc-cni add-on
Agent Sandbox controller	Not installed; the chart’s preflight fails without it	sandbox.yaml from release v1.0.3 (the version OpenShell’s pull-request CI tests; the docs say “latest”, v1.0.9 on 2026-11-01)
Images	GHCR, nvcr.io and registry.k8s.io, through one NAT gateway	All six in ECR (five copied, one built); global.image.registry and sandbox.image in the values
A default StorageClass	EKS has none	The EBS CSI add-on’s defaultStorageClass (gp3)
Gateway resources	resources: {} (BestEffort)	250m CPU, 512 MiB requested
Telemetry	On	server.telemetryEnabled: false
Exposing the gateway	The chart’s Service takes no annotations or source ranges, and also publishes the unauthenticated metrics port 9090	Port-forward by default; our own NLB Service for one test
Certificate names	The PKI hook never re-issues; SANs cover only in-cluster, loopback and Docker-host names	Create the NLB first, put its hostname in pkiInitJob.serverDnsNames
CLI registration	openshell gateway add --local copies a local gateway’s certificates, if the laptop has one	Register first, then write the cluster’s client bundle

Setup

The companion directory builds everything with Terraform and nine numbered scripts. Each one is meant to be safe to run again; I re-ran 01 to 06, and each converged instead of failing.

Step 1: Tools

Terraform 1.9 or later, the AWS CLI v2, kubectl, Helm (I used 4.3.1; the docs say 3.x, and the chart installed fine), Docker with buildx, jq, envsubst and uv, plus the OpenShell CLI from part 1. No secrets: the gateway’s PKI is generated inside the cluster.

Step 2: The cluster

01_cluster.sh runs Terraform (one VPC with one NAT gateway, EKS 1.37, two m7i.2xlarge nodes), installs the AWS Load Balancer Controller, and checks the result. Two settings matter for OpenShell:

vpc-cni = {
  addon_version        = "v1.23.2-eksbuild.8"
  before_compute       = true
  configuration_values = jsonencode({ enableNetworkPolicy = "true" })
}

cpu_options = {
  nested_virtualization = "enabled"   # /dev/kvm on M7i, for Kata
}
metadata_options = {
  http_tokens                 = "required"
  http_put_response_hop_limit = 1     # no node credentials for pods
}

The whole thing took 13 minutes 30 seconds, 12:48 of it Terraform.

Step 3: Images to ECR

03_images.sh copies five images with their multi-architecture indexes and builds a sixth. It then compares every digest in ECR with its source:

Image	Source
openshell/gateway:0.1.6, supervisor:0.1.6, sandbox:0.1.6	ghcr.io/nvidia/openshell/* (tags have no v)
agent-sandbox/agent-sandbox-controller:v1.0.3	registry.k8s.io
base/ubuntu:24.04	nvcr.io/nvidia/base/ubuntu, pinned by digest
workload:0.1.6	built here: the base plus the AWS CLI 2.39.3, boto3 1.43.123, git, gh, curl, jq

It took 2 minutes 34 seconds. When I deleted one tag, --verify-only exited 1 and named the missing image. A re-run copied only that image, and the workload image rebuilt from cache to the same digest.

Step 4: Agent Sandbox and the chart

05_install.sh installs Agent Sandbox v1.0.3 with its image rewritten to ECR. It then labels namespace openshell with Pod Security restricted, creates the NLB Service (so its hostname exists), and runs Helm with this values file:

global:
  image:
    registry: ${ECR_REGISTRY}/${ECR_PREFIX}
    tag: "0.1.6"
sandbox:
  image:
    repository: ${ECR_REGISTRY}/${ECR_PREFIX}/workload
    tag: "0.1.6"
server:
  telemetryEnabled: false
  drivers:
    kubernetes:
      allowDriverConfig: true   # lets a sandbox pick a RuntimeClass
resources:
  requests:
    cpu: 250m
    memory: 512Mi

The script passes the NLB hostname with --set-json pkiInitJob.serverDnsNames=[...], starts a port-forward on 127.0.0.1:17680, registers the CLI and then writes the client bundle.

Testing

Test 1: The cluster meets the bar

01_cluster.sh ends with checks, and every one passed:

ok    Kubernetes v1.37.1-eks-577a51e
ok    ip-10-70-10-204.ec2.internal: crw-rw-rw-. 1 root 36 10, 232 ... /host/dev/kvm (vmx on 8 vCPUs)
ok    ip-10-70-10-204.ec2.internal: kernel 6.18.56-125.164.amzn2023.x86_64, Landlock active (ABI 7),
      LSMs lockdown,capability,landlock,yama,safesetid,selinux,bpf,ima
ok    i-0da86c3ab4ddf08b1 (m7i.2xlarge): HttpTokens required, HttpPutResponseHopLimit 1, NestedVirtualization enabled
ok    VPC CNI network policy enforcement on (aws-node 2/2 ready, aws-eks-nodeagent present)
ok    eks-pod-identity-agent DaemonSet 2/2 ready
ok    aws-load-balancer-controller 2/2 ready, credentials from EKS Pod Identity
ok    default StorageClass ebs-csi-default-sc (ebs.csi.aws.com, ...)

Gate 1 passed. Landlock is in the active LSM list of the EKS AL2023 kernel, at ABI 7; OpenShell needs 3. The node check reads the ABI with a direct landlock_create_ruleset call from a debug pod. A second run of the script reported Terraform “No changes”.

Test 2: The fence before OpenShell

OpenShell’s Kubernetes driver puts every workload Pod behind a NetworkPolicy with no egress except to its supervisor, and trusts the CNI to enforce it (#3862 asks for a check). So before installing OpenShell, 04_fence.sh tested what the VPC CNI enforces, with plain pods and the same kind of policy:

Check	Result
Pod under deny-all egress vs no policy	Deny-all: internet and cluster DNS blocked. No policy: both open
New pods in standard mode, 10 sharing a name prefix	1 of 10 got requests out before the policy applied (10 of its 40 requests; each is a curl with a 1 s timeout, then a 100 ms pause)
New pods in standard mode, 10 with unique name prefixes	10 of 10 got 9 or 10 requests out first
Fenced pod to its own node’s IP, port 18080	Open
Fenced pod to the other node’s IP	Blocked
Strict mode (NETWORK_POLICY_ENFORCING_MODE=strict)	0 of 10 leaked; a new pod no policy selects starts with all traffic denied

Three things to take from that:

• There’s a startup gap of a second or a little more in the default mode. The network policy agent programs a new pod’s rules after the pod has started. The VPC CNI agent shares those rules between pods whose names match up to the last -. So only a pod with a new prefix opens a gap: here that was the first of ten sharing a prefix, and all ten with unique prefixes. OpenShell names workload Pods <workspace>--<sandbox>, so sandboxes with hyphen-free names in workspace default all share the prefix default-, while a name like warm-1 gets its own prefix, and its own gap.
• The node’s own IP is always reachable. That’s deliberate in the VPC CNI’s network policy agent: its source adds the node’s IP to every allow list so that kubelet probes work. A fenced pod can reach every port on its own node.
• Strict mode closes the gap, but any pod no policy selects then starts with all traffic denied, system pods like CoreDNS included (host-network pods are exempt), so the script turned it on, measured, and put it back. Each switch took 23 seconds.

How much does the gap matter for OpenShell? Less than for a plain pod. OpenShell holds the agent’s launch until the supervisor confirms, and the agent’s sockets go through the in-pod broker either way. The NetworkPolicy is the second layer, not the first. But it’s the layer that catches anything the broker misses, so I’d run strict mode on a dedicated sandbox cluster.

Test 3: Agent Sandbox and the preflight

Installing the chart before Agent Sandbox fails before anything is created:

Error: execution error at (helm-chart/templates/agent-sandbox-preflight.yaml:9:4): Agent Sandbox is required
but neither agents.x-k8s.io/v1beta1 nor agents.x-k8s.io/v1alpha1 is served

With the v1.0.3 controller installed (CRD v1beta1), the install took 31 seconds. The NLB had a hostname 6 seconds after its Service was created.

Test 4: The gateway

openshell-0                  1/1   Running
openshell-data-openshell-0   Bound   1Gi   RWO   ebs-csi-default-sc
ok    gateway env OPENSHELL_TELEMETRY_ENABLED=false
ok    server certificate lists the NLB hostname

• Pod Security restricted held. The gateway and the certificate hook were admitted under restricted, and so were all sandbox Pods (test 6).
• The data survives. Deleting openshell-0 brought it back on the same volume.
• A re-run converges. It produced Helm revision 2 with identical manifests, and the certificate hook skipped because its three Secrets existed.
• Authentication is per call, not per connection. A TLS client without a certificate completes the handshake, and the first API call returns gRPC status 16, missing authorization header. With the client certificate from the cluster, the CLI reports Authenticated (mTLS transport).

The CLI registration hit the only real snag of the install. The docs register a Kubernetes gateway with openshell gateway add https://127.0.0.1:8080 --local. My laptop also runs the local gateway from part 1, and --local (the help text says “a local mTLS gateway running in Docker on this machine”) filled the new registration with that gateway’s certificates:

Error:   × client error (Connect)
  ╰─▶ invalid peer certificate: BadSignature

The fix is to register first, then write the cluster’s client bundle from Secret openshell-client-tls into ~/.config/openshell/gateways/osh-eks/mtls/. Port 8080 was taken on my laptop, so the port-forward uses 17680.

Test 5: The CLI through an NLB

The chart’s Service can’t take load balancer annotations, so manifests/gateway-nlb.yaml is a Service of our own: internet-facing, IP targets, TCP passthrough on port 8080 only, loadBalancerSourceRanges set to my IP. TLS passes through, so the client certificate reaches the gateway:

Gateway: osh-eks-nlb
Server: https://k8s-openshel-openshel-....elb.us-east-1.amazonaws.com:8080
Status: Connected
Authentication: Authenticated (mTLS transport)
metrics port 9090 through the NLB: 000 (not exposed)

When I set the source range to 203.0.113.7, the CLI was locked out and the script stopped with an error, though not instantly: even after the security group showed only the new address, one more status call got through before the next call was refused. A re-run put my IP back. My first version of that test passed by mistake because it ran before the AWS Load Balancer Controller had updated the security group, so the script now waits for the rule before testing. After the test, 06_remote.sh --delete removed the NLB.

Test 6: First sandboxes on EKS

Gate 2 passed. Sandboxes reach Ready, so the VPC CNI admits the workload-to-supervisor path. One sandbox is nine Kubernetes objects:

sandboxes.agents.x-k8s.io    1  default--first
pods                         2  default--first (workload), os-supervisor-<id> (supervisor)
services                     2  os-boundary-<id>, os-supervisor-<id>
networkpolicies              1  os-control-<id>
secrets                      2  os-sandbox-<id>-..., os-supervisor-<id>-...
persistentvolumeclaims       1  workspace-default--first

Plus three policies for the whole namespace: the two fences (workloads get no egress, supervisors get open egress) and the chart’s SSH ingress rule. Inside the workload:

$ id
uid=10001 gid=10001 groups=10001
$ sh -c "grep -E '^(Seccomp|NoNewPrivs|CapEff)' /proc/self/status"
CapEff:	0000000000000000
NoNewPrivs:	1
Seccomp:	2
Seccomp_filters:	6
$ sh -c "env | grep -ciE 'proxy|token'"
0

The driver runs the workload as UID 10001 whatever the image’s USER says, with no capabilities and a seccomp filter, and there’s no proxy or token variable to steal.

Timings, create call to Ready:

Create	Time
First, image removed from both nodes but its layers still cached (pull 81 ms)	16.8 s
Warm	8.3 s
Five more warm, median (fastest, slowest)	10.4 s (8.9 s, 10.7 s)
Delete, per sandbox	about 31 s

The image isn’t what costs time: the kubelet reported pulling the 175 MB workload image in 81 ms, because its layers were still on the node. The events for one warm create show where the ten seconds go:

• the EBS volume: about 2 s to provision and 2 to 3 s to attach,
• three containers starting: about 2 s,
• the supervisor session.

Locally on Docker the same create took 2.5 seconds. Deleting is slower still, because DeleteSandbox returns only once the Pods and the volume are gone. That’s just past the Python SDK’s default 30-second RPC timeout, so the companion code raises it to 120.

Test 7: Kata

kata-deploy 3.32.0 installed in 66 seconds (a 1.6 GB pull per node). A plain Kata pod started in 3 seconds with its own guest kernel, 6.18.35 against the node’s 6.18.56. OpenShell picks a runtime per sandbox, once the RuntimeClass carries an approval label:

kubectl label runtimeclass kata-qemu openshell.ai/sandbox-attachable=true
openshell sandbox create --name kata1 \
  --driver-config-json '{"kubernetes":{"pod":{"runtime_class_name":"kata-qemu"}}}'

Without the label, creation is refused at once: RuntimeClass 'kata-qemu': external resource not admitted by required labels.

With the label, gate 3 failed. The workload Pod got its Kata VM, and the first init container finished inside it. The second, workspace-init, runs openshell-sandbox seed-workspace as UID 10001 to fill the workspace volume. About 50 seconds after the create, it ended with exit code 255 and no output, and the Pod went to Failed. Kata’s runtime on the node, which logs through containerd, reported this in the previous five minutes:

3 Dead agent
1 QEMU exited with an error

Dead agent means the host had lost contact with the Kata agent, the process inside the VM that runs the containers. QEMU exited with an error (in full, signal: killed) means the VM’s process was killed. So the VM went down. These lines don’t show why, or which came first: they have no timestamps, and Kata also kills QEMU when it tears down a failed Pod. I didn’t capture the guest’s own logs, so the cause is still open. Kata isn’t in OpenShell’s CI yet either (#4459, open).

The sandbox itself stayed in Provisioning (reason ConfigurationPending) until the 300-second provisioning timeout. It failed the same way with --cpu 2 --memory 4Gi. A plain Kata pod with the same UID, the same fsGroup and an EBS volume wrote to the volume without trouble, so the failure is specific to OpenShell’s workload Pod under Kata, not to Kata, the UID or the volume on their own. I didn’t try a bare-metal node: Kata itself works on these nested-virtualization nodes, so a different host seemed unlikely to help.

So the rest of this series runs OpenShell sandboxes on runc, which shares the node’s kernel. Part 3 explains what that means.

Results Summary

	Result
Gate 1: Landlock on EKS 1.37 AL2023	Passed (kernel 6.18.56, ABI 7)
Gate 2: workload-to-supervisor path under VPC CNI	Passed
Gate 3: OpenShell sandbox in a Kata VM	Failed: workspace seeding fails inside the VM and the VM goes down; the cause is still open (plain Kata pods work)
Cluster build	13 min 30 s
Images to ECR	2 min 34 s (6 images)
OpenShell install	31 s for the chart, NLB hostname in 6 s
Sandbox create	8.3 s to 16.8 s (median of five warm: 10.4 s)
Sandbox delete	about 31 s each
Objects per sandbox	9, plus 3 namespace-wide policies
VPC CNI startup gap	a second or a little more for a pod with a new name prefix; closed by strict mode
Re-runs	Scripts 01 to 06 converged on a second run

Pricing

Idle cost of what this builds, in us-east-1 (AWS price list, October 2026):

Item	Price	Per hour	Per day
EKS control plane (standard support)	$0.10/h	$0.100	$2.40
2 × m7i.2xlarge on demand (nested virtualization costs nothing extra)	$0.4032/h each	$0.806	$19.35
NAT gateway (+ $0.045/GB processed) and its public IPv4	$0.045/h + $0.005/h	$0.050	$1.20
EBS gp3 root volumes, 2 × 100 GiB, and small change (ECR, the gateway’s 1 GiB)	$0.08/GB-month	≈ $0.023	≈ $0.55
Total idle		≈ $0.98	≈ $23.50
NLB while it exists (2 AZs, idle)	$0.0225/h + IPv4	≈ $0.033	≈ $0.78
Each running sandbox’s 2 GiB gp3 volume	$0.08/GB-month	≈ $0.0002	≈ $0.005

The nodes are 80% of it. 09_scale.sh down takes them to zero between sessions and leaves about $0.15 an hour.

Cleanup

./09_scale.sh down     # between sessions: zero nodes
./09_scale.sh up       # two nodes, gateway, port-forward
./99_teardown.sh       # everything, when you're done with the series

99_teardown.sh deletes the sandboxes and the NLB, and uninstalls the three Helm releases and Agent Sandbox. It then removes what the chart leaves behind on uninstall: its TLS, JWT and key-encryption Secrets, the gateway’s volume and any workspace volumes. Last come every openshell-eks/* ECR repository, terraform destroy, and the Container Insights log group from part 6, which its agent creates outside Terraform. It finishes by listing any EBS volume, load balancer, ECR repository, osh-eks IAM role or bucket, or Container Insights log group left behind, and fails if there is one. If you ran part 5, destroy its Terraform first: 99_teardown.sh stops before deleting anything while part 5’s roles and bucket still exist.

Verdict

OpenShell installs cleanly on EKS once you know the six or so things the chart assumes and EKS doesn’t provide: an enforcing CNI, the Agent Sandbox controller, a default StorageClass, images you can pull reliably, a certificate name for remote access, and a CLI registration that doesn’t borrow a local gateway’s certificates. Nothing OpenShell installs runs privileged (kata-deploy does, by design), everything in namespace openshell passed Pod Security restricted, and authentication is enforced on every call. For an alpha-exit release that the project doesn’t test on EKS, that’s a good result.

Three things deserve attention before you rely on it:

• The fence depends on a setting that is off by default. Without enableNetworkPolicy, the policies OpenShell creates are accepted and ignored. With it, a pod with a new name prefix still gets a startup gap of a second or so, and a fenced pod can reach its own node; strict mode on a dedicated cluster closes the gap.
• Each sandbox costs an EBS volume. That’s about 10 seconds to create, 30 seconds to delete, and one of the per-node volume attachments. Part 6 measures what that does to density.
• VM isolation isn’t there yet. Kata runs on nested-virtualization M7i nodes, but with an OpenShell 0.1.6 sandbox under Kata, workspace seeding fails inside the VM and the VM goes down; the cause is still open. If you need a separate kernel per agent today, this release on this setup doesn’t give you one.

Compared with running it on a laptop, EKS gives OpenShell what the Docker driver can’t: Pod Security admission, per-sandbox NetworkPolicies, persistent workspaces and a gateway on a real volume. It also gives you more to get right. The next post, part 3, follows a sandbox’s life on this cluster, and what isolation you get with runc alone.