OpenShell Sandbox Lifecycle and Isolation: Hands-On on Amazon EKS
By Fabio Douek
Jump to section
Explain (TLDR) like I am...
Every robot gets its own little room with a grown-up at the door. I tested what happens when the robot falls asleep, when I switch the room off and on again, and when I take the grown-up away.
The robot's toy box survives everything, but its scribbles on the whiteboard get wiped each time. Without the grown-up, the room locks itself. The robot really can't open the doors it isn't allowed to, though all the rooms still sit on the same floor. And a room nobody remembers keeps its lights on until someone turns them off.
This post records how OpenShell 0.1.6 behaves on Amazon EKS when sandboxes restart, stop, lose a component, or cannot be scheduled, and which controls apply inside a sandbox. Every result is the author's own test on his own cluster.
For due diligence: persistent data survives restarts, but memory and running processes do not; losing either half of a sandbox stops or fails it by design rather than leaving it unguarded, and only a manual start brings it back; and the agent shares the host kernel, which a published Red Hat test has shown to be a real exposure.
I examined a sandbox's life cycle on Amazon EKS: restarts, stops, the loss of either of its two Pods, and an unavailable zone, then checked each isolation layer from inside. The containment reflexes were consistent: when the supervisor dies, the patient is sedated, not left to wander.
Residual risk: the workload and the node share one kernel, so a kernel exploit bypasses every layer above it. User namespaces reduce the damage of a breakout; a separate kernel per sandbox was not available in this release on these nodes.
It helps to know exactly what you lose when something goes wrong. Here the answer is reassuringly consistent: files in the workspace stay, everything held in memory goes, and when the trusted half of a sandbox disappears, the untrusted half is stopped, not set free.
The harder conversation is about the shared kernel. Every safeguard tested here holds, but they all sit on one kernel with the node. Naming that limit early saves a team from trusting the sandbox more than it has earned.
Picture a band where every song is a new take. When a song ends, the restart policy decides whether to play it again; each new take gets a fresh mix and fresh musicians, while the recorded tracks in the workspace carry over from take to take.
If the sound engineer walks out, the band stops rather than playing unmonitored. The players can't touch gear outside their corner of the stage, but they all stand on the same stage as the house rig, and that floor is the one thing nobody has replaced yet.
The story here is predictability. A sandbox restarts, stops and starts in seconds, keeps its workspace, and fails closed when part of it goes missing. The isolation checks inside the sandbox pass one after another with clear, explainable denials.
The honest caveat buyers should hear: sandboxes share the node's kernel in this release, and a workspace volume ties each sandbox to one availability zone. Both are manageable, and both belong in the pitch rather than the fine print.
See the source code here: openshell-lifecycle
Overview
OpenShell, a CNCF Sandbox project since 2026-09-26, splits every agent sandbox into an untrusted workload and a trusted supervisor. Part 1 explained that split, and part 2 installed OpenShell 0.1.6 on Amazon EKS 1.37. This post, part 3 of 6, uses that cluster to answer two questions.
The first is about a sandbox’s life. What happens when its main process exits, when you stop and start it, when one of its two Pods disappears, and when the node its volume needs is gone? On Kubernetes, each of these touches Pods, volumes and an availability zone, and the docs cover only part of it.
The second is about isolation. From inside a sandbox, I checked what the workload can and can’t do with the default policy: files, syscalls, privileges, credentials, network, and the kernel. Then I turned on user namespaces.
All sandboxes here run on runc. Part 2’s attempt to run them in Kata VMs failed, so the workload shares the node’s kernel, and that’s the last thing this post looks at. Part 4 covers network policy and credentials, part 5 AWS credentials, and part 6 a stress test.
The Lifecycle at a Glance
On Kubernetes, a sandbox’s lifetime spans several generations. A generation is one run of a pair of Pods, with its own tokens and certificates. The workspace volume is the only thing that outlives them all:
| Event | Pods | Generation | /sandbox | /tmp, processes, memory |
|---|---|---|---|---|
Main process exits, policy never | stay running (checked after exit 3); phase Error (nonzero) or Completed (zero) | same | kept | out of reach: exec is refused until start |
Main process exits, on-failure or always | both replaced | new | kept | lost |
stop, then start | removed, then a new pair | new | kept | lost |
| Supervisor Pod deleted | both removed after about a minute | stopped | kept | lost |
| Workload Pod deleted | replacement never runs; phase Error | same until stop and start | kept | lost |
suspend | not supported in 0.1.6 | |||
delete | removed | deleted | lost |
The sandbox keeps its name and ID through all of it. If you’ve used Kubernetes, the surprise is that OpenShell, not the kubelet or the controller, decides when Pods come back. A workload Pod deleted by hand is recreated by the controller, but it never runs.
Setup
Everything runs on the cluster and gateway from part 2: two m7i.2xlarge nodes in two availability zones, OpenShell 0.1.6 with the Kubernetes driver, and the CLI connected through a kubectl port-forward on 127.0.0.1:17680. If you scaled the nodes down, bring them back first:
cd my2centsai-blog-samples/openshell-eks
./09_scale.sh upThe scripts in openshell-lifecycle are plain bash. They source part 2’s env.sh through a small lib.sh, which also defines a few helpers:
os() { openshell -g osh-eks "$@"; } # -g before the subcommand, never after a -- command
in_sb() { local n="$1"; shift; os sandbox exec "$n" --no-login-shell --no-tty -- "$@"; }
# One line per change: phase, restart count, last exit code, seconds to next restart, generation.
sandbox_state() {
os sandbox get "$1" -o json 2>/dev/null | jq -r '[.phase, (.restart_count|tostring), (.exit_code // "-" | tostring),
(if (.next_restart_at_ms // 0) > 0 then "\(((.next_restart_at_ms / 1000) - now) | floor)s" else "-" end),
(.annotations["internal.openshell.ai/runtime-generation"] // "-" | .[0:8])] | join(" ")'
}The os wrapper’s comment is from experience. My first version appended -g osh-eks at the end, which put it after -- and turned it into an argument of the sandbox command.
watch_sandbox polls sandbox_state once a second and prints a line whenever the phase, restart count, exit code or generation changes. The generation comes from the annotation internal.openshell.ai/runtime-generation, shortened to eight characters. Each script deletes its sandboxes in a trap, and 99_cleanup.sh also uncordons nodes and switches user namespaces back off, in case a run was interrupted.
Testing
Test 1: Restart policies and generations
The command after -- in openshell sandbox create is the sandbox’s main process. The restart policy decides what happens when it exits. The policy belongs to the gateway, not to Kubernetes; the Pods themselves have restartPolicy: Never. I gave each policy a command that sleeps 5 seconds and exits:
os sandbox create --name rp-never --restart-policy never --detach --no-tty -- sh -c 'sleep 5; exit 3' >/dev/null 2>&1
watch_sandbox rp-never 45 '^(Completed|Error)$'Output: three restart policies
$ openshell sandbox create --name rp-never --restart-policy never -- sh -c 'sleep 5; exit 3'
t phase restarts exit next restart generation
0.6s Ready 0 - - 1c611af4
5.3s Error 0 3 - 1c611af4
$ openshell sandbox create --name rp-onfail --restart-policy on-failure -- sh -c 'sleep 5; exit 3'
t phase restarts exit next restart generation
0.5s Ready 0 - - 94c9c2fe
5.2s Starting 1 3 - d05122fc
25.2s Ready 1 - - d05122fc
29.8s Starting 2 3 0s d05122fc
31.4s Starting 2 3 - 0a15a471
40.6s Ready 2 - - 0a15a471
45.3s Starting 3 3 1s 0a15a471
48.4s Starting 3 3 - a81ca285
62.1s Ready 3 - - a81ca285
66.7s Starting 4 3 3s a81ca285
71.5s Starting 4 3 - 884a1227
$ openshell sandbox create --name rp-always --restart-policy always -- sh -c 'sleep 5; exit 0'
t phase restarts exit next restart generation
0.5s Ready 0 - - 232fb351
5.1s Starting 1 0 -1s 232fb351
6.6s Starting 1 0 - da5b8a76
23.6s Ready 1 - - da5b8a76
28.2s Starting 2 0 0s da5b8a76
29.7s Starting 2 0 - a809c4d9
42.1s Ready 2 - - a809c4d9What the three runs show:
never(the default): exit code 3 put the sandbox inErrorand kept it there, with the same generation. A separate run with exit 0 ended inCompleted. After the exit 3, both Pods kept running, butexecwas refused (“not ready (phase: Error)”). The gateway acceptsstartfromCompletedor from anErrorcaused by the main process, which starts a new generation.on-failure: every nonzero exit started a new generation. Thenext restartcolumn shows the backoff growing. The docs describe it as 1 second, doubling up to 3 minutes, and reset by 10 seconds of uptime. My process lived only 5 seconds, so it never reset.always: the same, for exit 0.
From exit to Ready again took 11 to 20 seconds in these runs, at most 4 seconds of it backoff. A restart isn’t a process restart: a separate check showed that each restart replaces both Pods. The supervisor Pod and the workload Pod get new UIDs, and the workspace volume is mounted again in the new workload Pod. In that check, with a command that ran for 12 seconds, the sandbox was Ready again about 6 seconds after each exit. I don’t know why the two differ that much.
A new generation is more than a counter. OpenShell’s tokens and TLS certificates are bound to a sandbox and a generation, so every restart issues fresh ones. A credential leaked from one generation is useless in the next. The flip side is that anything in memory, including an agent’s conversation state, is gone.
Then stop, start and suspend:
$ openshell sandbox create --name life (then stop, start, suspend)
ready, generation 6b2ca65f
stop: Stopped after 8.7 s
supervisor pods while stopped: 0
start: Ready after 12.7 s, generation dff0118f
$ openshell sandbox suspend life
Error: × code: 'The system is not in a state required for the operation's
│ execution', message: "compute driver does not support sandbox suspend"And a one-shot sandbox, os sandbox create --name oneshot --no-keep --no-tty -- echo "hello from a one-shot sandbox":
hello from a one-shot sandbox
oneshot gone after 20.3 s: not listedstop took 8.7 seconds and removed both Pods. The Agent Sandbox Sandbox resource stays, set to operatingMode: Suspended, and so does the workspace PVC. start brought it back in 12.7 seconds with a new generation.
suspend is in the CLI and the API, but no built-in driver implements it in 0.1.6. The gateway checks the driver’s capabilities first and refuses with FAILED_PRECONDITION, whose standard description is the odd first line above. There is also no idle timeout: a sandbox you forget about runs, and bills, until you stop or delete it.
--no-keep works as advertised: the command ran, printed its line, and the sandbox was gone 20.3 seconds after the create call, including the create itself and the delete.
Test 2: What survives, and where the volume lives
02_persistence.sh writes a file to the workspace, a file to /tmp, and starts a background process, then stops and starts the sandbox:
in_sb "$NAME" sh -c 'echo kept > /sandbox/keep.txt; echo lost > /tmp/gone.txt; nohup sleep 3600 >/dev/null 2>&1 & echo " background sleep pid $!"'
# then, simplified: os sandbox stop "$NAME"; os sandbox start "$NAME"; wait for Ready; read the files back$ openshell sandbox stop persist && openshell sandbox start persist
stopped after 7.5 s
ready again after 12.5 s
/sandbox/keep.txt: kept
/tmp/gone.txt: missing
sleep process: gone/sandbox is the workspace: a 2 GiB EBS volume (/dev/nvme2n1 inside the sandbox). /tmp is on the container’s root filesystem, which goes with the Pod. So the rule for agents is simple: anything that must survive a restart goes under /sandbox.
The volume is the catch. An EBS volume lives in one availability zone, and part 2’s node group has one node in us-east-1a and one in us-east-1b. To see what that means, the script stops the sandbox, cordons the node in the volume’s zone, and starts it again:
$ openshell sandbox stop persist; kubectl cordon ip-10-70-20-30.ec2.internal; openshell sandbox start persist
the workspace volume is an EBS volume in us-east-1a; the nodes:
ip-10-70-20-30.ec2.internal us-east-1a
ip-10-70-40-237.ec2.internal us-east-1b
+122.4s workload pod: Pending openshell: Starting
0/2 nodes are available: 1 node(s) didn't match PersistentVolume's node affinity, 1 node(s) were unschedulable. preemption: 0/2 nodes are available: 2
supervisor pod on: ip-10-70-40-237.ec2.internal
$ kubectl uncordon ip-10-70-20-30.ec2.internal
t phase restarts exit next restart generation
0.5s Starting 0 - - f000503a
13.0s Ready 0 - - f000503a
usable 14.3 s after uncordon; /sandbox/keep.txt: keptFor two minutes, the workload Pod sat in Pending, and OpenShell’s phase stayed Starting. The scheduler’s event is where I saw why. As soon as the node was schedulable again, the sandbox was Ready in 13 seconds with its file.
The supervisor Pod went to the other zone. The supervisor Pod has no affinity to its workload, so the two halves of one sandbox can run in different availability zones. All of the agent’s network traffic passes through its supervisor, so in that case all of it crosses zones, and AWS charges for data that crosses zones.
What to take from it:
- A sandbox is pinned to its volume’s zone for life. If you run nodes in several zones, keep enough capacity in each, or a zone without a free node leaves its sandboxes in
Starting(for up to 30 minutes, per the docs’ 1,800-second start deadline, before they go toError). - If cross-zone traffic matters, a single-zone node group (or a node group per zone) keeps both halves together.
Test 3: Losing a Pod
The two halves of a sandbox are separate Pods, and either can disappear: a node failure, an eviction, an operator with kubectl delete. 03_supervisor_loss.sh deletes each one in turn.
The supervisor first. The sandbox README says the workload freezes when its supervisor disconnects, and is terminated if no supervisor reconnects within a deadline (30 seconds by default). The architecture docs add that a new supervisor can’t take over a running sandbox, even with valid credentials; the exception is the shared supervisor pool, which uses a takeover grant signed by the gateway.
$ kubectl delete pod os-supervisor-be28e52f-... (the supervisor)
t phase restarts exit next restart generation
0.5s Ready 0 - - bb3450ce
exec while the supervisor is gone: refused
31.1s Provisioning 0 - - bb3450ce
65.3s Stopped 0 - - bb3450ce
(no pods)
$ openshell sandbox start sup
started again: Ready after 11.7 s, generation 2fb3eded
/sandbox/marker: written-beforeexec was refused within seconds, since every session goes through the supervisor. At 31 seconds, matching the 30-second deadline, the phase changed to Provisioning, which per the docs is also what a sandbox shows until its supervisor reconnects. At 65 seconds it was Stopped, with no Pods left. Nothing restarted by itself. One start later, it was Ready in 11.7 seconds with a new generation and the workspace intact.
That’s the behaviour I’d want. Per the docs, the workload was frozen in the meantime, and then it was removed. The cost is a stop and a lost session. Neither the gateway nor the Agent Sandbox controller brought the sandbox back without a start.
Then the workload.
$ kubectl delete pod default--sup (the workload)
t phase restarts exit next restart generation
0.5s Ready 0 - - 2fb3eded
30.3s Error 0 - - 2fb3eded
default--sup workload Pending
os-supervisor-be28e52f-... supervisor Running
$ openshell sandbox stop sup && openshell sandbox start sup
started again: Ready after 36.1 s, generation 0a2f7d82
/sandbox/marker: written-beforeThe Agent Sandbox controller noticed the missing Pod and recreated it, but the replacement never ran. A separate check showed what holds it: the scheduling gate openshell.ai/bootstrap. OpenShell removes that gate only after it validates the Pod during its own bootstrap, so a Pod the controller recreates by itself never runs. The Pod stays SchedulingGated, and after 30 seconds the gateway marks the sandbox Error. A plain start is refused in Error; stop and then start brought it back in 36 seconds with the workspace intact.
Both cases fail closed, by design: OpenShell doesn’t replace a lost Pod while the agent keeps running. Recovery is manual and keeps the workspace: start after a supervisor loss, stop and then start after a workload loss. That second path exists since 0.1.5 (#4231, which fixed #3308, where an errored sandbox refused both commands). I simulated both losses with kubectl delete and didn’t fail a real node. I also ran them with the default restart policy, never; the docs describe restart policies for exits of the main process only, and I didn’t test on-failure or always here. For supervisor loss, the experimental shared supervisor pool (part 6) is the exception: per the docs, the gateway moves a failed member’s sandboxes to live members without restarting them. I didn’t test that either. So if your agents must come back unattended, something outside OpenShell has to watch for Error and Stopped and start them again. I watched for the stuck-in-Provisioning state reported in #4371 and didn’t hit it.
Test 4: Inside the workload
04_isolation.sh runs a list of checks in one sandbox with the default policy, which allows no network destinations at all. Each line shows a check, its exit code and its output:
check() { # label, shell command run inside the sandbox
local out rc
out="$(in_sb "$NAME" sh -c "$2" 2>&1)"; rc=$?
printf ' %-34s exit %-3s %s\n' "$1" "$rc" "$(tr '\n' ' ' <<<"$out" | cut -c1-90)"
}
check "write /var/tmp/x (not in policy)" "echo x > /var/tmp/x && echo written"
check "unshare a user namespace" "unshare -U true && echo unshared"Output: every check
$ who am I
id exit 0 uid=10001 gid=10001 groups=10001
capabilities, seccomp exit 0 CapEff: 0000000000000000 NoNewPrivs: 1 Seccomp: 2
kernel (shared with the node) exit 0 6.18.56-125.164.amzn2023.x86_64
node kernel: 6.18.56-125.164.amzn2023.x86_64
$ files (Landlock: read-write /sandbox and /tmp, read-only system paths, nothing else)
write /sandbox/ok exit 0 written
write /tmp/ok exit 0 written
write /var/tmp/x (not in policy) exit 2 sh: 1: cannot create /var/tmp/x: Permission denied
list /var/tmp (not in policy) exit 2 ls: cannot open directory '/var/tmp': Permission denied
write /etc/x (read-only) exit 2 sh: 1: cannot create /etc/x: Permission denied
read /etc/hostname (read-only) exit 0 default--iso
$ processes and syscalls (seccomp, no privileges)
unshare a user namespace exit 1 unshare: unshare failed: Operation not permitted
raw packet socket exit 1 Traceback (most recent call last): File "<string>", line 1, in <module> File "/usr/lib
mount a tmpfs exit 32 mount: /tmp/m: must be superuser to use mount. dmesg(1) may have more information a
processes visible exit 0 PID USER COMMAND 1 10001 openshell-sandb 63 10001 bash 95 10001 sh 96 10001 ps 97 10
$ credentials and network
service account token exit 2 ls: cannot access '/var/run/secrets/kubernetes.io/serviceaccount': No such file or directo
env: proxy or token variables exit 0 0
DNS for pypi.org exit 0 198.18.0.2 pypi.org
https://pypi.org exit 0 curl: (7) Failed to connect to pypi.org port 443 after 1 ms: Couldn't connect to server 00
instance metadata 169.254.169.254 exit 0 curl: (7) Failed to connect to 169.254.169.254 port 80 after 0 ms: Couldn't connect to serLayer by layer:
| Layer | Check | Result |
|---|---|---|
| Identity | id | UID and GID 10001, whatever the image’s USER says |
| Privileges | /proc/self/status | no effective capabilities, NoNewPrivs: 1 |
| seccomp | Seccomp: 2 | filter mode (6 filters stacked in part 2’s run) |
| Landlock | write /sandbox, /tmp | allowed |
| Landlock | write /etc | denied (read-only path); reading /etc/hostname works |
| Landlock | write or list /var/tmp | denied: a path the policy doesn’t name can’t even be read |
| syscalls | unshare -U | Operation not permitted |
| syscalls | raw AF_PACKET socket | PermissionError: [Errno 1] |
| syscalls | raw ICMP socket | OSError: [Errno 93] Protocol not supported |
| syscalls | mount -t tmpfs | must be superuser |
| Processes | ps | PID 1 is OpenShell’s runtime, /.openshell/runtime/openshell-sandbox, as UID 10001 |
| Credentials | ServiceAccount token | no token mounted |
| Credentials | environment | no proxy, token or secret variables |
| Network | DNS for pypi.org | 198.18.0.2, a synthetic answer from OpenShell’s policy DNS |
| Network | HTTPS to pypi.org | refused within a millisecond |
| Network | instance metadata 169.254.169.254 | refused |
A few things stand out.
Landlock is allow-list, including reads. The default filesystem policy grants read-write on the workspace and /tmp, read-only on system paths such as /usr, /lib and /etc, and nothing else. /var/tmp isn’t in the list, so the agent can’t even list it. That’s stricter than a read-only root filesystem, and it applies to every process the agent starts.
The runtime is PID 1 and the agent is its child. OpenShell’s runtime starts as the container’s entrypoint, sets up Landlock and seccomp for the processes it starts, and intercepts their network connections. It does that before it starts the agent. It is not the trusted half: it shares the boundary with the agent, runs as the same UID, and, per the architecture docs, never makes policy decisions.
The network has two locks. The workload Pod’s NetworkPolicy allows no egress except to its own supervisor. Inside the Pod, the runtime intercepts every connect() and hands it to the supervisor, which decides; with no policy rules, it refuses all of them. DNS still answers, with an address from 198.18.0.0/15 that means nothing outside the sandbox. The metadata endpoint was refused too, and OpenShell’s docs say link-local addresses are always blocked, whatever the policy says. Part 4 opens the network one rule at a time.
Nothing to steal. No ServiceAccount token, no proxy variables, no credentials in the environment. The supervisor has a projected token for the gateway; the workload doesn’t. Part 4 and part 5 show how the agent uses credentials it gets only as placeholders, and where that has limits.
The kernel is the node’s. uname -r inside the sandbox and the node’s kernel version are the same string. Every layer above is enforced by that kernel, so a kernel bug sits underneath all of them. That isn’t hypothetical: Red Hat published a test in which a kernel page-cache exploit (CVE-2026-31431) run from a pod protected only by OpenShell compromised the host. Their explanation: the supervisor and proxy “share the same kernel and page cache as the workload, so a kernel-level attack could potentially corrupt the enforcement mechanism, not just bypass it.” I didn’t run an exploit; the point is that runc gives you OpenShell’s layers on a shared kernel, nothing more.
A VM per sandbox is the usual answer, and OpenShell supports one through a Kubernetes RuntimeClass. On these nodes, Kata itself ran fine, but an OpenShell 0.1.6 sandbox under Kata failed while seeding its workspace, and the VM went down for a reason I haven’t found (part 2, Test 7). Until that works, the closest thing on EKS is the next test.
Test 5: User namespaces
OpenShell can run workload Pods with hostUsers: false, so UIDs inside the Pod map to an unprivileged range on the node. It’s off by default. The chart value is server.enableUserNamespaces:
# what 05_user_namespaces.sh runs, simplified
helm upgrade openshell "${INSTALL}/charts/helm-chart-0.1.6.tgz" -n openshell --reuse-values \
--set server.enableUserNamespaces=true --wait --timeout 10m
kubectl -n openshell rollout restart statefulset/openshell$ helm upgrade openshell --reuse-values --set server.enableUserNamespaces=true
sandbox: Ready
workload Pod hostUsers: false
$ inside the sandbox
id: uid=10001 gid=10001 groups=10001
/proc/self/uid_map: 0 1439956992 65536
server.enableUserNamespaces back to falseInside, nothing looks different: still UID 10001. The UID map says the Pod’s UIDs 0 to 65535 are node UIDs from 1,439,956,992 up, so the agent’s UID 10001 is 1,439,966,993 on the node, a user that owns nothing there. If something did break out of the container, it would arrive as that user, not as UID 10001 shared with every other sandbox on the node.
The sandbox started and exec worked with no other change, on EKS 1.37’s containerd and kernel 6.18. I didn’t rerun the other tests with it on. Switching it on and off restarts the gateway each time, so the script restarts the port-forward too.
User namespaces don’t help against a kernel exploit like the one above: the kernel is still shared. They make a container escape land as an unprivileged stranger instead of a known UID. In my test on EKS 1.37 they cost nothing visible, so I’d turn them on.
Results Summary
| Result | |
|---|---|
never | Exit 3 → Error, exit 0 → Completed; Pods stay (checked after exit 3), exec refused until start |
on-failure / always | Restart with growing backoff; each restart replaces both Pods and starts a new generation |
Exit to Ready again | 11 to 20 s in Test 1; about 6 s in a separate check |
stop / start | 7.5 to 8.7 s / 11.7 to 12.7 s; Pods removed, workspace kept, new generation |
suspend | Not supported by any built-in driver in 0.1.6 |
--no-keep | Gone 20.3 s after the create call |
/sandbox across stop and start | Kept (EBS volume) |
/tmp, processes, memory | Lost |
| Start with the volume’s zone cordoned | Workload Pending, sandbox Starting for the two minutes I waited; Ready 13 s after uncordon |
| Supervisor and workload placement | Can land in different zones |
| Supervisor Pod deleted | exec refused; stopped at 65 s; start → Ready in 11.7 s |
| Workload Pod deleted | Replacement gated forever; Error at 30 s; stop + start → Ready in 36 s |
| Inside the workload | UID 10001, no capabilities, seccomp filter, Landlock allow-list, no token, no network |
| Kernel | Same as the node’s |
| User namespaces | Worked: hostUsers: false, UID 10001 → 1,439,966,993 on the node |
Cleanup
./99_cleanup.sh # sandboxes from these tests, uncordon, user namespaces off
cd ../openshell-eks
./09_scale.sh down # zero nodes between sessions99_cleanup.sh deletes any sandbox these scripts created, uncordons any cordoned node, and turns server.enableUserNamespaces back off if it’s still on. The cluster itself goes with part 2’s 99_teardown.sh at the end of the series.
Verdict
OpenShell’s lifecycle on Kubernetes is predictable once you see the rule behind it: the gateway owns the sandbox, and a Pod is only ever a disposable run of it. Restarts, stops and starts all create a fresh Pod pair with fresh credentials; the workspace volume is the only thing that carries over. When a Pod disappears, the sandbox stops or fails rather than running half-guarded. That’s the right default for a security boundary.
The isolation inside the workload is solid for what it is. Every check I ran was refused for a clear reason, and Landlock’s allow-list is tighter than I expected.
Three things to plan for:
- Nothing heals itself, by design. A lost Pod leaves the sandbox
StoppedorError;startbrings it back with its workspace, after astopif it’s inError. Unattended recovery needs something outside OpenShell watching for those phases. There’s no idle timeout and no suspend, so a forgotten sandbox keeps its Pods and its volume. - Zones matter. The workspace pins a sandbox to one zone, a sandbox waiting for capacity sits in
Starting(for up to 30 minutes), and its two halves can land in different zones and pay for cross-zone traffic. - The kernel is shared. Every layer tested here sits on the node’s kernel. User namespaces worked out of the box and are worth turning on. A VM per sandbox, the real fix, didn’t work with OpenShell 0.1.6 on these nodes.
For an agent that runs untrusted code on a shared cluster, that last point is the one to weigh. Next, part 4 gives the sandbox a network: policies, binary-scoped rules, L7 inspection and credentials the agent gets only as placeholders.