OpenSandbox Egress and Credential Vault: Hands-On on Amazon EKS
By Fabio Douek
Jump to section
- Overview
- Architecture
- Setup
- Step 1: The companion code
- Step 2: A workload image with the clients
- Step 3: An echo server as the allowed destination
- Step 4: A sandbox with a policy and a vault
- Testing
- Test 1: No policy, no sidecar
- Test 2: Rules, a runtime patch, and a connection that outlives its rule
- Test 3: The proxy’s CA, and which clients trust it
- Test 4: Real credentials for GitHub and Anthropic
- Test 5: Every auth type, and substitutions
- Test 6: Getting the credentials out
- Test 7: Rotation and revocation
- Test 8: Pause and resume wipe the vault
- Test 9: What gets logged
- Results summary
- What the vault does not solve
- Cleanup
- Verdict
- What you sign up for
- Who should use it
Explain (TLDR) like I am...
Imagine a robot helper in a playroom with one door, and a guard at the door who holds all the real keys. The robot only carries pretend keys. When it walks to a door it is allowed through, the guard swaps in the real key, so the robot never touches it.
I tried to trick the guard. Most tricks failed: the robot could not find the keys or sneak out another way. One trick worked: when the robot called out a different door's name, the guard handed over that door's key. And after the playroom took a nap, the guard had forgotten every key.
This post tests a control that keeps API credentials away from code an AI agent runs: the Credential Vault in OpenSandbox 1.2.0, an Apache 2.0 project of the Agentic AI Foundation, on Amazon EKS. The sandbox holds placeholders, and a proxy in the same pod adds the real credential to approved outbound requests.
The material finding is an open defect. The proxy chose the credential from a header the sandbox writes, so a request to one approved host received another host's credential. A fix is proposed but not merged. Audit records are partial, and a pause erases the stored credentials.
The presenting problem is an agent that needs real API keys but should never be able to read them. The intervention is a proxy beside the sandbox that holds the keys and adds them to approved requests, while the sandbox holds placeholders. I tested it on Amazon EKS with GitHub and Anthropic credentials.
Environment variables, process records, files and the metadata services yielded nothing. One known defect reproduced: a request to an approved host drew another host's credential, because the proxy trusts a header the sandbox writes. Side effects: a pause wipes the keys, and audit lines arrive late, in batches.
There is real comfort in hearing that the keys never live where the risky code runs. It lets you hand an agent real work, git pushes and model calls, without the low hum of worry about what it could copy. In my runs that promise mostly held: the sandbox's environment, processes and files only ever held placeholders.
The harder conversation is about the one trick that worked. Naming a different host was enough to draw that host's key, and the fix is still waiting to be merged. Add a vault that forgets everything on pause and logs that arrive in batches, and trust becomes something you keep checking.
Think of the egress sidecar as a stage manager at the only door of the rehearsal room. The player inside carries a stand-in pass, and when they head for a venue on the list, the manager swaps in the real one. Every call I tried, GitHub, git pushes and a model API, landed on the beat.
The off note was in how the manager read the setlist: from the player's own shout, not from where they were actually headed, so one venue's pass ended up at another. A pause also empties the manager's pockets, and the log of who got which pass comes in late, in batches.
The story is keys that never enter the sandbox. OpenSandbox 1.2.0 adds real GitHub and Anthropic credentials at the egress, so an agent can push code and call a model while holding only placeholders. On Amazon EKS that worked in every run, and every direct way I tried to read the stored keys from inside failed.
The honest positioning comes with caveats buyers will ask about. A known flaw let a request to one approved host draw another host's key, and its fix is unmerged. Credentials vanish on pause, and the audit trail is log lines that arrive in batches.
Overview
OpenSandbox has been a project of the Agentic AI Foundation, a directed fund of the Linux Foundation, since the announcement on October 22, 2026; part 1 covers that move.
This post, part 4 of 6, tests the two features that carry the series’ angle, own the sandbox, and keep the keys outside it, on the EKS cluster from part 2. The egress sidecar holds a sandbox to a deny-by-default allowlist. The Credential Vault, built on the sidecar, adds real credentials to outbound requests, so the code in the sandbox holds only placeholders. Nine tests cover when a sandbox gets a sidecar, the rule types, the proxy’s certificate authority, real credentials, substitutions, leak attempts, rotation, pause and logging.
The vault worked with real services: the GitHub API, git clone and git push to a private test repository, and the Anthropic Messages API all authenticated from a sandbox whose environment held only osb-placeholder-* strings. The exception is the one that matters most. With dummy credentials on release 1.2.0, a request to one allowed host that carried another binding’s Host header received that binding’s credential, in both directions. I ran it over plain HTTP, and the code picks the binding the same way for HTTPS. That’s #1758, open since September 8, 2026; its HTTPS fix, PR #1759, was still open on November 7, 2026. Test 6 has the details, and the Results summary the rest.
Part 3 covers the sandbox lifecycle, pools and the switch to Kata, part 5 builds on this post to sign AWS requests at the egress, and part 6 stress-tests the same two nodes. Everything here runs under runc, with the server’s egress in dns+nft mode, as part 2 left it. Every number comes from my runs on November 1, 2026 (UTC), and I say when a run isn’t in the recordings.
Architecture
The sidecar and the workload are two containers in one pod, so they share one network namespace: the same interfaces, the same routing table, and the same 127.0.0.1. That’s how the sidecar can steer the workload’s traffic without the workload’s cooperation. It’s the only running container with the NET_ADMIN capability (the execd-installer init container runs privileged for a moment to turn off IPv6, then exits), and it installs redirect and filter rules for the whole namespace; the workload container has NET_ADMIN dropped, so it can’t undo them. The same sharing is also the vault’s weak point, as What the vault does not solve explains.
dns+nft mode) polices every packet that leaves, the proxy's own connections included. On ports other than 80 and 443, or with the credential proxy off, the workload's connections meet nftables directly; on 80 and 443 with the proxy on, it's the proxy's upstream connection that nftables drops. Blue is the trusted side, green is traffic that's allowed, red is what nftables drops.- A trusted client writes the vault, credentials and bindings, through the sidecar’s API on port 18080 with the sandbox’s egress token. Here that’s the Python SDK, through the lifecycle server’s proxy. Reads return names and a revision, never a value.
- The DNS proxy answers
NXDOMAINfor a name the policy doesn’t allow and forwards an allowed one to CoreDNS. The answer’s addresses go into nftables’ allow sets for the record’s TTL plus 60 seconds (60 to 360 seconds), renewed while a connection is open. - Ports 80 and 443 are redirected to the transparent proxy (mitmdump), which presents a certificate signed by the sidecar’s CA.
- The proxy reads the active vault over a private Unix socket and picks the binding whose scheme, host, port, method and path match. The host comes from the request’s
Hostheader. No match passes the request on unchanged. - The proxy’s own connection upstream leaves through nftables like any other packet.
- An allowed upstream gets the real credential. Injected values inside response headers are replaced with
[REDACTED]; response bodies aren’t touched. - Anything else meets nftables directly, and an address no rule or lease allows is dropped. nftables accepts packets of established connections before any rule (Test 2).
The vault has hard prerequisites: the server’s egress in dns+nft mode, a network policy on the create request, credentialProxy.enabled, and the egress token the server sets up for each sandbox (Credential Vault guide). The policy should be deny-by-default; default-allow still works in 1.2.0 but logs a deprecation warning. Here is what each layer did in my runs:
| The sandbox tried | Policy | What answered | What the sandbox saw | Test |
|---|---|---|---|---|
https://example.org/ | deny, allow pypi.org | DNS proxy | NXDOMAIN (curl exit 6) | 1, 2 |
https://github.com/ | allow *.github.com | DNS proxy | NXDOMAIN (curl exit 6) | 2 |
1.1.1.2 | allow 1.0.0.1/32 | nftables | timeout (curl exit 28) | 2 |
1.1.1.1 port 80 | allow 1.1.1.1/32 port 443 | nftables | timeout (curl exit 28) | 2 |
| The instance metadata service and the Pod Identity agent | deny, no rule for them | nothing (consistent with nftables dropping the proxy’s upstream connection) | timeout (curl exit 28) | 6 |
api.github.com/user with a placeholder token | allow, bearer binding | proxy and vault | an authenticated user | 4 |
127.0.0.1:18080/credential-vault | any | the sidecar’s API | HTTP 401 | 6 |
Exit 6 is curl’s “couldn’t resolve host”, exit 28 its timeout: a denied name fails like a missing domain, and a dropped packet like an unreachable host. Neither tells the code in the sandbox that a policy was involved.
Setup
Step 1: The companion code
The scripts run on the cluster from part 2 and use two neighboring directories: ../opensandbox-eks for the server address, the API key and env.sh, and ../opensandbox-lifecycle from part 3 for the port-forward keeper and, for 08_pause_wipe.py, the snapshot registry that pause and resume need. Run part 2’s 01_cluster.sh to 07_first_sandbox.py first.
git clone https://github.com/fabiodouek/my2centsai-blog-samples.git
cd my2centsai-blog-samples/opensandbox-egress
| File | What it does |
|---|---|
00_image.sh, image/Dockerfile | The workload image (Step 2) |
00_echo.sh, manifests/echo-server.yaml | The echo server (Step 3) |
01_sidecar.py to 08_pause_wipe.py | One script per test; Tests 8 and 9 share 08 |
98_cleanup.py | Deletes sandboxes these scripts left behind |
common.py | Part 2’s helpers, the secrets parser, the scrubber |
Each script tags its sandboxes example=opensandbox-egress, gives them a 20-minute TTL, deletes them at the end (also on failure), exits 1 if a check fails, and starts the port-forward to the server on 127.0.0.1:18080 if nothing answers there. Only 04_vault_services.py uses real credentials, a fine-grained GitHub token with push access to a private test repository and an Anthropic API key, read from a secrets file outside the repository with a line parser, never sourced. The other tests use made-up dummy-<label>-<hex> values. All output goes through a scrubber that replaces the token and the key, and their base64 forms, with <redacted>.
Step 2: A workload image with the clients
The part 2 Python image has no curl, git, Node or AWS CLI, which the CA test compares. 00_image.sh builds one that has them, for linux/amd64, from the same ECR Public base:
FROM public.ecr.aws/docker/library/python:3.12-slim
ARG AWSCLI_VERSION=2.39.3
RUN apt-get update \
&& apt-get install -y --no-install-recommends ca-certificates curl git nodejs unzip procps iproute2 \
&& curl -fsSL "https://awscli.amazonaws.com/awscli-exe-linux-x86_64-${AWSCLI_VERSION}.zip" -o /tmp/awscli.zip \
&& unzip -q /tmp/awscli.zip -d /tmp \
&& /tmp/aws/install \
&& rm -rf /tmp/aws /tmp/awscli.zip /var/lib/apt/lists/* \
&& pip install --no-cache-dir requests
./00_image.sh
It pushes opensandbox-eks/egress-tools:1 through a throwaway Docker config. The build and push took 62.7 seconds in the recording.
Step 3: An echo server as the allowed destination
To see what a request looks like when it arrives, I needed a destination that tells me. manifests/echo-server.yaml is a small Python server in the opensandbox namespace that answers any method on any path with what it received, as JSON: method, path, every header, the body, and which of its two listeners took the connection. Two Services point at the one pod:
apiVersion: v1
kind: Service
metadata: { name: echo, namespace: opensandbox }
spec:
selector: { app: echo }
ports: [{ name: http, port: 80, targetPort: 8080 }]
---
apiVersion: v1
kind: Service
metadata: { name: echo-alt, namespace: opensandbox }
spec:
selector: { app: echo }
ports: [{ name: http, port: 80, targetPort: 8081 }]
Two listeners let the Host-header test tell where a request really went, whatever its Host header says, and a /stream path sends one line a second for the open-connection test. The server speaks plain HTTP on port 80 on purpose: the proxy intercepts 80 as well as 443, and a binding with schemes: ["http"] injects there (Test 5 confirms it).
./00_echo.sh
The script applies the manifest, waits for the rollout and fetches /probe through both Services from a throwaway pod.
Step 4: A sandbox with a policy and a vault
Nothing changes on the server. Part 2 already set [egress] mode = "dns+nft" and the 1.2.0 egress image, and the server runs sandboxes under runc. A sandbox gets the sidecar and the credential proxy from its create request. Simplified from common.py and 06_leaks.py:
sb = SandboxSync.create(
workload_image(), # egress-tools:1 from Step 2
connection_config=connection("proxy"), # through the server's proxy
metadata=own_metadata(SCRIPT, RUN), # example=opensandbox-egress
network_policy=NetworkPolicy(
default_action="deny",
egress=[NetworkRule(action="allow", target="echo.opensandbox.svc.cluster.local")],
),
credential_proxy=CredentialProxyConfig(enabled=True),
env={"CARRIER_A": "osb-placeholder-a"},
timeout=timedelta(minutes=20),
resource={"cpu": "500m", "memory": "1Gi"},
)
sb.credential_vault.create(
credentials=[{"name": "cred-a", "source": {"type": "inline", "value": dummy_a}}],
bindings=[{
"name": "bind-a",
"match": {"hosts": ["echo.opensandbox.svc.cluster.local"], "schemes": ["http"]},
"auth": {"type": "apiKey", "name": "x-binding-a", "credential": "cred-a"},
}],
)
The vault is written after the create, by the trusted side, and the SDK reaches the sidecar’s API through the server’s proxy, as it reaches execd in part 2. The value is write-only: credential_vault.get() returns names, bindings and a revision.
Testing
All runs below are the recorded ones, trimmed: I dropped the lines where a script restarted the port-forward and the cleanup lines. In output from Test 4, the GitHub login is replaced with example-org and the lines that show the test repository’s URL are removed.
Test 1: No policy, no sidecar
01_sidecar.py creates two sandboxes. A has no network policy. B has defaultAction: deny, one allow rule for pypi.org, and the credential proxy on.
== A: no network policy
pod 9ca405ab-5203-43b4-b722-f442e5b2f823-0: containers ['sandbox'], init containers ['execd-installer']
PASS A has no egress container
https://pypi.org/simple/ reached (http 200)
PASS A reaches https://pypi.org/simple/
https://example.org/ reached (http 200)
PASS A reaches https://example.org/
== B: networkPolicy deny + allow pypi.org, credentialProxy enabled
pod ba420421-4b1e-4d87-b6df-c61765897fbd-0: containers ['sandbox', 'egress'], init containers ['execd-installer']
PASS B has an egress container (found: egress)
egress policy as the sidecar enforces it: {'defaultAction': 'deny', 'egress': [{'action': 'allow', 'target': 'pypi.org'}]}
PASS B reaches the allowed pypi.org (http 200)
PASS B cannot reach example.org (not in the policy) (curl rc 6)
The sidecar exists only when the create request carries a networkPolicy. Without one, the pod has a single container and outbound traffic is unfiltered, and the operator’s always-deny list (deny.always), which lives in the sidecar, doesn’t apply either. If you run OpenSandbox for other people, a sandbox without a policy is a sandbox with no egress control at all, so make a policy mandatory in whatever creates sandboxes for you.
Test 2: Rules, a runtime patch, and a connection that outlives its rule
02_rules.py runs one sandbox under deny-default with five rules: an FQDN, a wildcard, a CIDR, a port-scoped IP, and the echo server by name. There’s no credential proxy here, so nothing is intercepted: this is DNS and nftables alone.
The rule checks, trimmed
policy: {'defaultAction': 'deny', 'egress': [{'action': 'allow', 'target': 'pypi.org'}, {'action': 'allow', 'target': '*.github.com'}, {'action': 'allow', 'target': '1.0.0.1/32'}, {'action': 'allow', 'target': '1.1.1.1/32', 'ports': [443]}, {'action': 'allow', 'target': 'echo.opensandbox.svc.cluster.local'}]}
== 1. FQDN rule
pypi.org (allowed) reached (http 200)
example.org (not listed) refused (curl rc 6)
== 2. wildcard *.github.com
api.github.com (sub-label) reached (http 200)
github.com (the apex) refused (curl rc 6)
== 3. CIDR rule 1.0.0.1/32 (a direct-IP dial)
1.0.0.1 (in the CIDR) reached (http 301)
1.1.1.2 (not listed) refused (curl rc 28)
== 4. port-scoped rule 1.1.1.1/32 TCP 443
1.1.1.1:443 (allowed port) reached (http 301)
1.1.1.1:80 (same IP, other port) refused (curl rc 28)
== 5. runtime patch: add example.org
policy now: ['example.org', 'pypi.org', '*.github.com', '1.0.0.1/32', '1.1.1.1/32', 'echo.opensandbox.svc.cluster.local']
example.org (after the patch) reached (http 200)
Every rule type did what the docs say. The wildcard matches a sub-label, not the apex. Port-scoped rules, new on BatchSandbox in 1.2.0, work only in dns+nft mode and never on a domain, because names are filtered at the DNS layer. patch_egress_rules took effect on the next request. The echo server was reachable by its FQDN: the DNS proxy forwards allowed names to CoreDNS, so a Service name works like a public one.
Then the behavior #2329 reports. The script starts a 40-line stream from the echo server, one line a second, deletes the echo server’s rule four seconds in, and tries two new connections:
== 6. an established connection outlives a tightened policy (#2329)
echo.opensandbox.svc.cluster.local by name reached (http 200)
stream started: 4 line(s) after 4 s
policy tightened, echo server removed: ['example.org', 'pypi.org', '*.github.com', '1.0.0.1/32', '1.1.1.1/32']
a NEW connection by name refused (curl rc 6)
a NEW connection by ClusterIP refused (curl rc 28)
the open stream after the removal:
tick 39/40
tick 40/40
PASS the established connection delivered all 40 ticks (40 lines)
The removal stopped new connections, by name and by address, but the open stream delivered all 40 ticks: nftables accepts packets of established connections before any rule, so tightening a policy never cuts an open connection. In an “install, then lock down” flow, a connection opened during the install outlives the lockdown. PR #2346 proposes an opt-in dropEstablished flag for the policy API; it was still open on November 7, 2026. Until then, lock down before the code you don’t trust runs, not after.
Test 3: The proxy’s CA, and which clients trust it
With the credential proxy on, every HTTPS connection on port 443 that sends a server name terminates at the sidecar, which signs a certificate for the requested host with its own CA. Clients have to trust that CA or fail. 03_ca_trust.py looks at what execd’s bootstrap sets up, then makes an HTTPS call from four clients. All four trusted it:
What the bootstrap set up, and the four clients
== what the bootstrap set up
-rw-r--r--. 1 root root 237035 Nov 1 12:40 merged-ca-certificates.pem
-rw-r--r--. 1 root root 1172 Nov 1 12:40 mitmproxy-ca-cert.pem
REQUESTS_CA_BUNDLE=/opt/opensandbox/merged-ca-certificates.pem
SSL_CERT_FILE=/opt/opensandbox/merged-ca-certificates.pem
NODE_EXTRA_CA_CERTS=/opt/opensandbox/mitmproxy-ca-cert.pem
AWS_CA_BUNDLE=<unset>
CURL_CA_BUNDLE=<unset>
GIT_SSL_CAINFO=<unset>
opensandbox-mitmproxy-ca.crt
== the same HTTPS call from four clients
curl trusts the CA: HTTP 200
Python requests trusts the CA: HTTP 200
Node trusts the CA: HTTP 200
AWS CLI v2 trusts the CA: aws: [ERROR]: An error occurred (InvalidClientTokenId) when calling the GetCallerIdentity operation: The security token included in the request is invalid.
== who signs the certificate the sandbox sees
* subject: CN=api.github.com
* issuer: CN=mitmproxy; O=mitmproxy
Two files land in /opt/opensandbox: the sidecar exports its CA there (1,172 bytes), and execd’s bootstrap waits for it and writes a merged bundle of public roots plus that CA (237,035 bytes). The bootstrap points REQUESTS_CA_BUNDLE and SSL_CERT_FILE at the merged bundle and NODE_EXTRA_CA_CERTS at the CA, and drops the CA into the OS store’s directory (the last line, from /usr/local/share/ca-certificates/). It doesn’t set AWS_CA_BUNDLE. The AWS CLI call uses made-up keys against STS: InvalidClientTokenId means TLS succeeded and AWS rejected the keys, while a client that doesn’t trust the CA stops at the handshake.
I expected the AWS CLI v2 to fail here. It didn’t, so the script changes one setting at a time in the same sandbox to find out why:
| Environment (AWS CLI 2.39.3, Debian image) | Result |
|---|---|
| As the bootstrap leaves it | Trusts the CA |
REQUESTS_CA_BUNDLE and SSL_CERT_FILE unset, so only the OS store | Fails |
Only REQUESTS_CA_BUNDLE set | Trusts the CA |
Only SSL_CERT_FILE set | Fails |
Only AWS_CA_BUNDLE set | Trusts the CA |
Both failures were the same:
aws: [ERROR]: SSL validation failed for https://sts.us-east-1.amazonaws.com/ [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1082)
In my run, the AWS CLI v2 didn’t trust the CA through the OS store’s directory or SSL_CERT_FILE, only through REQUESTS_CA_BUNDLE, which the bootstrap happens to set, or AWS_CA_BUNDLE (or the ca_bundle setting in its config). botocore’s code explains why: it always hands its HTTP library an explicit CA file, its own unless told otherwise, so it never consults the OS store, and it reads REQUESTS_CA_BUNDLE as a fallback that the CLI’s environment variable docs don’t list.
The practical rule: in any image where the AWS CLI or boto3 runs behind the proxy, set AWS_CA_BUNDLE=/opt/opensandbox/merged-ca-certificates.pem yourself, the merged bundle the bootstrap writes. The default works today through an undocumented fallback, and an entrypoint that resets REQUESTS_CA_BUNDLE breaks it. Part 5’s broker sets AWS_CA_BUNDLE for every sandbox.
Test 4: Real credentials for GitHub and Anthropic
04_vault_services.py is the one test with real credentials. The sandbox may reach api.github.com, github.com and api.anthropic.com, and nothing else. Its environment gets two placeholders, and the vault gets the real values with one binding per service (from the script):
sb.credential_vault.create(
credentials=[
credential("github-token", github_token),
credential("github-basic", basic_credential("x-access-token", github_token)),
credential("anthropic-key", anthropic_key),
],
bindings=[
binding("github-api", ["api.github.com"], {"type": "bearer", "credential": "github-token"}),
binding("github-git", ["github.com"], {"type": "basic", "credential": "github-basic"}),
binding("anthropic", ["api.anthropic.com"], {"type": "apiKey", "name": "x-api-key", "credential": "anthropic-key"}),
],
)
A basic credential holds the already-encoded user:password (here x-access-token:<token>, which GitHub accepts for HTTPS git), and the binding sends it as Authorization: Basic <value>. Inside the sandbox, every command uses the placeholder: curl -H "Authorization: Bearer $GITHUB_TOKEN", a plain git clone, and x-api-key: $ANTHROPIC_API_KEY.
sandbox 439b1d78-43c5-4c17-950d-f319e5a67617; vault revision 1: ['anthropic-key', 'github-basic', 'github-token'] / ['anthropic', 'github-api', 'github-git']
(values are write-only: the sanitized state above has names and revisions, never a value)
== what the sandbox holds
ANTHROPIC_API_KEY=osb-placeholder-anthropic
GITHUB_TOKEN=osb-placeholder-github
PASS the sandbox env has only the placeholders
== bearer: GitHub API with the placeholder
authenticated as example-org
PASS api.github.com sees an authenticated user (bearer injected over the placeholder) (authenticated as example-org)
== basic: git clone and push to the test repo, throwaway branch osb-test-bd774df525
8373223 Initial commit
PASS git clone of the private repo authenticates (no credentials in the sandbox) (8373223 Initial commit)
PASS git push to the throwaway branch succeeds
git push --delete rc 0; git ls-remote rc 0, branch absent
PASS the throwaway branch is deleted from the remote and absent from ls-remote
== apiKey: Anthropic Messages API with the placeholder
ready
PASS api.anthropic.com returns a completion (ready)
== the real values are nowhere in the sandbox
PASS env, /proc/*/environ and git config hold no real token or key
All three authenticated. The GitHub API returned the token’s user, so the bearer binding replaced the placeholder Authorization header (Test 5 shows that directly). git clone and git push of a throwaway branch worked with no credential helper; the script deleted the branch, confirmed with git ls-remote that it was gone, and never touches the default branch. The Anthropic API returned a one-word completion from claude-haiku-4-5. Neither real value was in the environment, any /proc/*/environ, or the git config. That’s the vault’s promise working; Test 6 is about how far it holds.
Test 5: Every auth type, and substitutions
05_substitutions.py binds six dummy credentials to the echo server in five bindings, one per auth type (customHeaders takes two), separated by path glob (/bearer/*, /basic/* and so on), and reads what arrives:
What the echo server received
== auth types: the sandbox sends no credential
bearer Authorization = Bearer dummy-bearer-45970947
basic Authorization = Basic c3ZjOmR1bW15LWJhc2ljLWU2ZWI1NTk1
apiKey X-Api-Key = dummy-apikey-db8e4cc4
custom X-Tenant = dummy-tenant-45d1cfcd X-Trace = dummy-trace-fda9e071
== bearer replaces a header the sandbox sets itself
sandbox sent Authorization: Bearer sandbox-own-value
echo received Authorization: Bearer dummy-bearer-45970947
PASS the injected bearer replaced the sandbox's own Authorization header
== substitutions: the sandbox sends PLACEHOLDER_TOKEN
sandbox ran: curl -sS -m 15 -X POST -H 'Content-Type: application/json' -H 'X-Custom: PLACEHOLDER_TOKEN' -d '{"token":"PLACEHOLDER_TOKEN"}' 'http://echo.opensandbox.svc.cluster.local/sub/PLACEHOLDER_TOKEN/resource?key=PLACEHOLDER_TOKEN'
echo received path : /sub/dummy-sub-38c6ba26/resource?key=dummy-sub-38c6ba26
echo received header : X-Custom: dummy-sub-38c6ba26
echo received body : {"token":"dummy-sub-38c6ba26"}
PASS the placeholder itself did not arrive anywhere
== a request outside every binding gets nothing injected
/other/x: injected headers none
PASS no binding matches /other/*: nothing is injected
Each header type arrived, and the injected bearer replaced an Authorization header the sandbox had set itself, which is what makes Test 4’s placeholders work. The fifth binding is a passthrough (no header) with a substitution: the sandbox sent the literal PLACEHOLDER_TOKEN in the path, query, a header and a JSON body, and the echo server received the credential in all four places. Substitutions are opt-in, literal and case-sensitive, and skip compressed and multipart bodies. A path no binding covers got nothing.
Note that the sandbox read those values out of the echo server’s response body. The proxy redacts injected values from response headers only, so an allowed upstream that reflects a request hands the credential back, which is why I never bound a real secret to the echo server.
Test 6: Getting the credentials out
Host header received the other binding's credential, in both directions. The proxy picks the binding from the Host header, not from where the connection goes. My run proved it over plain HTTP; the 1.2.0 code picks the binding the same way for HTTPS and never checks the TLS server name, so HTTPS should behave the same until the fix lands. A sandbox that may reach any host whose response it can read can draw onto that connection any binding whose scheme, method and path the request fits. This is #1758; the fix, PR #1759, covers HTTPS only and was still open on November 7, 2026.
06_leaks.py sets up a deny-default sandbox that may reach the two echo Services and binds a dummy credential to each: cred-a as header x-binding-a on echo (listener 8080), cred-b as x-binding-b on echo-alt (listener 8081). Then it tries five ways to get a credential out. The script runs the spoof last, as section 5; here it comes first:
== 5. the Host-header spoof (#1758): which injected headers reach which listener
control, connect echo.opensandbox.svc.cluster.local: reached listener 8080 (echo), Host echo.opensandbox.svc.cluster.local, injected ['x-binding-a']
control, connect echo-alt.opensandbox.svc.cluster.local: reached listener 8081 (echo-alt), Host echo-alt.opensandbox.svc.cluster.local, injected ['x-binding-b']
PASS control: each host draws its own credential
spoof, connect to echo-alt, Host: echo: reached listener 8081 (echo-alt), Host echo.opensandbox.svc.cluster.local, injected ['x-binding-a']
the credential bound to echo.opensandbox.svc.cluster.local arrived on the connection to echo-alt.opensandbox.svc.cluster.local
spoof, connect to echo, Host: echo-alt: reached listener 8080 (echo), Host echo-alt.opensandbox.svc.cluster.local, injected ['x-binding-b']
the credential bound to echo-alt.opensandbox.svc.cluster.local arrived on the connection to echo.opensandbox.svc.cluster.local
HOST-SPOOF VERDICT (#1758, release 1.2.0): LEAK
The controls show each host drawing only its own credential. In the spoofs, curl http://echo-alt.../spoof -H 'Host: echo...' connected to echo-alt, and the listener field proves the request landed there, but the proxy matched the binding on the Host header and added x-binding-a; the reverse delivered x-binding-b to listener 8080. The values that arrived were the dummy secrets, which the sandbox never had, so the verdict can’t come from headers it set itself. And because the echo server reflects what it receives, the sandbox read each foreign credential straight out of the response.
The run is over plain HTTP with http bindings, where there’s no TLS server name at all. For HTTPS I read the 1.2.0 code instead of running it: on the default path, the proxy takes the host from the Host header (:authority on HTTP/2) for every scheme (_request_host, used by _binding_matches), and nothing on that path reads the server name (SNI); only the experimental live mode, a development-only switch that’s off by default, checks it. In transparent mode the proxy forwards to the address the sandbox connected to, as the listener field showed over HTTP. So a sandbox allowed api.github.com and any host whose responses it can read (a paste site, a webhook catcher, its own server) should be able to send Host: api.github.com there and get the GitHub token back. Part 5, Test 6, later ran the HTTPS case with AWS bindings: over a connection whose TLS server name was STS, a Host naming S3 got the vault’s headers added, and a Host outside the bindings got none, which confirms the selection by Host. PR #1759 adds the missing check, the SNI must fall inside the binding’s hosts, and by its own description leaves http bindings unaffected. So the HTTPS case stays open until #1759 lands, and the HTTP case after it.
The other four attempts all failed, as they should:
Attempts 1 to 4, as recorded
== 1. read it from inside
CARRIER_A=osb-placeholder-a
CARRIER_B=osb-placeholder-b
file search: 115 find_rc=0 (file count, then find/cat exit status)
searched 228,482 characters of environment, process and file contents
PASS env holds only placeholders
PASS neither dummy credential is in env, /proc/*/environ, /proc/*/cmdline or small files
== 2. the sidecar's API from inside the pod (shared network namespace)
GET 127.0.0.1:18080/credential-vault -> HTTP 401
GET 127.0.0.1:18080/credential-vault/_active -> HTTP 403
GET 127.0.0.1:18080/policy -> HTTP 401
== 3. instance metadata and the Pod Identity agent
IMDS 169.254.169.254 http=000 curl_rc=28
Pod Identity agent 169.254.170.23 http=000 curl_rc=28
== 4. direct-IP dials
1.1.1.1 (no rule) refused (curl rc 28)
echo ClusterIP before any DNS lookup refused (curl rc 28)
| Attempt | Result |
|---|---|
Environment, /proc/*/environ, /proc/*/cmdline, and the small readable files (under 64 KiB) under /etc, /root, /home, /tmp, /opt and /run on the container’s own filesystem: the search didn’t cross into mounted volumes, so the /opt/opensandbox volume shared with the sidecar wasn’t searched (115 files; 228,482 characters in all) | Placeholders only; neither dummy value anywhere |
The sidecar’s API on 127.0.0.1:18080, which the workload can reach because the namespace is shared | 401 for /credential-vault and /policy: the API wants the egress token, which the workload container doesn’t have. 403 for /credential-vault/_active, the path the proxy fetches real values from, which only the sidecar’s private Unix socket serves |
The EC2 instance metadata service (IMDSv2 token request) at 169.254.169.254 and the EKS Pod Identity agent at 169.254.170.23 | Timeouts: no rule allows either address. These were port 80 dials with the proxy on, so the timeout is consistent with nftables dropping the proxy’s own upstream connection |
A direct dial to 1.1.1.1, and to the echo Service’s ClusterIP before resolving its name | Timeouts: an allowed name opens its addresses only once the DNS proxy has resolved it |
The metadata result deserves a note. Nothing in the sidecar exempts link-local addresses: under deny-default, IMDS and the Pod Identity agent are unreachable because no rule allows them, and an allow rule for 169.254.0.0/16 would open them. IMDS has a second wall on this cluster: part 2 set the nodes’ IMDS hop limit to 1, so pods can’t get an IMDSv2 token in any case. The Pod Identity agent has no such wall, since it exists to answer pods; here the policy is what keeps it out of reach.
Test 7: Rotation and revocation
07_rotation.py binds one dummy credential (X-Api-Key on the echo server), then rotates and deletes it while the sandbox runs:
revision = sb.credential_vault.get().revision
sb.credential_vault.patch(expected_revision=revision,
credentials={"replace": [credential("rotating", v2)]})
The rotation run
== 1. before rotation
request echo received X-Api-Key = dummy v1 (HTTP 200)
== 2. patch with the right expectedRevision
expectedRevision=1 -> accepted; vault revision is now 2
request, same sandbox, no restart echo received X-Api-Key = dummy v2 (HTTP 200)
== 3. patch with a stale expectedRevision
expectedRevision=1 (stale, current is 2) -> HTTP 409
request after the refused patch echo received X-Api-Key = dummy v2 (HTTP 200)
== 4. delete as revocation
vault deleted
next request: reached the echo server with no credential (HTTP 200)
PASS after delete no credential is injected (reached the echo server with no credential)
Rotation is clean. A patch with the current revision bumped it from 1 to 2, and the next request in the same session carried the new value, with no restart. A patch with a stale expectedRevision got HTTP 409 and changed nothing. The status matters if you write the retry loop: the SDK’s docstring for server-routed vaults mentions 412, and the sidecar uses 412 for “vault not ready” (missing prerequisites), so on this release, treat 409 as the conflict and re-read the revision.
Revocation is where my expectations were wrong. I planned for “after delete, the request is refused”, but it reached the echo server with HTTP 200 and no X-Api-Key. For the proxy, an absent vault isn’t an error, so deleting the vault removes the secret and the network policy alone decides whether the host is still reachable. If revocation should also cut access, remove the host from the policy too, and remember Test 2: that doesn’t close a connection that’s already open.
Test 8: Pause and resume wipe the vault
On Kubernetes, the vault lives only in the sidecar’s process memory. 08_pause_wipe.py binds a dummy credential, writes a marker file, pauses the sandbox (the root filesystem goes to ECR and the pod is deleted, as in part 3), and resumes it:
The pause and resume run
== 1. before pause
request echo received X-Api-Key = the dummy value
PASS the credential is injected
== 2. pause
state Paused after 7 s
== 3. resume
resumed in 8 s
marker file after resume: 'written before pause'
PASS the root filesystem came back (marker file)
credential_vault.get(): HTTP 404 (no vault)
PASS the vault is empty after resume
== 4. the same request after resume
request echo received X-Api-Key = no credential
PASS no credential is injected
== 5. the trusted caller re-creates the vault
request echo received X-Api-Key = the dummy value
PASS the credential is injected again
The files came back; the vault didn’t. After the resume, credential_vault.get() returned 404, and the same request went out with no credential, which, as in Test 7, means it still went out. Writing the vault again restored injection. This is #1366, open since July 2026 and still open on November 7, 2026: on Kubernetes, pause deletes the pod, and the new sidecar starts empty. 1.2.0 carries the vault across pause and resume for the Fast Sandbox profile only.
In practice this means the vault isn’t something you write once at create time. Whatever resumes sandboxes has to re-create the vault, so it needs the real values, or access to wherever they live, at resume time. A sandbox resumed by anything else runs uncredentialed, and its requests still leave.
Test 9: What gets logged
The last section of 08_pause_wipe.py looks for the audit trail. The proxy logs a line per injection, naming the binding, never the value. It took me a while to see them:
== 6. audit trail: the sidecar log
after the re-created vault's single request: 0 'applied binding=' line(s) in the resumed pod's sidecar log
after 90 more requests: 83 'applied binding=' line(s), 180 egress.outbound event(s)
{"level":"warn","ts":"2026-11-01T12:42:02.771Z","logger":"opensandbox.egress","msg":"[mitmproxy] credential proxy: applied binding=echo-key revision=1 host=echo.opensandbox.svc.cluster.local method=GE
PASS the sidecar log carries 'credential proxy: applied binding=' lines (names, no values) once enough output has accumulated
(The script cuts the line at 200 characters. The full line goes on with the method, the header names and the substitution surfaces.)
After one request, the sidecar’s log had no applied binding= line. After 90 more, it had 83 of the 91, at warn level and prefixed [mitmproxy]. In an earlier test sandbox, not in the recording, I counted 0 lines after 1 request, 43 after 61, 109 after 121, 174 after 181 and 239 after 241. From my reading of the egress process’s source (not instrumented): the Go side relays only the mitmdump lines that start with credential proxy:, at warn level, and drops the rest. The batching is my inference: mitmdump writes to a pipe, so Python buffers its output. Either way, a single request can leave no trace for a long time, and if the buffering reading is right, lines still buffered when a pod is deleted never arrive; I didn’t test that.
The other log stream is the Go side’s structured events: 180 egress.outbound events over the same requests, which record hosts and resolved addresses, not credentials. And there are no audit metrics. OSEP-0012, the Credential Vault design, suggests audit fields and opensandbox_credential_proxy_* metrics; neither exists in the 1.2.0 code. The egress does export OpenTelemetry metrics for DNS, policy denials and nftables updates when you give it an OTLP endpoint, but nothing per credential.
Results summary
| Test | Script | Recorded run | Result |
|---|---|---|---|
| 1. No policy vs deny-by-default | 01_sidecar.py | 21.0 s | Sidecar only with a networkPolicy |
| 2. Rules and #2329 | 02_rules.py | 86.1 s (a 40 s stream) | Every rule type held; an open stream outlived its rule |
| 3. The proxy’s CA | 03_ca_trust.py | 29.3 s | AWS CLI v2 only via REQUESTS_CA_BUNDLE or AWS_CA_BUNDLE |
| 4. Real services | 04_vault_services.py | 20.2 s | GitHub, git and Anthropic authenticated with placeholders |
| 5. Auth types and substitutions | 05_substitutions.py | 19.1 s | All five auth types and four substitution surfaces worked |
| 6. Getting the credentials out | 06_leaks.py | 43.5 s | Four attempts failed; the Host-header spoof leaked both ways |
| 7. Rotation and revocation | 07_rotation.py | 14.8 s | Stale revision 409; delete sends requests uncredentialed |
| 8. Pause and resume | 08_pause_wipe.py | 52.7 s, with Test 9 | Paused in 7 s, resumed in 8 s; vault gone (404) |
| 9. What gets logged | 08_pause_wipe.py | same run as Test 8 | Names only, in batches (0 after 1 request, 83 after 91) |
All eight scripts exited 0 with every check passing, in 4 minutes 47 seconds together. The spoof isn’t a pass/fail check: 06_leaks.py records its verdict as a finding and exits 0 either way, so the LEAK sits next to passing controls and four failed attempts. Across my earlier runs that day, pause took 7 to 11 seconds and resume 6 to 8.
What the vault does not solve
The vault does what it promises for what the sandbox holds: the real values never showed up in its environment, its processes’ environments or its git config (Test 4), the dummies weren’t in any file I searched (Test 6), and the sidecar’s API refused it. What it doesn’t do is just as specific.
- It picks the credential from a header the sandbox writes (#1758). Test 6 has the evidence and the scope. Until PR #1759 ships for HTTPS, and for
httpbindings even after, the allowlist is the real boundary: a sandbox should reach only hosts that can’t reflect a request back to it, and only hosts whose credentials it may use together. That rules out a general web allowlist next to a valuable binding. The PR’s diff also targets an older layout of the proxy script, so it needs a rebase before it can merge. - Bindings are host-wide unless you scope them (#1776). A binding without methods or paths is normalized to all common methods and
/*, and the setting that requires explicit scopes,OPENSANDBOX_EGRESS_CREDENTIAL_VAULT_REQUIRE_SCOPED_MATCH, is off by default. It’s one of the egress settings a create request may set. My Test 4 bindings were host-wide too: the GitHub token would have been added to a request for any path onapi.github.com, with any of the five default methods (GET, POST, PUT, PATCH and DELETE). I didn’t test the scoped mode. - A pause forgets everything (#1366). Test 8. Your control plane owns re-creating the vault after every resume, and a sandbox it misses still sends its requests, without credentials.
- The audit trail is thin. Log lines with binding names, relayed in batches, and no metrics (Test 9). If you need to answer “which sandbox used which credential, when”, you’ll be reconstructing it from container logs that can lag or lose their tail.
- Response bodies aren’t redacted. Only response headers are. An API that echoes the request, or an error page that quotes the
Authorizationheader, puts the credential in front of the sandbox. Test 5 shows it with the echo server. - The real secret lives in the same pod as the code you don’t trust. The sidecar holds every value in memory, in the same network namespace and on the same kernel as the workload. In my tests the boundary was the sidecar’s API token and the
NET_ADMINcapability the workload doesn’t have, and both held. But anything that breaks out of the sandbox container (a kernel bug, a container escape under runc) lands next to the secrets. Kata represents a pod as one VM, sidecar included, so it protects the node, not the sidecar. - The proxy has known open dependency alerts. The 1.2.0 release notes list five open Dependabot alerts on the egress image’s
h2library, two of them request-smuggling CVEs that wait on a move to mitmproxy 12. As of November 7, 2026, I found no tracking issue for that move.
None of this makes the vault useless; it makes it a layer. A placeholder in the sandbox still beats a real token in its environment, where any env or cat /proc/self/environ would print it.
Cleanup
Every script deletes its own sandboxes. After an interrupted run, delete what’s left, then the echo server:
uv run python 98_cleanup.py # sandboxes with example=opensandbox-egress only
./00_echo.sh delete # the echo Deployment, both Services and the ConfigMap
04_vault_services.py deletes its throwaway branch on the test repository in the normal flow and again on every failure path, and prints a loud message if it can’t; check the repository’s branches if you ever see it. The egress-tools:1 image stays in ECR with the cluster’s other images. Between sessions, scale the nodes to zero from part 2’s directory, and remove everything with its teardown:
../opensandbox-eks/08_scale.sh down
../opensandbox-eks/99_teardown.sh
Verdict
The egress sidecar’s rules held in every test. Deny-by-default is one field on the create request, rules can be patched at runtime, the CA plumbing makes curl, Python and Node work behind the proxy unchanged, and the vault did the thing that matters: a sandbox pushed to a private repository and called a model API while its environment held only placeholders. Everything I tried from inside to read a stored credential directly (the environment, /proc, files, the sidecar’s API, the metadata services) failed.
The weak spot is binding selection. On 1.2.0, a sandbox that can reach one host whose responses it reads can draw onto that connection any binding whose scheme, method and path the request fits. The run showed it over HTTP, and the code says HTTPS behaves the same; the HTTPS fix has been a pull request since September 8, with no approving review as of November 7, 2026. The rest I’d plan for rather than worry about: a vault that a pause erases, a revocation that leaves the host reachable, connections that outlive a tighter policy, and an audit trail in batches.
What you sign up for
| Concern | The egress sidecar and Credential Vault on 1.2.0, as tested |
|---|---|
| Coverage | Only sandboxes created with a networkPolicy; without one there’s no sidecar and no filtering |
| Enforcement | Names at the DNS proxy, packets in nftables (dns+nft); open connections survive a tighter policy (#2329) |
| Clients | curl, Python requests and Node trust the proxy’s CA out of the box; the AWS CLI v2 through an undocumented fallback, so set AWS_CA_BUNDLE |
| Where the secrets live | In the sidecar’s memory, in the same pod and network namespace as the workload; the sandbox holds placeholders |
| Which credential is used | Chosen by the Host header, over HTTP (tested) and HTTPS (from the code; run in part 5 with AWS bindings) (#1758; the HTTPS fix, PR #1759, open as of November 7, 2026), host-wide unless scoped (#1776) |
| Rotation and revocation | patch with expectedRevision (409 on a conflict), effective on the next request; delete stops the credential, not the traffic |
| Pause and resume | The vault is wiped; your control plane writes it again (#1366) |
| Audit | Log lines with names only, relayed in batches; no credential metrics |
Who should use it
The vault fits a team that already controls what its sandboxes may reach and wants credentials out of the sandbox’s environment: a coding agent with access to one Git host and one model API, say, with a deny-by-default policy that lists only those hosts. Keep the allowlist short, keep reflecting hosts off it, scope bindings to the paths you use, set AWS_CA_BUNDLE in your images, and make whatever resumes sandboxes re-create their vaults. On those terms, it’s a real improvement over handing the agent the token.
It doesn’t fit, yet, a sandbox that needs broad web access and valuable credentials at the same time, or a compliance story that depends on a per-credential audit trail. Part 5 takes the same vault to AWS, where a credential can’t just be added as a header, and signs requests at the egress instead.