Google AX Credential Injection: Hands-On with Claude Code and GitHub

Overview

Sandboxing a coding agent protects your machine from the agent. It doesn’t protect your credentials. If Claude Code needs the Anthropic API and GitHub, the usual setup puts ANTHROPIC_API_KEY and GITHUB_TOKEN in the sandbox’s environment, and one prompt-injected printenv is enough to lose them.

Agent Substrate, the runtime under Google AX, has the alternative built in. Everything a sandbox sends, DNS aside, goes through an egress gateway. For hosts an https rule lists, the gateway terminates the sandbox’s TLS, and a replaceHeaders effect swaps a placeholder header for a real credential it fetches from a credential provider, here Kubernetes Secrets. The sandbox holds a placeholder; the real value only exists in the Secret and on the wire to the upstream API.

This is part 3 of 4 in a series on running AX myself on Amazon EKS. Part 1 covered the install, part 2 the task lifecycle, and part 4 stress-tests the same cluster. Here I ran Claude Code in an AX task with three credentials injected (the Anthropic key, a GitHub token for the REST API, the same token in Basic auth form for git) and every other host blocked. Then I tested the mechanics, had Claude Code open a real pull request, tried to get the credentials out, and rotated them under a running task. Everything ran on the same cluster: AX a496fc1, Substrate d3f2363 (v0.6.0), Claude Code 2.1.314 and gh 2.103.3.

The headline caveat first: AX doesn’t wire any of this up yet. Its Model resource reads a key from a Secret for AX’s own use, and a proposal to inject credentials into tasks is open as #475. Everything below is configured on Substrate’s side, around AX.

Architecture

Four pieces have to line up, and AX provides none of them today:

Piece	What it does	Who sets it up
Credential provider	Reads Secrets from Kubernetes for the gateway; atespace-to-namespace allow list, deny by default	Substrate install (--credential-provider='{"name":"k8s.io"}') plus a ConfigMap edit
EgressPolicy per task	Lists allowed hosts and the headers to replace	kubectl ate create egress-policy, because AX creates no policy (#449)
Trust in the gateway’s CA	The task must accept the gateway’s certificate for the hosts it intercepts	A boot wrapper in the task image, because AX doesn’t add Substrate’s trust-bundle volume
Placeholders	Claude Code, gh and git must send some value for the gateway to replace	spec.env and a git extraHeader

Setup

Step 1: The task image

AX tasks run AX’s own runner image unless spec.image says otherwise. Mine extends it with Claude Code and gh, and a small wrapper:

# BASE: the ax-task-runner digest from the install post
ARG BASE
FROM ${BASE}
ARG CLAUDE_CODE_VERSION=2.1.314
ARG GH_VERSION=2.103.3
RUN apt-get update && apt-get install -y --no-install-recommends nodejs npm jq \
    && npm install -g "@anthropic-ai/claude-code@${CLAUDE_CODE_VERSION}" \
    && curl -fsSL -o /tmp/gh.deb "https://github.com/cli/cli/releases/download/v${GH_VERSION}/gh_${GH_VERSION}_linux_amd64.deb" \
    && dpkg -i /tmp/gh.deb && rm /tmp/gh.deb
ENV NODE_EXTRA_CA_CERTS=/etc/ssl/certs/ca-certificates.crt
COPY egress-ca.sh /usr/local/bin/egress-ca
# (apt cache cleanup and chmod omitted)

No credentials go into the image. The wrapper exists because of the CA problem in Step 3.

Step 2: Secrets, and who may read them

01_setup.py is the only script that reads the real values (from a local .env), and it sends them to the Kubernetes API, never into a task:

basic = base64.b64encode(f"x-access-token:{env['GITHUB_TOKEN']}".encode()).decode()
secrets = [
    {"kind": "Secret", "metadata": {"name": "anthropic", "namespace": "ax-creds"},
     "stringData": {"api-key": env["ANTHROPIC_API_KEY"]}},
    {"kind": "Secret", "metadata": {"name": "github", "namespace": "ax-creds"},
     # gh and the REST API take a bearer token; git over HTTPS only speaks Basic,
     # and the gateway can't base64-encode, so the encoded pair is its own key.
     "stringData": {"token": env["GITHUB_TOKEN"], "basic": basic}},
    {"kind": "Secret", "metadata": {"name": "canary", "namespace": "ax-creds"},
     "stringData": {"value": "canary-1111-not-a-real-credential"}},
]

The provider refuses everything by default. A ConfigMap grants an atespace the namespaces it may resolve, and the provider only reads it at startup:

# ate-system/k8s-credential-provider-namespace-policy
policies:
- atespace: creds
  allowedNamespaces:
  - ax-creds

== namespace policy: atespace 'creds' may resolve Secrets in 'ax-creds'
configmap/k8s-credential-provider-namespace-policy unchanged
deployment.apps/k8s-credential-provider restarted
deployment "k8s-credential-provider" successfully rolled out

The GitHub token is a fine-grained token scoped to one throwaway private repo, with Contents and Pull requests read/write. That scoping matters more here than anywhere else in the post; see What injection does not solve.

Step 3: Making the task trust the gateway

To swap a header inside HTTPS, the gateway has to terminate the task’s TLS, presenting a certificate it mints from its own CA. Code in the sandbox has to trust that CA. Substrate can project it into an actor through a systemInfo trust-bundle volume on the actor template, but AX builds the templates for its tasks and doesn’t add that volume. So the task gets the CA another way: 01_setup.py reads the public certificate from its ClusterTrustBundle, every task carries it in spec.env, and the image’s wrapper adds it to the system store before the command starts:

#!/bin/sh
# egress-ca: runs on every boot, because the root filesystem is rebuilt on every resume
set -eu
if [ -n "${EGRESS_CA_PEM:-}" ]; then
  printf '%s\n' "$EGRESS_CA_PEM" > /usr/local/share/ca-certificates/ate-egress-mitm.crt
  update-ca-certificates >/dev/null 2>&1
fi
exec "$@"

The certificate is public, so spec.env is fine for it. curl, git and Python use the system store; Node uses it through NODE_EXTRA_CA_CERTS, which the image points at the same file.

Step 4: An egress policy per task

AX names each task’s Substrate actor after the task, so Substrate’s own CLI, kubectl-ate, can attach a policy to it. The helper the scripts use:

def inject(host, header, secret, key, prefix=""):
    """An https rule for one host that swaps one header for a Kubernetes Secret value."""
    entry = {"header": header, "credentialUri": f"ate-secret://k8s.io/default/ax-creds/{secret}/{key}"}
    if prefix:
        entry["prefix"] = prefix
    return {"https": {"hostnames": [host], "effects": {"replaceHeaders": [entry]}}}

def set_policy(task, rules):
    kate("delete", "egress-policy", task, "-a", "creds", check=False)
    kate("create", "egress-policy", task, "-a", "creds", "-f", "-", input=json.dumps({"rules": rules}))
    time.sleep(6)   # the DNS allow-list follows a policy change within about 5 s

A task without a policy has no network at all: its DNS answers NXDOMAIN for everything, which is also why a fresh AX task can’t git clone. DNS follows a policy change within about 5 seconds; the gateway caches each task’s policy for up to 10.

Testing

Test 1: How injection behaves (no real credentials)

02_mechanics.py uses the throwaway canary secret and httpbin.org/headers, which echoes back the request it received:

create_task("cred-mech")
show("cred-mech", "getent hosts httpbin.org || echo 'DNS: NXDOMAIN'")
set_policy("cred-mech", [inject("httpbin.org", "X-Canary", "canary", "value")])
show("cred-mech", "curl -H 'X-Canary: placeholder' https://httpbin.org/headers")

== a task with no egress policy: no network at all
DNS: NXDOMAIN
curl exit 6

== policy: https to httpbin.org, swap X-Canary for the canary secret
$ openssl s_client -connect httpbin.org:443 ... | openssl x509 -noout -issuer
issuer=CN=BHBIPH7LG73P7I4NHSYRI64N7S
$ curl -H 'X-Canary: placeholder' https://httpbin.org/headers
    "X-Canary": "canary-1111-not-a-real-credential"
 -> HTTP 200

== the header is only filled in when the request carries it
    "Host": "httpbin.org",            (no X-Canary)
 -> HTTP 200

== a host the policy doesn't list, and plain HTTP to a listed one
https://example.com/    -> HTTP 000, curl exit 6 (no DNS)
http://httpbin.org/get  -> HTTP 403

== a task that doesn't trust the gateway's CA
 -> HTTP 000, curl exit 60 (certificate problem)

Situation	What the sandbox saw
No policy	No DNS for anything
Listed host, header present	The secret’s value arrived upstream; the certificate was the gateway’s
Listed host, header absent	Request forwarded unchanged, nothing injected
Host not listed	No DNS
Plain HTTP to a host listed only for HTTPS	403 from the gateway
Task without the gateway’s CA	TLS verification failure

The first result is also the warning. httpbin echoes request headers, so the “secret” came straight back into the sandbox. The gateway can’t tell a trustworthy API from an echo; the policy is the only thing that does.

Test 2: Claude Code with a placeholder key

ENV = {"ANTHROPIC_API_KEY": "placeholder-not-a-secret",
       "CLAUDE_CODE_DISABLE_NONESSENTIAL_TRAFFIC": "1"}
ASK = "timeout 90 claude -p 'Reply with exactly: injected-ok' --model claude-haiku-5-5 < /dev/null"

set_policy("cred-claude", [https("api.anthropic.com")])                                  # allow, no injection
set_policy("cred-claude", [inject("api.anthropic.com", "x-api-key", "anthropic", "api-key")])

== the sandbox's view of the key
ANTHROPIC_API_KEY=placeholder-not-a-secret
2.1.314 (Claude Code)

== allow api.anthropic.com, but inject nothing
Invalid API key · Fix external API key

== now swap x-api-key for the real key
injected-ok
real	0m1.621s

== streaming works too
1
2
3
4
5

Same sandbox, same placeholder; the only change is the replaceHeaders effect. Streaming (--output-format stream-json) passed through the gateway intact. Two practical notes from getting here. claude -p over ax ssh hung until I redirected standard input (< /dev/null); with it, every call returned normally. And on a two-node cluster, injection only worked on one node until I opened node-to-node traffic in the security group, covered in the install post.

Test 3: git and gh without a token

gh sends Authorization: token <GH_TOKEN> to api.github.com. git sends nothing until it’s challenged, so the task configures a header for it to send up front:

git config --global http.https://github.com/.extraHeader 'Authorization: Basic placeholder'

The gateway replaces the whole header value, prefix included, so token placeholder leaves as Bearer <token> and Basic placeholder as Basic <base64(x-access-token:token)>:

GITHUB_RULES = [
    inject("api.github.com", "Authorization", "github", "token", prefix="Bearer "),
    inject("github.com", "Authorization", "github", "basic", prefix="Basic "),
]

== the sandbox's view: placeholders only
GH_TOKEN=placeholder-not-a-secret
http.https://github.com/.extraheader Authorization: Basic placeholder

== without a policy: no DNS for github.com
check your internet connection or https://githubstatus.com

== gh against the REST API
fabiodouek/my2centsai-sandbox-test private=true

== git clone, commit, push a branch
8373223 Initial commit
remote:      https://github.com/fabiodouek/my2centsai-sandbox-test/pull/new/ax-demo-push
pushed

== delete the branch again
deleted

A private repo cloned and pushed to, from a sandbox with no token in it.

Test 4: Claude Code opens a pull request

05_agent_task.py gives one task all three rules and hands Claude Code the whole job:

PROMPT = (f"In this git repo, create a new branch named {BRANCH}. Add a file NOTES.md with a short "
          "paragraph explaining that this pull request was opened by Claude Code running in a Google AX "
          "sandbox on Amazon EKS, and that the sandbox never held the Anthropic key or the GitHub token. "
          "Commit it, push the branch, and open a pull request with gh. Reply with only the PR URL.")
sh(NAME, f"cd /workspace/repo && claude -p {PROMPT!r} --model claude-sonnet-5-5 "
         "--dangerously-skip-permissions --max-turns 20 < /dev/null")

IS_SANDBOX=1 in the task’s environment lets Claude Code skip its permission prompts while running as root, which is what a sandbox is for.

== Claude Code does the rest
https://github.com/fabiodouek/my2centsai-sandbox-test/pull/13
  (11 s)

== the pull request, seen from GitHub
#13 Add NOTES.md
https://github.com/fabiodouek/my2centsai-sandbox-test/pull/13
author: fabiodouek

Eleven seconds from prompt to pull request, with Claude Code calling the Anthropic API, then git, then the GitHub API, all through the gateway.

Test 5: Trying to get the credentials out

06_leak_attempts.py holds the real values from .env only to check every piece of output for them:

def check(label, output):
    found = [n for n, v in SECRETS.items() if v in output]
    print(f"  {'LEAKED ' + ', '.join(found) if found else 'no real credential'}  <- {label}")

check("environment of every process", sh(NAME, "cat /proc/*/environ | tr '\\0' '\\n'"))
check("every file under /", sh(NAME, "grep -rIl -e sk-ant- -e github_pat_ / --exclude-dir=proc ..."))

== inside the sandbox, as root
  no real credential  <- environment of every process
  no real credential  <- every file under /, grep for sk-ant- and github_pat_
  no real credential  <- ax-task-runner's view of the task (metadata server)

== on the wire, from the sandbox's side
  > x-api-key: placeholder-not-a-secret
  < HTTP/2 200
  no real credential  <- curl -v request headers

== exfiltration and reflection
  000 exit 6
  000 exit 6
  api.github.com/user -> {"login":"fabiodouek","id":29859411}
  no real credential  <- GitHub's response to an injected request

== ask Claude Code for its key
  The `ANTHROPIC_API_KEY` environment variable is set to the placeholder string
  `placeholder-not-a-secret`. It is not a real key.
  no real credential  <- Claude Code's answer

== from the outside: the task spec and the snapshot in S3
  no real credential  <- ax get task
  no real credential  <- the actor's snapshot in S3

The two 000 exit 6 lines are an arbitrary exfiltration endpoint (webhook.site) and the echo service from Test 1, which isn’t in this task’s policy: neither resolves.

Where I looked	Result
Every process’s environment, as root	Placeholders only
Every file on the root filesystem	No sk-ant- or github_pat_ anywhere
AX’s metadata server inside the task	Placeholders only
curl -v, from the sandbox	The request leaves with the placeholder; the 200 proves it was swapped later
An arbitrary exfiltration host	No DNS
Asking Claude Code	It reported the placeholder
ax get task	Placeholders only
The task’s snapshot in S3, after a suspend	No credential, but a weak check: AX snapshots only /workspace, which was nearly empty

Test 6: Rotating and revoking under a running task

The Kubernetes Secret is the only copy, so changing it should change the next request. 07_rotation.py patches the Secret and polls the GitHub API from inside a running task:

put_token("github_pat_revoked")    # kubectl patch secret github --type merge -p {"stringData": {...}}
poll("401")
put_token(TOKEN)
poll("200")
kubectl("-n", "ax-creds", "delete", "secret", "github")
poll("403")

== baseline
  HTTP 200
== revoke: replace the token in the Secret with a dead one
  HTTP 401 after 3.2 s
== restore the real token
  HTTP 200 after 3.1 s
== delete the Secret entirely
  HTTP 403 after 3.0 s
egress denied
== put it back
  HTTP 200 after 3.1 s

Every change showed up on the first probe after it. About 3 seconds is the probe’s own round trip (ax ssh plus curl): the provider reads the Secret from the Kubernetes API on every request, with no cache. A deleted Secret fails closed with a 403 from the gateway (egress denied), not an unauthenticated request upstream.

Results summary

Test	Result
Mechanics	Header swapped only for listed hosts and only when present; everything else gets no DNS
Claude Code	Invalid API key without injection, injected-ok in 1.6 s with it, streaming works
git and gh	Private repo read, cloned, pushed, branch deleted, no token in the task
End-to-end PR	Opened by Claude Code in 11 s
Leak attempts	No real credential in env, files, metadata, ax get, the snapshot, or the agent’s answer
Rotation	Revoke, restore and delete each took effect on the next request; delete fails closed (403)

What injection does not solve

• The agent can still use the credentials. Anything it can send to api.anthropic.com or api.github.com goes out with the real value attached. The fine-grained, single-repo token is what limits the damage, not the gateway.
• An allowed host that echoes headers returns the secret. Test 1 showed it with httpbin. List only APIs you trust not to reflect credentials.
• The policy is per task, and AX doesn’t create it. Every task needs kubectl ate create egress-policy after creation, by whoever holds Substrate access. In this setup that’s the shared ate-client identity, a Substrate owner, which is fine for an evaluation cluster and nothing else.
• The CA wrapper is a workaround. It puts the gateway’s CA into the task’s environment because AX doesn’t project Substrate’s trust bundle. If the CA rotates, existing tasks keep the old one until they’re recreated: AX task specs are immutable, so every boot reinstalls the CA the task was created with.

Cleanup

08_cleanup.py deletes the tasks (their egress policies go with them), closes the pull requests it opened, deletes its branches, removes the ax-creds namespace and resets the provider’s namespace grant:

for pr in github("GET", f"/repos/{REPO}/pulls?state=open&per_page=100"):
    if pr["head"]["ref"].startswith("ax-agent-"):
        github("PATCH", f"/repos/{REPO}/pulls/{pr['number']}", {"state": "closed"})
kubectl("delete", "namespace", "ax-creds", "--ignore-not-found")

The cluster itself is torn down with part 1’s 99_teardown.sh.

Verdict

The mechanism is the best part of Agent Substrate I’ve tested. Credentials live in one place, the gateway adds them only for named hosts, everything else has no DNS, a revoke lands on the next request, and every leak attempt I made, as root and through the agent itself, came back with placeholders. Claude Code, git and gh needed nothing exotic: a placeholder value and, for git, one header.

The weak part is that none of it comes from AX. Today you:

• grant the atespace access to the Secrets’ namespace and restart the provider;
• attach an egress policy to every task with Substrate’s own CLI, after AX creates it;
• smuggle the gateway’s CA into every task, because AX’s templates don’t project it;
• and run all of that as a Substrate owner identity.

AX’s open proposal (#475) and the missing egress policies (#449) are exactly these gaps. Until they close, this works, but it’s platform-team plumbing around AX rather than an AX feature.

If you already run AX, wire this in from day one: a task without a policy can’t reach anything anyway, so the choice is between a placeholder and a key in plain text in spec.env, which ax get task prints to anyone who can read the task.