Google AX Task Lifecycle: Hands-On on Amazon EKS

Overview

A Google AX task is a command that runs in a gVisor sandbox, and it spends most of its life not running. AX’s pitch is that agents idle for long stretches while they wait on a model or a human, so a task is cheap to create, cheap to suspend, and quick to bring back. This post, part 2 of 4 in the series, measures every step of that life on the cluster from part 1, the install: two m7i.xlarge nodes on Amazon EKS 1.37, AX a496fc1, Agent Substrate d3f2363 (v0.6.0) underneath.

The thing to understand first is what suspend means in AX. Substrate can snapshot a sandbox’s memory, but AX asks it for a volumes-only snapshot of one directory, /workspace. AX’s runner doc says it plainly: a resumed task sees “the same files but a new process tree”. Everything else in this post follows from that.

There are nine Python scripts, one per test, all driving the real ax CLI. Every timing is from my runs on 2026-10-30, timed from the client, so each one includes the CLI’s round trip through its port-forward: what an AX user actually waits for.

The lifecycle at a glance

Setup

The scripts need the cluster and the ax CLI from the install post, Python 3.11 and nothing else: no API keys, no external services. Every task runs sleep infinity or a short shell command, and all of them live in an atespace called lifecycle, so cleanup never touches anything else.

cd my2centsai-blog-samples/google-ax-lifecycle
python3 01_create.py      # and so on, through 09_cleanup.py

common.py wraps the CLI. Two helpers do most of the work: one builds a task manifest, the other runs a shell snippet inside a task over ax ssh:

def task_manifest(name, command, cpu=None, memory=None):
    ...  # apiVersion: ax.io/v1alpha1, kind: Task, spec.debug: true (needed for ax ssh)

def sh(name, script):
    """Run a shell snippet inside the task over `ax ssh` and return its output."""
    p = ax("ssh", name, "--", "sh", "-c", script, check=False, quiet=True)
    return (p.stdout + p.stderr).rstrip()

Testing

Test 1: Create

for i in range(1, 6):
    timed(f"ax apply lc-create-{i}", create, f"lc-create-{i}")
ax("get", "tasks")

  [   1883 ms] ax apply  lc-create-1
  [    684 ms] ax apply  lc-create-2
  [    682 ms] ax apply  lc-create-3
  [    681 ms] ax apply  lc-create-4
  [    694 ms] ax apply  lc-create-5
  create: n=5 median 684 ms, min 681, max 1883

NAME          ATESPACE    PHASE       ACTOR         WORKER-IP   AGE
lc-create-5   lifecycle   Suspended   lc-create-5   <none>      0s
...

A created task is Suspended, not running, and holds no worker: the two worker pods don’t change. The first ax apply of a session takes longer (1.9 s) because the CLI opens its tunnel to ax-server; after that, a create costs about 0.7 s.

Test 2: The first resume, and how big a sandbox is

for i in range(1, 6):
    timed(f"ax resume lc-create-{i}", ax, "resume", "task", f"lc-create-{i}")
print(sh("lc-create-1", "uname -r; echo nproc $(nproc); grep MemTotal /proc/meminfo; df -h /workspace"))

  [   2780 ms] ax resume lc-create-1
             -> Running, ax-workers-7d9cc594c5-4hs5z on ip-10-40-33-18.ec2.internal
  [   2804 ms] ax resume lc-create-2
             -> Running, ax-workers-7d9cc594c5-7rv4w on ip-10-40-28-182.ec2.internal
  ...
  first resume: n=5 median 2780 ms, min 2772, max 2804

== inside a sandbox with AX's default size (1 CPU, 512 MiB)
4.19.0-gvisor
nproc 2
MemTotal:         524288 kB

== a task with limits: cpu 2, memory 2Gi
  [   2820 ms] ax resume lc-limits
MemTotal:        2097152 kB

A first resume takes about 2.8 s: Substrate boots the task from its template’s golden snapshot and AX waits for the runner’s readiness probe. Tasks alternate between the two worker pods. A task without limits gets AX’s default of 1 CPU and 512 MiB; spec.resources.limits changes that, and the sandbox’s /proc/meminfo follows the memory limit exactly. nproc reports 2 in both cases, so don’t size thread pools from it inside AX.

Test 3: Suspend and resume, five times

The task’s command appends a line to /workspace/boots.log every time it starts, which shows what a resume actually does:

BOOT = ["sh", "-c", 'echo "boot $(date +%T) pid $$" >> /workspace/boots.log; exec sleep infinity']
create(NAME, BOOT)
sh(NAME, "dd if=/dev/urandom of=/workspace/blob bs=1M count=50; sha256sum /workspace/blob > /workspace/blob.sha; "
         "echo 0 > /workspace/counter; echo /tmp-was-here > /tmp/marker")
for c in range(5):
    timed("ax suspend", ax, "suspend", "task", NAME)
    timed("ax resume", ax, "resume", "task", NAME)
    sh(NAME, "increment /workspace/counter; sha256sum -c /workspace/blob.sha; ls /tmp/marker")

== cycle 1
  [    820 ms] ax suspend
  [    817 ms] ax resume
counter=1
blob: /workspace/blob: OK
tmp: ls: cannot access '/tmp/marker': No such file or directory
...
== results
  suspend: n=5 median 820 ms, min 773, max 861
  resume : n=5 median 817 ms, min 790, max 821
boot 03:43:23 pid 9
boot 03:43:30 pid 9
boot 03:43:35 pid 9
boot 03:43:44 pid 10
boot 03:43:50 pid 9
boot 03:43:55 pid 9

What	Across 5 cycles
Suspend	median 820 ms
Resume	median 817 ms
/workspace (50 MB file, counter)	Intact every time; the counter reached 5
/tmp	Gone after every resume
spec.command	Ran again on every resume: 6 boots for 1 create and 5 resumes

Suspend and resume are both well under a second, and the workspace is byte-for-byte intact. But each resume is a boot: the command starts again from the top, as a new process. An agent that wants to continue where it stopped has to write its state to /workspace and read it back on start.

Test 4: Wake on request

AX’s networking doc says every request to a task goes through Substrate’s router, which “resumes it first if it was suspended”. It also says to port-forward the router’s port 80 and curl it. On this install there is no port 80:

NAME            TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)            AGE
atenet-router   ClusterIP   172.20.180.61   <none>        443/TCP,8444/TCP   66m

The router serves 443 with mutual TLS and only admits the SPIFFE IDs given to the installer, here ax-server’s. So the test runs curl from a pod that runs as the ax-server service account and requests the same pod certificate:

spec:
  serviceAccountName: ax-server
  volumes:
  - name: podidentity            # this pod's client certificate (SPIFFE ID of ax-server)
    projected:
      sources:
      - podCertificate:
          signerName: podidentity.podcert.ate.dev/identity
          keyType: ECDSAP256
          credentialBundlePath: credential-bundle.pem

CURL = ("curl -s -o /dev/null -w 'HTTP %{http_code} in %{time_total}s' "
        "--cacert /run/servicedns-ca/trust-bundle.pem "
        "--cert /run/podidentity/credential-bundle.pem --key /run/podidentity/credential-bundle.pem "
        "-H 'ate-target-actor: lifecycle/lc-wake' https://atenet-router.ate-system.svc/readyz")

== lc-wake: create, resume, suspend
{'phase': 'Suspended', 'worker': None, 'ready': 'Suspended'}

== request 1 to the suspended task, through the router
HTTP 200 in 0.100859s
{'phase': 'Running', 'worker': '10.40.55.108', 'ready': 'TaskRunning'}

== request 2 to the suspended task, through the router
HTTP 200 in 0.103489s

== without the ate-target-actor header, or without a client certificate
HTTP 404 in 0.006015s
no client cert: HTTP 000, curl exit 55

One request to a suspended task came back 200, and AX then reported the task Running. curl, running inside the cluster, measured about 100 ms for the whole request, wake included. That’s the one number in this post not timed from my laptop: from the client, the same call took about a second, almost all of it kubectl exec. Without the header the router answers 404; without a client certificate it drops the connection.

Two things to take from it. Wake on request works and is fast, but nothing suspends idle tasks automatically yet; that’s on AX’s roadmap, so today you call ax suspend yourself. And the access boundary for every task’s network endpoint is “can create pods as ax-server in ax-system”: anyone with that can reach any task.

Test 5: When a snapshot reaches S3

Substrate’s suspend is asynchronous by default. The checkpoint is written on the node, atelet uploads it to object storage in the background after --snapshot-upload-delay (5 minutes by default), and it drops the node’s copy after --snapshot-detach-delay (30 minutes). 05_snapshot_s3.py watches that happen through the actor’s own record in Substrate, using Substrate’s CLI, kubectl-ate, since AX doesn’t surface any of it:

def snap():
    snaps = actor(NAME).get("status", {}).get("snapshots") or [{}]   # kubectl ate get actor -o json
    s = snaps[-1]
    return {"durable": s.get("durableSnapshot", {}).get("status"),   # PENDING -> COMPLETED
            "uri": s.get("durableSnapshot", {}).get("object", {}).get("snapshotUri"),
            "locality": s.get("locality") or "none (detached)"}       # the node holding a local copy

The task wrote 100 MB of random data and 100 MB of zeros to /workspace, then suspended:

== suspend, then watch the actor's snapshot status every 30 s
  [    839 ms] ax suspend
  +    2s  durable copy: PENDING    local copy on: ip-10-40-33-18.ec2.internal
  +  324s  durable copy: COMPLETED  local copy on: ip-10-40-33-18.ec2.internal
  + 1801s  durable copy: COMPLETED  local copy on: none (detached)

== the snapshot in S3
     104.9 MB  7aca69ec-.../fs/pages.img.zstd
       0.0 MB  7aca69ec-.../manifest.json
     104.9 MB  in S3 for 200 MB of files (100 MB random, 100 MB zeros), zstd-compressed

== resume, now that the only copy is in S3
  [   1623 ms] ax resume
/workspace/random: OK
/workspace/zeros: OK

Time after suspend	Where the snapshot is
0 to 5 minutes	Only on the node’s disk
5.4 minutes	Node and S3 (COMPLETED)
30 minutes	Only in S3

Two consequences. For the first five minutes after a suspend, the workspace exists only on one node; Substrate’s API guide (at the pinned commit) says a suspended actor “loses its local snapshot when its node is deleted before that snapshot finishes uploading”. And once the node copy is gone, a resume downloads the snapshot: 1.6 s here against 0.8 s from the node, for 105 MB. The compression pays: the 100 MB of zeros cost almost nothing to store.

Test 6: A 2 GB workspace

sh(NAME, "for i in 1 2 3 4; do dd if=/dev/urandom of=/workspace/part$i bs=1M count=512; done; "
         "sha256sum /workspace/part* > /workspace/sums")
for c in (1, 2, 3):
    timed("ax suspend", ax, "suspend", "task", NAME)
    timed("ax resume", ax, "resume", "task", NAME)
    sh(NAME, "sha256sum -c --quiet /workspace/sums && echo 'all 4 parts intact'")

== cycle 1
  [  13122 ms] ax suspend
  [    875 ms] ax resume
all 4 parts intact
...
== results
  suspend: n=3 median 15497 ms, min 13122, max 16254
  resume : n=3 median 875 ms, min 865, max 922

Suspend time grows with the workspace: about 15 s for 2 GB against 0.8 s for 50 MB. Resume doesn’t, at under a second, because the node still has the copy. Every byte came back on all three cycles. I couldn’t reproduce #453, where a large workspace was lost while AX reported success; that report involves a 15-second CLI deadline, and the AX build I ran allows lifecycle calls 5 minutes, so the trigger likely isn’t present here.

Test 7: How many tasks fit

Substrate books each task’s CPU and memory limits against its worker’s capacity. 07_density.py creates two waves on an otherwise idle cluster and resumes them ten at a time:

wave("lc-default", 12)                                # AX's default: 1 CPU, 512 MiB
wave("lc-small", 40, cpu="250m", memory="256Mi")

NODE                           CPU     MEMORY
ip-10-40-28-182.ec2.internal   3920m   15065436Ki
ip-10-40-33-18.ec2.internal    3920m   15065440Ki

== wave 1: 12 tasks at AX's default size (1 CPU, 512 MiB)
  resume (10 at a time): n=6 median 2862 ms, min 2844, max 2932
  running: 6, refused: 6
  refused with: no worker has room for the actor
    3 on ax-workers-7d9cc594c5-jkwf9 on ip-10-40-33-18.ec2.internal
    3 on ax-workers-7d9cc594c5-nvfpm on ip-10-40-28-182.ec2.internal

== wave 2: 40 tasks at cpu 250m, memory 256Mi
  resume (10 at a time): n=30 median 2951 ms, min 2880, max 3085
  running: 30, refused: 10
   15 on ax-workers-7d9cc594c5-jkwf9 on ip-10-40-33-18.ec2.internal
   15 on ax-workers-7d9cc594c5-nvfpm on ip-10-40-28-182.ec2.internal

== what 30 running sandboxes use (kubelet stats summary)
  ip-10-40-28-182.ec2.internal: node working set 1.6 GiB
      ax-workers-7d9cc594c5-nvfpm: 0.24 GiB for its sandboxes

Task size	Running at once	Per worker	CPU booked per worker
AX default (1 CPU, 512 MiB)	6	3	3.0 of 3.92
250m, 256 MiB	30	15	3.75 of 3.92

The ceiling is CPU, and it’s the node’s allocatable CPU: 3.92 on an m7i.xlarge. A fourth default task (4 CPU) or a sixteenth small one (4.0 CPU) doesn’t fit, so Substrate refuses it with no worker has room for the actor. Substrate’s API guide says a worker that declares no capacity is “unconstrained”. In practice it’s sized by the node: Substrate reads a worker’s limits through the Kubernetes downward API, which reports the node’s allocatable CPU when no limit is set. Memory isn’t the constraint at all: 15 idle sandboxes used 0.24 GiB of a 15 GiB node.

So the default size is expensive for idle agents. Six tasks fill two four-core nodes while they sit in sleep. Declaring smaller limits on each task is the lever.

Test 8: When things end badly

08_failures.py covers three ways a task’s life ends badly.

The command exits. The task stays Running: AX’s runner is PID 1 and keeps serving ax ssh and the metadata endpoints after the command is gone, as AX’s runner contract says it should. AX doesn’t read the exit code back, so nothing in ax get task tells you the work finished.

== the command exits with code 3
{'phase': 'Running', 'worker': '10.40.38.184', 'ready': 'TaskRunning'}
  PID COMMAND
    1 /usr/local/bin/ax-task-runner

A worker pod dies. The script puts a running task and a suspended task on the same worker, writes a file the running task never snapshots, and deletes the pod:

sh("lc-victim-running", "echo never-snapshotted > /workspace/unsaved")
kubectl("-n", "ax-system", "delete", "pod", pod, "--wait=true")

  running task:   {'phase': 'Failed', 'worker': None, 'ready': 'Failed'}
  suspended task: {'phase': 'Suspended', 'worker': None, 'ready': 'Suspended'}

== can either come back?
  [    774 ms] resume the suspended one
still-here
  [    786 ms] resume the failed one
  failed task now: {'phase': 'Running', 'worker': '10.40.30.123', 'ready': 'TaskRunning'}
 05:11:41 up 0 min
cat: /workspace/unsaved: No such file or directory

The suspended task didn’t notice: its snapshot was on the node’s disk, not in the pod, and it resumed with its file. The running task went Failed. Resuming it worked, but it had never been suspended, so it had no snapshot of its own: it booted fresh from its template’s golden snapshot, and the file it wrote was gone.

The WorkerPool changes. Any change to the pool’s template rolls the worker pods. Here I declared CPU and memory capacity on it:

  running before: ['lc-exit', 'lc-victim-running', 'lc-victim-suspended']
workerpool.ate.dev/ax-workers patched
  [   2986 ms] worker rollout
  lc-exit: {'phase': 'Failed', ...}
  lc-victim-running: {'phase': 'Failed', ...}
  lc-victim-suspended: {'phase': 'Failed', ...}

Every running task failed (the rollout itself took 3 seconds). There’s no drain: suspend your tasks before you touch the pool, or treat a pool change as an outage.

Results summary

Test	Result
Create	0.7 s; the task starts Suspended and holds no worker
First resume	2.8 s (boot from the template’s golden snapshot)
Suspend / resume, 50 MB workspace	0.82 s / 0.82 s median over 5 cycles
What survives suspend	/workspace only; /tmp and processes don’t; the command reruns on every resume
Wake on request through the router	200 in about 100 ms (measured in-cluster), task back to Running
Snapshot durability	Node only for the first 5 minutes; in S3 after 5.4; node copy dropped at 30
Resume from S3 (105 MB stored)	1.6 s
2 GB workspace	Suspend 15 s, resume 0.9 s, every byte intact
Density on 2 × m7i.xlarge	6 default tasks, or 30 at 250m CPU; CPU is the limit
Command exits	Task stays Running; exit code not reported
Worker pod dies	Running task Failed (resume boots it fresh); suspended task unaffected
WorkerPool change	Every running task Failed

Cleanup

python3 09_cleanup.py

It deletes every task in the lifecycle atespace; their snapshots in S3 go with them. The cluster itself stays up for part 3, and comes down with part 1’s 99_teardown.sh.

Verdict

AX’s lifecycle is fast and predictable once you stop expecting it to pause a process. Suspend and resume are both under a second for a normal workspace, a request through Substrate’s router wakes a task in about a tenth of a second (measured inside the cluster), and the workspace came back byte-for-byte in every test, including 2 GB across three cycles. Those are good numbers for a two-node cluster.

What I’d plan around:

• Suspend is a restart with your files. Only /workspace survives, and the command runs again from the top. Design agents to persist their state there and resume from it.
• A fresh snapshot lives on one node for five minutes. Lose the node in that window and the work since the last durable snapshot is gone. If that matters, Substrate has a synchronous suspend mode; AX doesn’t expose it.
• Size your tasks. AX’s default of 1 CPU per task fills a four-core node with three idle tasks. Declaring resources.limits raised that five-fold here.
• Worker pods are not cattle yet. A dying worker fails its running tasks, and changing the pool fails all of them.
• Some of it you can only see with Substrate’s tools. Snapshot durability, the router’s mTLS, and actor state needed kubectl-ate or a pod with ax-server’s identity; AX’s CLI and docs don’t cover them, and its networking doc describes a router port that doesn’t exist on this install.

For agents that keep their working state on disk and spend most of their time waiting, AX’s model fits: cheap to keep suspended, quick to bring back, and woken by the request that needs them. For agents that need to resume mid-thought, it doesn’t do that today. Part 3 covers the other half of running agents for real: getting them API keys without putting the keys in the sandbox. Part 4 finds out how far two nodes go.