Run OpenBot on Kubernetes: Bots and all, proven on EKS

[davidmckayv]

Closes #216. Bots run on Kubernetes, proven on a real EKS cluster.

Proven, not argued

A real EKS cluster (1.34, two nodes, us-east-2), image built for amd64 and pushed to ECR:

openbot-computer-0                1/1  Running    ip-192-168-48-113
openbot-postgresql-0              1/1  Running    ip-192-168-8-53
openbot-server-57cbc9684b-4fp7f   1/1  Running    ip-192-168-48-113
openbot-server-57cbc9684b-kzcq5   1/1  Running    ip-192-168-8-53
openbot-migrations-1-6knkb        0/1  Completed

Real EBS volumes bound on gp3, replicas spread across both nodes. Then:

GET  /health                        -> 200
GET  /api/agents                    -> ["general-assistant","knowledge"]
GET  /api/computers/knowledge/status-> {"state":"ready"}
POST /api/computers/knowledge/navigate {"url":"https://example.com"}
  -> {"title":"Example Domain", ..., "elapsedMs":548}

A Bot opened a real page from inside AWS, a screenshot came back as a real PNG, and the audit trail recorded the decision:

computer.action_allowed | computer_navigate | knowledge | https://example.com

And the horizontal claim, on EKS: a channel created on replica A read straight back off replica B, same Intelligence thread id. Nothing held in one process.

The same install runs on a three-node kind cluster, so a clone can get here without a cloud account.

One chart, four targets

No cloud branching in any template. grep -i 'aws\|eks\|gke\|aks\|azure' across templates/ returns one comment and zero logic. Every place the clouds genuinely differ is a value whose default is what a plain self-hosted cluster does.

• Identity is one serviceAccount.annotations map, because IRSA, Workload Identity and AKS workload identity are all annotations on a ServiceAccount.
• Secrets are a plain Secret by default, an ExternalSecret against any backend when asked.
• Gateway API sits beside Ingress rather than replacing it.
• runtimeClassName unset by default, because GKE has managed gVisor, AKS offers Kata, and on EKS it is bring your own node configuration.

ci/ holds a values file per target. self-hosted-values.yaml turns nothing on, which is the test: if it ever needs to grow, a default is wrong.

A computer for each Bot

computers.mode picks the shape. shared runs one browser for every Bot and needs nothing installed; sandbox gives each Bot its own as a Sandbox from kubernetes-sigs/agent-sandbox, which is built for exactly this workload; external points at one somebody else runs.

Suspending is operatingMode: Suspended, which terminates the pod and keeps the volumes, so a computer comes back with its logins rather than signed out of everything.

What decides a computer is idle is the audit trail, not the browser. Asking the browser would wake it, so every computer anything asked about would come back up and the bill would never fall. That is the invisible way to lose scale-to-zero, and #216 names it.

Durable work, claimed by whichever replica gets there first

work_items plus select ... for update skip locked and a lease. No coordinator, no leader election, and a replica added is throughput added rather than contention added. Written once with all three of its users in view, as #193 asks: the idle culler here, scheduled routines, and a hop from one Bot to another.

A CronJob runs the sweep, because a timer in the API fires in every replica, and suspending a browser somebody just started using is not the audit-retention sweep. The decision is re-checked at the moment of acting, since somebody may have come back in between.

Nine integration tests against real PostgreSQL, including ten replicas racing for ten items and taking each exactly once. That test fails without skip locked — I removed the clause to check.

Five things only a real install could find

drizzle-kit migrate cannot run in the shipped image. It reads a TypeScript config, needing the esbuild bun install --production correctly leaves out. It prints one line, exits 1, and says nothing. So EMBEDDED_POSTGRES=on has been starting containers whose database was never migrated, and the first symptom is the API reporting that users does not exist. server/scripts/migrate.ts uses the migrator inside drizzle-orm, already a runtime dependency, and keeps the same journal.

sessionOf stopped checking as soon as there were two replicas. It answered from a map in whichever process started the computer. The replica that took a snapshot is usually not the one handling the click, and an unknown session means "no opinion", so the check that stops a ref from a replaced computer resolving against a live one was silently absent on exactly the shape it was written for. It now asks by listing, never ensuring, so asking cannot wake a computer that had stopped. This is the item #216 lists under "Not decided" as worth a test rather than an assumption.

A browser cannot live in a replicated API pod. The image's computer gets the same EMBEDDED_COMPUTER switch its database already has, defaulting on so the one-container image is unchanged. Asking for it with more than one replica is refused at install.

Migration hook ordering deadlocks both ways with a bundled database. pre-install runs before the subchart exists, so waiting cannot help; post-install with --wait never runs, because the Deployment cannot become ready against an empty schema. On a first install with the bundled database it is an ordinary Job, created in the same pass and waiting for the database itself. Every other case is a pre hook, because the database already exists.

A fresh EKS cluster very often has no default StorageClass. eksctl creates gp2, unmarked and on the in-tree kubernetes.io/aws-ebs provisioner that current Kubernetes no longer has. A volume asking for the default never binds, the computer sits Pending, and nothing says why. Verified on the real cluster and written down where somebody configuring one will read it.

Refused at install, not in a crash loop

No database or two of them; nobody who could sign in; nobody who would be an administrator; a KEY_ENCRYPTION_KEY of the wrong shape; a bundled database with no password that could be supplied again on upgrade; a computer with no token; external with no address; both routers; both secret sources; Okta without an issuer; a browser inside more than one replica. Each names the value to change, and I drove every one.

Where it runs

• New state that outlives a request? work_items, which is the point: durable, claimed, leased, and the alternative is state in a process.
• What happens on the second replica? The question this PR exists to answer. sessionOf is the one place the old answer was wrong, and the culler is built so the answer stays right.
• Anything serialised? No.
• Anything fanned out to a browser? No.
• New listener, port, or schedule? The computer's port, inside the cluster. A CronJob, which is a schedule with a claim under it rather than a timer.

Boundary and audit

• Every acting call still goes through the gateway. Untouched.
• The sandbox provider's credential is a Role, not a ClusterRole, scoped to one namespace and to sandboxes alone. Smaller blast radius than the Docker supervisor, which holds a socket that is root-equivalent on its host.
• Nothing new is trusted from the client.

Known and deliberate

The bundled database runs as the superuser, because migration 0000 creates the vector extension and 0010 drops it, and only a superuser or the owner can do either. That is what docker-compose.yml already does, and safe for the same reason. A managed database is not this, which the values file says where somebody configuring RDS will read it.

Full gate green: 1468 pass, 0 fail. helm lint clean on all four targets.