Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
25 commits
Select commit Hold shift + click to select a range
05085d0
feat(kubernetes): add proxy-pod supervisor topology
TaylorMutch Jul 2, 2026
c65ea8f
refactor(kubernetes): run proxy-pod workloads directly
TaylorMutch Jul 10, 2026
77ac3b7
fix(kubernetes): harden proxy-pod workloads
TaylorMutch Jul 13, 2026
655767a
docs(rfc): add proxy-pod supervisor topology draft RFC
russellb Aug 20, 2026
55e02e9
feat(kubernetes): configurable proxy-pod cluster DNS peers
russellb Aug 20, 2026
da54eb8
fix(kubernetes): stop the proxy-pod supervisor when the sandbox stops
russellb Aug 20, 2026
c41d714
feat(helm): gated OpenShift nonroot-v2 SCC grant for sandbox pods
russellb Aug 20, 2026
8c67e38
fix(kubernetes): make the proxy-pod DNS peer port configurable
russellb Aug 20, 2026
59f6d9c
docs(rfc): record proxy-pod OpenShift validation results
russellb Aug 20, 2026
1ef2e11
feat(compute): let drivers declare that a sandbox has no supervisor s…
russellb Aug 21, 2026
028801d
feat(kubernetes): workload entrypoint override for proxy-pod topology
russellb Aug 21, 2026
6ce0a74
fix(kubernetes): derive proxy-pod resource names from the sandbox name
russellb Aug 21, 2026
dc65ac4
docs(rfc): correct the readiness and workload-command analysis
russellb Aug 21, 2026
0bb909f
docs(rfc): explain the relay boundary and observability tradeoffs
russellb Aug 21, 2026
9e0647e
fix(cli): explain sessionless topologies instead of surfacing a raw e…
russellb Aug 21, 2026
2a16278
fix(cli): detect a sessionless topology before opening a session
russellb Aug 21, 2026
be8ca57
docs(rfc): note Kata kernel isolation and clarify workload log location
russellb Aug 21, 2026
575a447
fix(kubernetes): strip MAIN_PROCESS_SPEC from proxy-pod workloads
russellb Aug 21, 2026
50f0793
fix(cli): warn before implicit detach when a sessionless topology dro…
russellb Aug 21, 2026
3e957f5
fix(kubernetes): correct proxy-pod companion lifecycle, isolation, an…
russellb Aug 22, 2026
3e83dd7
fix(server): release the sessionless marker when a sandbox is removed
russellb Aug 22, 2026
c7c1ede
fix(cli): detect a sessionless topology before session-bound create s…
russellb Aug 22, 2026
c5c3af9
test(e2e): add a capability-scoped proxy-pod suite
russellb Aug 22, 2026
57e3bd7
docs(rfc): reflect gateway-forward removal in proxy-pod design
russellb Aug 22, 2026
59e081b
fix(kubernetes): reference proxy-pod companions by CR name in the pod…
russellb Aug 22, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
23 changes: 22 additions & 1 deletion .agents/skills/debug-openshell-cluster/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -505,7 +505,28 @@ The shared state directory should preserve `sandbox_gid` inheritance
`@openshell-sidecar-ssh`; the network sidecar verifies its peer PID before
bridging gateway relay requests. No `ssh.sock` file should appear in the shared
state directory.
Inspect all three when sandbox registration or egress enforcement fails:

If `topology = "proxy-pod"` is rendered, each sandbox should have a
separate supervisor Deployment with one supervisor pod, a headless supervisor
Service, a proxy CA Secret, and two per-sandbox NetworkPolicies. The agent pod
should have `openshell.ai/sandbox-role=agent`; the supervisor pod should have
`openshell.ai/sandbox-role=supervisor`; both should share the same
`openshell.ai/sandbox-id`. The supervisor Deployment must have a controlling
`Sandbox` ownerReference. The Deployment pod template must carry the
`openshell.io/sandbox-id` annotation so the TokenReview bootstrap path can mint
a sandbox JWT. For supervisor pods, the gateway validates the
`Pod -> ReplicaSet -> Deployment -> Sandbox` owner chain, so missing
`apps/replicasets get` RBAC can also break bootstrap. Helm renders the
Deployment, ReplicaSet, Service, Secret, and NetworkPolicy RBAC only when
`supervisor.topology=proxy-pod`; if those resources fail with forbidden errors,
confirm both the rendered `gateway.toml` and Helm values use proxy-pod topology.
If the agent cannot reach the gateway, check DNS to the headless Service, the
agent egress NetworkPolicy DNS exception for kube-dns/CoreDNS, and the
supervisor ingress NetworkPolicy allowing only that agent pod on ports `3128`
and `18080`.

Inspect the relevant containers when sandbox registration or egress enforcement
fails:

```bash
kubectl -n openshell get configmap openshell-config -o jsonpath='{.data.gateway\.toml}' | grep -E '^\[openshell\.drivers\.kubernetes\]|^topology\s*='
Expand Down
52 changes: 48 additions & 4 deletions .agents/skills/helm-dev-environment/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -70,16 +70,28 @@ mise run helm:skaffold:run:sidecar
mise run helm:skaffold:run:sidecar-mtls
```

Both commands build the `gateway` and `supervisor` images and deploy the OpenShell Helm
**Supervisor proxy-pod topology** (build once and leave running):
```bash
mise run helm:skaffold:run:proxy-pod
```

All Skaffold commands build the `gateway` and `supervisor` images and deploy the OpenShell Helm
chart. The sidecar profile renders an `openshell-network-init` init container for
nftables setup and an `openshell-supervisor-network` runtime sidecar for proxying.
Binary-aware policy mode runs that sidecar as UID 0 with `SYS_PTRACE` and
`DAC_READ_SEARCH`; relaxed mode can run it as the configured proxy UID, which
must be at least `1000` and distinct from the workload UID. The
sidecar-mTLS profile reuses `ci/values-sidecar.yaml` and restores
`server.disableTls=false` inline for Skaffold. The `pkiInitJob` hook (a pre-install
Job that runs `openshell-gateway generate-certs`) generates mTLS secrets on first
install. Envoy Gateway opt-in; see the Optional Add-ons section below.
`server.disableTls=false` inline for Skaffold. The proxy-pod profile renders
network supervision in a separate supervisor Deployment with one pod and relies
on Kubernetes NetworkPolicy enforcement so the agent pod can reach only its
paired supervisor plus DNS. The
default local k3s/k3d cluster keeps k3s's embedded NetworkPolicy controller
enabled; if you replace the CNI, install a policy-enforcing CNI before using
proxy-pod. The
`pkiInitJob` hook (a pre-install Job that runs `openshell-gateway
generate-certs`) generates mTLS secrets on first install. Envoy Gateway opt-in;
see the Optional Add-ons section below.

The gateway Service uses ClusterIP. Access is via Envoy Gateway (port `8080`) or `kubectl port-forward`.

Expand All @@ -88,6 +100,31 @@ The gateway Service uses ClusterIP. Access is via Envoy Gateway (port `8080`) or
create the Secret named `openshell-ha-pg` with a `uri` key, then run
`mise run helm:skaffold:run` or `mise run helm:skaffold:dev`.

### Kubernetes e2e profiles

Run the default Kubernetes e2e environment:

```bash
mise run e2e:kubernetes
```

Run the sidecar topology e2e environment:

```bash
mise run e2e:kubernetes:sidecar
```

Run the proxy-pod topology e2e environment:

```bash
mise run e2e:kubernetes:proxy-pod
```

The proxy-pod e2e task applies `ci/values-proxy-pod.yaml` through
`OPENSHELL_E2E_KUBE_EXTRA_VALUES`. Use an existing cluster with NetworkPolicy
enforcement, or let the wrapper create the default local k3d/k3s cluster with
k3s's embedded NetworkPolicy controller enabled.

### TLS behaviour

`ci/values-skaffold.yaml` sets `server.disableTls: true`, so Skaffold-based deploys run
Expand Down Expand Up @@ -150,6 +187,12 @@ For a sidecar-profile deployment:
mise run helm:skaffold:delete:sidecar
```

For a proxy-pod-profile deployment:

```bash
mise run helm:skaffold:delete:proxy-pod
```

### Delete the cluster entirely

```bash
Expand Down Expand Up @@ -275,6 +318,7 @@ for dependencies still declared in `Chart.yaml`.
| `deploy/helm/openshell/ci/values-high-availability.yaml` | HA test overlay (`replicaCount: 2` with external PostgreSQL Secret) |
| `deploy/helm/openshell/ci/values-keycloak.yaml` | Keycloak OIDC overlay |
| `deploy/helm/openshell/ci/values-sidecar.yaml` | Supervisor sidecar topology overlay for Kubernetes e2e/dev |
| `deploy/helm/openshell/ci/values-proxy-pod.yaml` | Supervisor proxy-pod topology overlay for Kubernetes e2e/dev; requires NetworkPolicy enforcement |
| `deploy/helm/openshell/ci/values-spire.yaml` | SPIFFE/SPIRE provider token grant overlay |
| `deploy/helm/openshell/ci/values-spire-stack.yaml` | SPIRE hardened chart values for local dev |
| `deploy/helm/openshell/ci/values-tls-disabled.yaml` | Lint-only: TLS + auth disabled (reverse-proxy edge termination) |
Expand Down
4 changes: 4 additions & 0 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -38,7 +38,7 @@ http-body-util = "0.1"
tokio-rustls = { version = "0.26", default-features = false, features = ["logging", "tls12", "ring"] }
rustls = { version = "0.23", default-features = false, features = ["std", "logging", "tls12", "ring"] }
rustls-pemfile = "2"
rcgen = { version = "0.13", features = ["crypto", "pem"] }
rcgen = { version = "0.13", features = ["crypto", "pem", "x509-parser"] }
webpki-roots = "1"
rustls-native-certs = "0.8"

Expand Down
13 changes: 9 additions & 4 deletions architecture/gateway.md
Original file line number Diff line number Diff line change
Expand Up @@ -219,12 +219,17 @@ Podman, and VM drivers deliver the initial token through supervisor-only
runtime material; Kubernetes supervisors exchange a projected ServiceAccount
token through `IssueSandboxToken`. The gateway validates that projected token
with Kubernetes `TokenReview`, requires the configured sandbox service account,
checks the returned pod binding against the live pod UID, and verifies the pod's
controlling `Sandbox` ownerReference against the live Sandbox CR UID and
sandbox-id label before minting the gateway JWT. The bootstrap path accepts
checks the returned pod binding against the live pod UID, and verifies the
pod's ownership against the live Sandbox CR UID and sandbox-id label before
minting the gateway JWT. Agent pods must be directly controlled by the
`Sandbox` CR. Proxy-pod supervisor pods may be controlled through the Kubernetes
`Pod -> ReplicaSet -> Deployment -> Sandbox` chain. The bootstrap path accepts
both `agents.x-k8s.io/v1beta1` ownerReferences from newer Agent Sandbox
controllers and `agents.x-k8s.io/v1alpha1` ownerReferences from existing
deployments. Supervisors renew gateway JWTs in memory before expiry only while
deployments. The proxy-pod gateway Role grants create/delete on its dependent
Service, Secret, and NetworkPolicy resources, plus create/delete/get on the
supervisor Deployment and get on its ReplicaSet for this owner-chain check.
Supervisors renew gateway JWTs in memory before expiry only while
the sandbox record still exists. Older tokens are not server-revoked; shared
deployments bound replay exposure with short `gateway_jwt.ttl_secs` lifetimes.
The config default is
Expand Down
Loading
Loading