feat(kubernetes): add proxy-pod supervisor topology - #2885
Draft
russellb wants to merge 25 commits into
Draft
Conversation
Add the Kubernetes proxy-pod topology with one supervisor Deployment and Service per sandbox, NetworkPolicy confinement, proxy-pod Helm/Skaffold configuration, topology documentation, and focused supervisor identity tests. Signed-off-by: Taylor Mutch <taylormutch@gmail.com>
Signed-off-by: Taylor Mutch <taylormutch@gmail.com>
Signed-off-by: Taylor Mutch <taylormutch@gmail.com>
Signed-off-by: Russell Bryant <rbryant@redhat.com>
The proxy-pod agent egress NetworkPolicy hardcoded its DNS peers as kube-system/k8s-app=kube-dns and kube-system/k8s-app=coredns. That is an upstream Kubernetes convention, not a guarantee. On OpenShift, cluster DNS runs in the openshift-dns namespace with pods labeled dns.operator.openshift.io/daemonset-dns=default, and kube-system holds no DNS pods at all. The hardcoded selector matches nothing, so DNS egress falls through to the policy's implicit deny and the agent pod cannot resolve any name, including its own paired supervisor Service. The sandbox is inert. Add proxy_pod.dns_peers (Helm: supervisor.proxyPod.dnsPeers), a list of namespace/pod label selector pairs, defaulting to the previous upstream behavior so existing deployments are unaffected. Reject an empty peer list at startup, and render no DNS rule at all rather than an empty 'to' array when the list is empty: in NetworkPolicy semantics an empty 'to' matches every destination, so emitting one would silently open DNS-port egress cluster-wide. Signed-off-by: Russell Bryant <rbryant@redhat.com>
In proxy-pod topology the network supervisor runs in its own Deployment, so it does not stop when the agent pod does. A stopped sandbox kept its supervisor pod running indefinitely, consuming a pod slot, CPU, and memory for a sandbox the user believes is stopped. Scale the paired Deployment to zero on stop and back to one on start. The scale-down runs only after the workload has actually stopped so a graceful shutdown that needs egress still has it, and scaling failures are logged rather than failing the start/stop RPC. Extract the stop wait loop into wait_for_sandbox_stopped so the scale-down has a single place to hook, and grant the sandbox Role 'patch' on deployments. Signed-off-by: Russell Bryant <rbryant@redhat.com>
The Kubernetes driver assigns explicit non-root UIDs to sandbox and supervisor containers. OpenShift's restricted-v2 SCC uses runAsUser: MustRunAsRange and admits only UIDs inside the namespace's openshift.io/sa.scc.uid-range annotation, so it rejects both pods. The built-in nonroot-v2 SCC resolves this without a custom SCC: it is restricted-v2 with runAsUser: MustRunAsNonRoot and fsGroup: RunAsAny, while keeping requiredDropCapabilities ALL, allowPrivilegeEscalation false, no privileged containers, no host namespaces, and seccomp runtime/default. Its volume allowlist already covers every volume type proxy-pod topology uses. Add sandboxServiceAccount.openshift.nonrootSCC, default false so non-OpenShift installs never reference OpenShift-only APIs. When enabled it renders only a ClusterRole and ClusterRoleBinding granting 'use' on the existing nonroot-v2 SCC; no SecurityContextConstraints object is created. This makes proxy-pod the first OpenShell topology that runs on OpenShift under an unmodified, Red Hat-shipped SCC. Signed-off-by: Russell Bryant <rbryant@redhat.com>
A NetworkPolicy egress rule whose peer is a podSelector is evaluated against the destination pod after Service address translation, so the rule must carry the DNS pods' container port, not the Service port. Upstream CoreDNS listens on 53, so the two coincide. OpenShift's dns-default listens on 5353 and its Service maps 53 onto it, so a rule allowing port 53 never matches and the agent pod still cannot resolve anything. Verified on OpenShift 4.22 / OVN-Kubernetes: with the correct selectors but port 53, DNS failed both via the Service ClusterIP and via the DNS pod IP directly; with port 5353 it resolves. Add a per-peer 'port' field defaulting to 53, and emit one egress rule per peer rather than one shared rule, since a rule's port list applies to all of its 'to' entries and peers may differ. Signed-off-by: Russell Bryant <rbryant@redhat.com>
Update the RFC with what a live OpenShift 4.22 / OVN-Kubernetes deployment showed: the DNS peer port mismatch, the measured SCC split between the two pods, and two usability gaps that block adoption -- the user-supplied workload command is silently discarded, and sandboxes never leave Provisioning because nothing opens the supervisor session the Ready transition depends on. Replace the now-answered open question about OVN-Kubernetes service address translation with the questions those findings raise. Signed-off-by: Russell Bryant <rbryant@redhat.com>
…ession The gateway forced SandboxPhase::Provisioning unless a ConnectSupervisor session was live. That session is opened only by openshell-supervisor-process and carries only relays -- SSH, exec, port forwarding, file transfer -- so proxy-pod topology, which has no in-sandbox process supervisor, could never reach Ready. Verified on OpenShift: both pods running and policy-enforced egress working end to end, while the sandbox reported Provisioning forever and every Ready-gated RPC, including stop and start, was unreachable. Add SupervisorSessionModel to DriverSandboxStatus. UNSPECIFIED preserves the existing contract, so drivers that never set it are unaffected. The Kubernetes driver reports NONE for proxy-pod and REQUIRED otherwise, and the gateway then derives readiness from the backend conditions alone. Ready must not become a lie in the process. The agent pod gains a wait-for-proxy init container that blocks on its paired supervisor's proxy port, so the pod is not Ready until egress actually works. This also closes a pre-existing ordering gap where the workload could start before the proxy existed and its early egress simply failed. Relay-backed RPCs now fail immediately with an explanation naming the topology instead of waiting out a session timeout that cannot succeed. Signed-off-by: Russell Bryant <rbryant@redhat.com>
proxy-pod runs the sandbox image directly, with no supervisor to launch a workload, so the container needs an entrypoint that stays running. OpenShell's own sandbox images use an interactive shell entrypoint, which reads EOF under kubelet and exits 0, leaving the pod in CrashLoopBackOff with empty logs. Add containers.agent.command and containers.agent.args to the Kubernetes driver_config passthrough, alongside the existing resources and volume_mounts. This needs no public API change: the initial command supplied to 'sandbox create' is delivered over the supervisor session, which this topology does not have. Reject the fields in combined and sidecar topology, where the driver replaces the container command with the supervisor binary and an override would be accepted and then silently dropped. Signed-off-by: Russell Bryant <rbryant@redhat.com>
Per-sandbox proxy-pod resources are named from the sandbox name, but the stop, start, and delete paths passed the Sandbox CR name. The two differ: a CR is named <workspace>--<sandbox>, so a sandbox named 'rdy' has CR 'default--rdy' and Deployment 'os-sup-rdy-<hash>'. The scale-down on stop therefore patched a Deployment that does not exist and silently did nothing, leaving the supervisor running for a stopped sandbox -- the exact problem the scaling was added to fix. Delete was affected too, but owner-reference garbage collection reclaimed the resources anyway and hid it. Read the sandbox name from the CR's sandbox-name label at both sites, and fall back to owner-reference GC with a warning if the label is missing. Caught by cluster testing; the unit tests passed throughout because they never exercised the CR-name-to-resource-name path. Signed-off-by: Russell Bryant <rbryant@redhat.com>
The earlier draft framed these as two independent gaps and said the driver silently discarded the workload command. That was wrong about the mechanism: the initial command from 'sandbox create' is delivered over the supervisor session after Ready, so it never ran because Ready never arrived. Rewrite both sections around what the code actually does -- Ready gated on a relay-carrying session that this topology cannot open -- and record the fixes and their cluster verification, including the CR-name versus sandbox-name bug that only on-cluster testing exposed. Signed-off-by: Russell Bryant <rbryant@redhat.com>
The feature list said SSH, exec, and file transfer were unavailable without saying why, which read as an implementation gap rather than a structural one. Record the mechanism: RelayOpen targets something 'inside the sandbox', the SSH server exists only in openshell-supervisor-process, and sessions need the workload's PID, mount, and network namespaces -- ssh.rs calls setns to enter the sandbox netns. The sidecar bridge to an abstract socket works only because both processes share a pod. Note that TCP relays are the exception and are recoverable for services bound to 0.0.0.0. Add the observability picture, measured on OpenShift. Network OCSF events, policy config events, and denial analysis all reach 'openshell logs' as usual, because log push is gated on sandbox ID and endpoint rather than topology. What is lost is workload stdout, which now reaches only the container log, and actor attribution on network events, which renders as -(0) because reading /proc across a pod boundary is impossible. Restructure the compatibility tables by concern and record the enforcement mechanism, pods per sandbox, and OpenShift SCC per topology. Signed-off-by: Russell Bryant <rbryant@redhat.com>
…rror Creating a sandbox in a topology with no in-sandbox supervisor succeeded, then failed at the interactive-session step with a bare gRPC error. The sandbox was running and its network policy enforced, but the output read as a failed create. Detect the gateway's rejection and print what actually happened: the sandbox is running, sessions are unavailable for this topology, egress is unaffected, and which topologies to use when interactive access is required. When a command was passed to 'sandbox create', say plainly that it did not run and point at the containers.agent.command entrypoint override instead. The command still exits non-zero. A command that did not run must not report success, and callers should not have to parse output to find that out. Detection keys on a stable marker constant shared through openshell-core rather than on prose, so rewording the message cannot silently break it, and it searches the whole error chain because the marker arrives wrapped in a transport error. Signed-off-by: Russell Bryant <rbryant@redhat.com>
The previous commit explained the topology after a relay RPC was rejected, but 'sandbox create' still spawned ssh first, so the failure surfaced through the subprocess as 'ssh exited with status 255' and the explanatory message was buried in wrapped stderr. Publish a SupervisorSession=False/NotApplicable condition in the public sandbox status when the driver reports SupervisorSessionModel::None, and have the CLI check it before attempting a session. When set, the CLI skips the connect/exec path entirely and prints the explanation directly: the sandbox is running, sessions are unavailable for this topology, egress is unaffected, and how to set a workload entrypoint when a command was supplied. The error-marker detection from the prior commit stays as the backstop for relay RPCs issued directly against an existing sandbox, where there is no create-time status to pre-check. Signed-off-by: Russell Bryant <rbryant@redhat.com>
Add the workload-to-supervisor kernel-isolation property to the topology comparison: because proxy-pod places the workload and supervisor in separate pods, a VM-based RuntimeClass like Kata gives them separate VMs and kernels, so a workload kernel compromise does not by itself reach the supervisor's gateway credentials. This is unique to proxy-pod; the in-pod topologies share one Kata VM between workload and supervisor. Clarify that lost workload stdout/stderr is specifically the agent container's log, reachable only via 'kubectl logs <agent-pod>', not 'openshell logs'. Signed-off-by: Russell Bryant <rbryant@redhat.com>
Main renamed the canonical-command transport from OPENSHELL_SANDBOX_COMMAND to the versioned OPENSHELL_MAIN_PROCESS_SPEC (NVIDIA#2726), which the supervisor decodes and launches. proxy-pod runs the sandbox image directly with no supervisor, so that env var is not only useless in the workload container but leaks the intended command into it. Strip MAIN_PROCESS_SPEC alongside the other supervisor-oriented variables, replacing the now-removed SANDBOX_COMMAND entry. Rebase adaptation: proxy-pod's workload command continues to flow through the containers.agent.command driver_config, since the canonical main process requires an in-sandbox supervisor this topology does not run. Signed-off-by: Russell Bryant <rbryant@redhat.com>
russellb
force-pushed
the
feat/kubernetes-proxy-pod-topology
branch
from
August 21, 2026 22:25
4308aa6 to
575a447
Compare
…ps the command A non-interactive persistent create takes main's implicit-detach path and returns before the sessionless check ran, so 'openshell sandbox create -- cmd' against a proxy-pod sandbox silently created a sandbox where the command never runs -- exactly the broken-looking outcome the sessionless messaging exists to prevent. Check for a command-bearing sessionless topology before the detach return and explain that the command will not run, pointing at containers.agent.command. The interactive no-command case still reports after detach. Both paths share a new abort_sessionless_create helper so ephemeral cleanup and messaging stay identical. Signed-off-by: Russell Bryant <rbryant@redhat.com>
…d placement Addresses several proxy-pod review findings in the Kubernetes driver: - Delete the Sandbox CR (tearing down the workload) before removing the companion resources, so the agent egress NetworkPolicy fence is never dropped while the workload can still egress, and a failed CR delete leaves the fence in place. - Derive companion resource names from the Sandbox CR name, which is unique in every workspace mode, instead of the bare sandbox name. In shared mode workspace-a/dev and workspace-b/dev share a sandbox name, so the previous scheme collided and a rollback could dismantle another sandbox's isolation. - Create companions in, and point the workload's proxy URL at, the sandbox's resolved target namespace rather than the static configured namespace, so proxy-pod works in managed and operator workspace modes. Add the proxy-pod resources to the cluster-scoped Role for those modes. - Remove the raw gateway-forward tunnel (supervisor :18080 to the gateway, reachable by the agent). Nothing on the agent consumed it, and with unauthenticated gateway access it was a policy-bypassing path to the admin API. The supervisor still connects to the gateway directly for its own policy, inference, and log traffic. - Return an error from supervisor Deployment scaling and propagate it from start_sandbox, so a transient scale-up failure surfaces instead of wedging the sandbox in Starting with the supervisor at zero replicas. Scale-down on stop stays best-effort. - Give the supervisor Deployment the workload's nodeSelector, tolerations, and priorityClassName, so required same-node affinity cannot pin the workload to a node its own placement excludes. Signed-off-by: Russell Bryant <rbryant@redhat.com>
set_sessionless recorded proxy-pod sandboxes in the supervisor session registry, but forget_sessionless was never called, so the set grew without bound as ephemeral proxy-pod sandboxes were created and deleted. Clear the marker in cleanup_sandbox_state, which runs on permanent removal. It is deliberately not cleared in the stopped-session cleanup: the sessionless property is a topology fact that must survive stop/start. Signed-off-by: Russell Bryant <rbryant@redhat.com>
…teps The sessionless check ran after the structured-output early return and after the upload, forward, and editor steps. So sandbox create --output json -- <cmd> exited zero while proxy-pod silently discarded the command, and upload, forward, or editor failures on --no-keep bypassed cleanup and leaked the ephemeral sandbox. Detect the sessionless topology at the top of the Ready arm. When a session-requiring operation was requested (a command, upload, forward, or editor), abort immediately -- cleaning up an ephemeral sandbox and reporting a non-zero exit -- before any of those steps or a structured-success print. A bare create with no such operation still succeeds (the network-only sandbox is created), emitting structured output when requested and detaching otherwise. Signed-off-by: Russell Bryant <rbryant@redhat.com>
The e2e:kubernetes:proxy-pod task ran the generic Kubernetes suite, whose smoke test execs a command and reads its output -- capabilities proxy-pod lacks, so the suite could not pass. Add tests/proxy_pod.rs (feature e2e-kubernetes-proxy-pod) covering the topology's actual contract: a workload whose entrypoint is set through containers.agent.command reaches Ready, and relay-backed operations (exec) are rejected with a topology-specific error. Scope the task to this suite instead of the incompatible generic one. The NetworkPolicy egress boundary is asserted at the unit level in the driver and validated manually on a policy-enforcing cluster; a self-probing egress e2e needs a workload image that tests its own egress and reports through 'openshell logs', tracked as follow-up. Signed-off-by: Russell Bryant <rbryant@redhat.com>
Update the topology diagram, NetworkPolicy contract, and validation table now that the supervisor no longer forwards a raw gateway tunnel, and note in the credential-isolation section that the workload has no network path to the gateway at all. Signed-off-by: Russell Bryant <rbryant@redhat.com>
… template The companion resources are named from the Sandbox CR name, but the workload pod template still derived the CA secret mount and HTTP_PROXY Service name from the bare sandbox name, and the create-rollback path cleaned up bare-name resources in the static namespace. In shared mode the workload then mounted a CA secret that did not exist and never became Ready. Thread the CR name through SandboxPodParams and use it for the pod template's companion references and the rollback cleanup, matching create_proxy_pod_resources. Caught by re-testing a shared-mode create on the cluster. Signed-off-by: Russell Bryant <rbryant@redhat.com>
11 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds the Kubernetes
proxy-podsupervisor topology: network enforcement andgateway forwarding move out of the sandbox pod entirely and into a paired,
per-sandbox supervisor
Deployment. The sandbox pod runs the agent imagedirectly — no supervisor binary, no gateway credentials, no privileged init
container, no shared process namespace. Egress is fenced by two per-sandbox
Kubernetes
NetworkPolicyobjects rather than by pod-local nftables rules.The result is the least-privileged sandbox pod any OpenShell topology produces:
runAsNonRoot, all Linux capabilities dropped, no added privilege at any layer.That is what makes it the first topology to run on OpenShift under a stock,
Red Hat-shipped SCC (
nonroot-v2) with no bespoke security grant.This continues @TaylorMutch's original proxy-pod work in #2077 (and the
earlier #2016), rebased onto current
mainwith the correctness,OpenShift-enablement, readiness, and workload-command work needed to make the
topology usable. It is the sibling of the
cni-sidecartopology in #2606.Topology tradeoffs (combined / sidecar / cni-sidecar / proxy-pod)
The topologies trade privilege against capability. The dividing line for
proxy-podis consistent: everything enforceable or observable at the networkboundary is retained, and everything requiring visibility inside the
workload's namespaces is given up.
combined(default)sidecarcni-sidecarproxy-podNetworkPolicyexec/ upload / sync0.0.0.0binds)openshell logsopenshell logsopenshell logsagentcontainer log (kubectl logs <agent-pod>), notopenshell logs-(0))NetworkPolicyenforcementprivilegedprivilegedCNInonroot-v2When to use which.
combinedstays the default and the only topology withthe full supervisor contract.
sidecar/cni-sidecarkeep that contract(including SSH, exec, and filesystem policy) while lowering pod privilege, at the
cost of a per-pod privileged init container or a node-level CNI DaemonSet.
proxy-podis for clusters that will not admit in-pod privilege at any level andfor workloads that need policy-enforced egress but never an interactive session —
batch jobs and autonomous agents that ship their own long-running entrypoint.
Operators who want the interactive workflow and low pod privilege on
OpenShift should use
cni-sidecar(#2606), notproxy-pod. The two arecomplementary, not competing.
proxy-podalso raises the isolation ceiling under a VM-basedRuntimeClass:because the workload and supervisor are in separate pods, Kata Containers
places them in separate VMs with separate kernels. A kernel compromise in the
workload VM does not by itself reach the supervisor or its gateway credentials.
In the in-pod topologies the two share one Kata VM, so the boundary between them
is a namespace boundary, not a hypervisor one. This is unique to
proxy-pod.Why SSH/exec are structurally impossible here, not merely unimplemented: every
relay targets something inside the sandbox, the SSH server lives only in the
process supervisor, and sessions need the workload's PID/mount/network
namespaces (
ssh.rscallssetnsto enter the sandbox netns). A supervisor ina separate pod holds none of those. Full analysis in the RFC.
RFC
rfc/proxy-pod-topology-DRAFT.md(included in this PR, unnumbered pending amaintainer-assigned number). It covers:
driving case.
proxy CA Secret, two NetworkPolicies), the privilege and credential-isolation
model, and the NetworkPolicy egress/ingress contract.
configurable (OpenShift runs DNS in
openshift-dnson container port 5353,not
kube-system/53), and why the built-innonroot-v2SCC suffices.supervisor to launch it — the two adoption blockers found during cluster
validation, with the fixes.
structural feature and telemetry losses, measured on-cluster.
Open questions.
The RFC records results measured against a live OpenShift 4.22 / OVN-Kubernetes
cluster, including the SCC split (the agent pod admits under stock
restricted-v2; only the supervisor needsnonroot-v2).Related Issue
Continues @TaylorMutch's proxy-pod PR #2077, which references #1827, #981,
#899, and #1305. Maintainers assign the RFC number from the originating issue
before it leaves draft.
Changes
Topology (builds on @TaylorMutch's #2077):
proxy-podsupervisor topology: per-sandbox supervisorDeployment,headless
Service, generated proxy CASecret, and agent-egress /supervisor-ingress
NetworkPolicypair, all owner-referenced to the SandboxCR for GC.
dropped; inject only proxy and CA-trust environment.
proxy_pod.proxy_uid/proxy_pod.affinityconfiguration and Helmvalues.
OpenShift enablement (new):
proxy_pod.dns_peers, Helmsupervisor.proxyPod.dnsPeers) with per-peer namespace/pod selectors andport, defaulting to the upstream
kube-systemconventions. An empty listis rejected, and an empty peer renders no rule rather than an allow-all rule.
nonroot-v2SCC grant (sandboxServiceAccount.openshift.nonrootSCC,default off) — a ClusterRole/Binding only, no custom SCC object.
Readiness and workload command (new):
SupervisorSessionModelon the driverDriverSandboxStatuscontract(
UNSPECIFIEDpreserves existing behavior). The gateway derives readinessfrom backend conditions when a topology reports it runs no in-sandbox
supervisor.
wait-for-proxyagent-pod init container (newwait-for-tcpsupervisorsubcommand) so pod readiness transitively means egress works.
containers.agent.command/argsvia the Kubernetesdriver_configpassthrough (no public API change), rejected in
combined/sidecarwhere thesupervisor is the entrypoint.
Correctness fixes (found during rebase and cluster testing):
proxy-pod(would landin the workload pod).
Deploymentto zero onstopand back onstart.(
default--rdyvsrdy) — the scale-down was silently patching anonexistent Deployment.
SupervisorSessionstatus condition, with an error-marker backstop) and print a clear
explanation instead of
ssh exited with status 255.Docs: RFC,
docs/kubernetes/topology.mdx,docs/reference/sandbox-compute-drivers.mdx,docs/reference/gateway-config.mdx,Helm README, and the
debug-openshell-cluster/helm-dev-environmentskills.Testing
cargo test --workspace --lib— 28 suites pass.cargo test -p openshell-driver-kubernetes --liband-p openshell-server --lib.mise run helm:test— 118 pass (proxy-pod DNS-peer, SCC-grant, and RBAC cases added).mise run pre-commit— clean.Ready;unproxied egress denied and proxied egress policy-evaluated (allow → 200,
deny → 403 at CONNECT) with the generated CA trusted; DNS resolves via
openshift-dns:5353;stop/startscale the supervisor down/up; the stockbase image runs via
containers.agent.command; all five per-sandbox resourcesGC'd on delete; both pods admit (agent under
restricted-v2, supervisor undernonroot-v2); relay RPCs rejected with a clear message rather than a timeout.Checklist