fix(core): fall back to podman CLI when no API socket is found - #1858
fix(core): fall back to podman CLI when no API socket is found#1858russellb wants to merge 1 commit into
Conversation
|
/ok to test e068662 |
elezar
left a comment
elezar
left a comment
There was a problem hiding this comment.
One question I have: This seems to fix detection, but doesn't change how the gateway interacts with the podman driver? Why are no changes needed there?
It's a fair question. I have podman on mac, but I don't have this problem. I was hoping the reporter would test this and see if it was enough. It seems like a reasonable change on the detection side. We go from "podman not detected at all" to either:
In either case, it's more consistent with docker in this part of the code. Docker does the same thing with falling back to a CLI check at this stage. A next improvement could be to discover the socket using cc @r3v5, reporter of the issue |
Hey @russellb ! Sure, I will test your fix, no worries. |
|
@r3v5 thanks. The output of that podman info command would be helpful too |
|
I have a couple of concerns / questions here. The first is the one that I've already mentioned. This change checks that Then, although this seems to align Podman functionality with Docker, there are subtle differences between the two paths. Although it is a slightly larger change than originally proposed, I think there is some benefit in trying to better align the detection paths for Podman and Docker. Ideally these would return a usable driver config (including, for example socket information) and not just a boolean. This config could then be used directly when instantiating the driver(s) instead of rediscovering the relevant config (as is done in the Docker case). |
|
Thanks, @elezar. I'm happy to work on the changes you described. |
|
I made a change to podman auto-detection recently (#1536), to avoid using just the existence of the CLI to determine that podman was available. In that change I made sure to align the auto-detection with the actual client socket choice mechanism. Podman unfortunately has different behavior on macos depending on how you install it, which I talked about in #1690 (comment) This PR could supersede #1690 (which is scoped to documentation), but it needs to keep the auto-detection mechanism and what the client uses to make the actual connection be aligned, like @elezar has raised. |
|
Great feedback, thanks @krishicks. I'll iterate on this. |
Hey @russellb ! I am coming back with results from local testing on my machine. Tested on macOS (Apple Silicon, M3 Pro RAM 36 GB), Podman 5.7.1 via Homebrew.Detection fix works — gateway now finds podman (Using compute driver driver=podman). Without this PR, it crashes with: I ran Gateway output with the fix: Socket mismatch — after detection, driver construction fails because default_socket_path() returns podman info output |
|
Perfect, thanks. This confirms the non-standard socket location and that discovery needs to include determining socket location to use. |
|
@r3v5 I've pushed follow-up commits that address the socket mismatch you confirmed. Changes:
Precedence for the socket path is: Could you re-test on your Homebrew Podman setup? The retry loop against the missing |
| Some(driver) => Ok(driver), | ||
| Some(detected) => Ok(detected), |
Hey @russellb ! Your fix is working, thanks! 1. No socket at the hardcoded path (confirms the bug) Input: Output: 2. Real socket discovered via podman machine inspect (confirms discovery works) Input: Output: 3. Gateway starts successfully with auto-detected Podman driver (fix complete) Input: Output: |
d9a1022 to
25e6db7
Compare
|
This pull request has had no activity for 14 days and is now marked stale. It may be closed in 7 days if there is no further activity. |
github-actions
Bot
added
the
state:stale
|
@russellb @maxamillion Any updates or discussion needed further on this PR? |
|
I was waiting for hear back from @russellb on this one. I thought maybe he was on PTO or something 🙂 |
|
Sorry, just missed the notification. I'm responding and rebasing now. |
ecb5e4f to
2a24776
Compare
|
I'm letting my agents fight over this for a bit. I'll comment again when it's ready for review. |
|
Sounds good @russellb - thanks for picking this one back up! |
e5b3f19 to
8329698
Compare
I think it's OK now. |
|
@krishicks and @russellb do we still want to get this in? |
I can look at the rebase next week. |
8329698 to
54179ac
Compare
Auto-detection only checked well-known Podman socket paths, so a Podman machine exposing its API socket at a non-standard location went undetected. The symlink at a well-known path is not always present; it varies by Podman version, machine provider, and platform. Extend detect_podman_socket() to fall back to podman CLI discovery when no well-known candidate responds: podman info --format json determines whether the service is local or remote, and podman machine inspect resolves the host-side forwarded socket for VM-backed machines. All existing callers (driver auto-detection, the Podman driver, and the VM driver's container-engine fallback) pick this up without change. Select the machine backing the active Podman connection instead of the first entry from podman machine inspect, honoring podman's connection precedence: CONTAINER_CONNECTION, then CONTAINER_HOST (mapped to a connection by URI), then the containers.conf default. An explicit endpoint that maps to no known machine is left unresolved rather than guessing an unrelated machine. When CONTAINER_HOST is an explicit unix:// socket, use that path directly since podman info connects through it. Update the gateway config reference and the Podman driver README, which described probe-only detection. Signed-off-by: Russell Bryant <rbryant@redhat.com>
54179ac to
dfc8150
Compare
|
@krishicks @johntmyers ready for another look |
|
@johntmyers did you close this on purpose? I think it was still relevant. |
|
Oops, no, got caught up in a stale batch! |
|
no worries! |
|
That being said I'll start a review agent on it if you're ready to finalize |
yeah, go for it |
johntmyers
left a comment
johntmyers
left a comment
There was a problem hiding this comment.
gator-agent
PR Review Status
Thanks @russellb—I checked the rebased implementation against the current callers, the linked macOS failure, the updated docs, and the connection-selection behavior. The fallback is project-valid, but two reachable startup defects need correction before pipeline handoff.
Action required: Please address both blocking findings and push an updated head for a focused follow-up review.
Blocking findings:
GATOR-dfc81508-01: Remote discovery can select a different Podman machine than the successfulpodman infoconnection.GATOR-dfc81508-02: Newly added Podman CLI probes can block gateway startup indefinitely.
Carried findings:
- None
Gator metadata
- Validation: Project-valid localized fix for linked bug #1834; issue #1690 is complementary documentation work, and merged #2327 enabled the current narrower implementation.
- Docs: Existing Fern gateway reference and Podman README are updated; navigation is unchanged and not needed.
- Checks: DCO and vouch are green; current-head Branch Checks and Helm Lint are pending.
- E2E:
test:e2eis required for compute-driver runtime discovery, but dispatch is deferred until blocking review feedback is resolved. - Head SHA:
dfc81508c752486b4aaff9e8a711369f20df02e9 - Base SHA:
2f7fb65591ee5746217ace154afc0aa6bf1afc6d - Merge base SHA:
dc374e88784a7e4a7366abd1fdb56c969efe6769 - Patch ID:
ff9f348ed171dc22e39bd07229f3ca1d3538aa56 - Gator payload:
7 - Review mode:
initial - Previous reviewed SHA: none
- Review budget exhausted: no
- Maintainer decision required: no
- Next state:
gator:in-review
| /// one — otherwise a host with multiple machines could be pointed at the wrong | ||
| /// machine's socket. | ||
| fn discover_podman_machine_socket() -> Option<PathBuf> { | ||
| let output = Command::new("podman") |
There was a problem hiding this comment.
gator-agent
Warning — GATOR-dfc81508-01 · Inspect the machine Podman actually selected
Summary: An operator with several Podman machines can make a non-podman-machine-default machine the default connection. podman info then reaches that machine, but no-argument podman machine inspect inspects only podman-machine-default; the later fallback accepts that first result. The gateway can therefore start against the wrong backend and create, inspect, or remove resources there.
Fix: Resolve the active machine first, pass its machine name to podman machine inspect (handling the generated -root suffix), and return None when a named default cannot be mapped instead of substituting another machine. Add an invocation-level regression test.
Verify: Shim podman info as remote, make connection work the default, and emulate no-argument inspect returning only podman-machine-default; discovery must never return that socket and must inspect work explicitly.
Agent context
- Location:
crates/openshell-core/src/config.rs:333 - Ownership: This PR changes bounded detection failure into a successful connection to a potentially different machine.
| /// 4. If `serviceIsRemote` is false, use `remoteSocket.path` directly | ||
| /// (on native Linux this IS the real local socket). | ||
| fn discover_podman_socket() -> Option<PathBuf> { | ||
| let output = Command::new("podman") |
There was a problem hiding this comment.
gator-agent
Warning — GATOR-dfc81508-02 · Bound Podman discovery subprocesses
Summary: During automatic driver selection, a stalled Podman machine, SSH connection, provider, or helper can keep podman info from exiting. This synchronous Command::output() has no deadline, and the same pattern is used for machine inspection and connection listing. Gateway startup then hangs indefinitely instead of trying Docker or returning an actionable configuration error.
Fix: Route all three discovery commands through one bounded helper with a documented startup-probe deadline. On expiry, kill and reap the child and return None so normal detection can continue or fail. Add a deterministic timeout-path test.
Verify: Put a blocking podman shim on PATH, invoke discovery with no responsive candidate sockets, and confirm it returns within the probe deadline, terminates the child, and permits the next detection outcome.
Agent context
- Location:
crates/openshell-core/src/config.rs:249 - Sibling sites: The new commands at lines 333 and 379 have the same unbounded wait.
johntmyers
added
the
gator:in-review
8 participants
Summary
Podman auto-detection only probed a fixed list of well-known socket paths. That socket is not always present — it varies by Podman version, machine provider, and platform — so on hosts where Podman is running and healthy but its socket lives elsewhere, the gateway failed to detect Podman at all.
This adds a Podman CLI fallback to
detect_podman_socket(): when no well-known candidate responds, ask Podman itself where its socket is.This repo's own e2e harness already documents the problem and works around it the same way. From
e2e/with-podman-gateway.sh:The harness shells out to
podman machine inspectto find the socket. This PR moves that same capability into the product so users don't have to setOPENSHELL_PODMAN_SOCKETby hand.Related Issue
Closes #1834
Changes
All in
crates/openshell-core/src/config.rs:detect_podman_socket()falls back todiscover_podman_socket()when no well-known candidate responds. Every existing caller — driver auto-detection (detect_driver), the Podman driver'sresolve_socket_path, and the VM driver's container-engine fallback — picks this up with no signature change.discover_podman_socket()runspodman info --format json. On a local service (native Linux) it usesremoteSocket.pathdirectly. On a remote service (macOS/Windows VM) it runspodman machine inspectto get the host-side forwarded socket, because theremoteSocketreported bypodman infois the VM-internal path and is not reachable from the host.CONTAINER_CONNECTION, thenCONTAINER_HOSTmapped to a connection by URI, then thecontainers.confdefault — rather than taking the first entry frompodman machine inspect. On a host with several machines, taking the first entry can point the gateway at the wrong machine's socket. An explicit endpoint that maps to no known machine is left unresolved rather than guessed; only the non-explicit default path falls back to the first entry, which is correct on the common single-machine host.unix://CONTAINER_HOSTis used directly, sincepodman infojust connected through it and a rawunix://URL has no machine to inspect.Docs:
docs/reference/gateway-config.mdxandcrates/openshell-driver-podman/README.mdboth described probe-only detection; updated to describe the fallback.Rebased on #2327
This PR predates #2327 and originally also changed
openshell-server/src/compute/driver_config.rs. #2327 removed the hardcodedsocket_pathdefault, made the fieldOption<PathBuf>, and addedresolve_socket_path(explicit config wins, else detect, else config error). That made the server-side changes here unnecessary — they existed only to stop auto-detection from clobbering an operator's explicit value, which a serde default made indistinguishable from "unset." That file is no longer touched and the net diff is smaller than before.Testing
mise run pre-commitpasses.podman infoparsing (local,unix://-prefixed, missing, empty),podman machine inspectparsing, active-machine selection (explicit, rootful-rootsuffix, unmatched-explicit, default fallback),CONTAINER_HOST/CONTAINER_CONNECTIONprecedence, andunix://URL parsing.cargo test -p openshell-core— 410 passed, 0 failed.HOMEwhere the well-known socket path does not exist, so only the new CLI fallback can find Podman:Using compute driver driver=podman,Connected to Podman cgroup_version=v2 network_backend=netavark rootless=true.sandbox createreachesReady;sandbox execreturnsHELLO_FROM_SANDBOX/aarch64/Ubuntu 24.04.3 LTSwith exit code 0;sandbox deletesucceeds.main: a gateway built frommainwithout this change, on the same host with the same environment andcompute_drivers = ["podman"], fails to start withno responsive Podman API socket found; set OPENSHELL_PODMAN_SOCKET or configure socket_path.libpod/_pingwith HTTP 200.e2e:podmanlane always exportsOPENSHELL_PODMAN_SOCKET(viaensure_podman_api_socket), which short-circuits detection as the first candidate, so it exercises no part of this change. Covering this automatically would mean a lane that deliberately hides the well-known socket path, which depends on host Podman state.Checklist