test(e2e): command-surface release gate (db/s3/expose/localstack) - #111
Merged
Conversation
Two runtime bugs surfaced while exercising the cloud/resource stack on WSL2. 1. shared-localstack "unhealthy after 1 attempt". LocalStack 3.x reports each configured SERVICE as "available" on startup — a service only flips to "running" after its first request. The healthcheck greps solely for "running", so nothing ever matches, the container is permanently unhealthy, and the up saga aborts before any traffic can flip a service to "running" (a deadlock). Gate on `available|running` — the honest "edge up + providers loaded" signal. Verified live on localstack 3.8.1: the new check exits 0, the old one exits 1. Golden regenerated through the real render path; regression test added (internal/generate). 2. `db create` → "connect to shared postgres on 127.0.0.1:45432: read: connection reset by peer". The imperative resource path (and the up provision phase) publishes the engine's host port via an up-time compose overlay, and `compose up -d` recreates the container to bind it. For a second or two the Docker userland proxy accepts the TCP connection but Postgres isn't listening yet, so it RSTs the handshake. A single immediate connect loses the race. Add retryingPgConnect: a capped-backoff retry (30s budget) around the admin connect that retries only transient "just-restarted" errors (reset/refused/ EOF/starting-up) and fails fast on real errors (bad creds/unknown db). Wired into both the imperative registry and the saga provision phase. Unit + race tested with an injected clock/connector (no live server needed). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…I clients `devstack shared expose [services...]` publishes the shared engines on stable 127.0.0.1 host ports so a developer's GUI clients — DataGrip/TablePlus, a Redis or S3 browser, the RabbitMQ management UI — can connect, without a duplicate stack per repo. `devstack shared ports` is the read-only projection (ports + connection strings); `shared expose --off` returns the stack to DNS-only. Design (spec 03 host-reachability): - Opt-in and loopback-only; the default is still "no host ports" (DNS over devstack_shared). Exposure is an UP-TIME compose overlay (compose.expose.yaml), so the deterministic, golden-asserted generated compose is untouched — same posture as the provisioning overlay. - Ports are ledger-allocated (FreeHostPort), stable across runs, and sit in a distinct 5xxxx range so the expose overlay and the 4xxxx provisioning overlay never publish the same host port (a duplicate binding). Kafka is the one deliberate exception: host clients must reach the fixed advertised 127.0.0.1:49092. - Persistent: the up saga re-applies the expose overlay so host ports survive up/down. Refused on a remote backend (a remote bridge is not host-routable). - Per engine it publishes the primary protocol port plus the useful secondary UI ports (MinIO console, RabbitMQ management, NATS monitor) and prints a client-ready connection string for each (the postgres admin DSN + a reminder that per-project DBs use the documented per-project dev creds). Tests: overlay write/read round-trip, instance resolution (all/named/rejects non-engines), per-engine connection URLs, the expose-vs-provision no-collision invariant, CLI table/JSON/quiet rendering, and command registration. Verified live on WSL2: `shared ports` projection, the overlay merges cleanly with the running shared compose (`docker compose config`), and a host psql client reaches shared-postgres on the published loopback port. Determinism/golden unchanged. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Owner directive: every command we built must actually work against a live stack
with solid tests before a release — the coverage that would have caught the
shared-localstack health deadlock and the `db create` connect race before they
reached main. Neither was caught by unit/golden tests because nothing drove these
commands end to end against a real daemon.
Adds tests/e2e/commands_test.go, two tiers:
- TestE2E_Commands_CoreResources (postgres + minio + redis): up (all three
health-gated) → shared status → db create/list/drop → s3 mb/ls/rb → resource
list → messaging degrades cleanly with no engine → shared expose + a real
TCP dial of the published port (what DataGrip needs) → down. Runs under
DEVSTACK_E2E=1 (per-PR CI) — the daily-driver commands.
- TestE2E_Commands_LocalStackHealthAndAws (adds localstack): the health-gate
regression (up must reach [ok], not the [failed] "unhealthy after 1 attempt")
+ the `aws` shim. Heavy image → gated on DEVSTACK_E2E_CLOUD=1 and run as a
nightly release gate (new nightly `e2e-cloud` job).
Stacks on #109 (health/connect fixes) and #110 (shared expose) — both are needed
for this gate to pass, so they are merged into this branch.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The release gate you asked for: every command we built must actually work against a live stack, with solid tests, before it reaches main — the coverage that would have caught the LocalStack health deadlock and the
db createconnect race that you hit by hand.What it drives (real binary, real daemon)
Tier 1 —
TestE2E_Commands_CoreResources(postgres + minio + redis), runs per-PR underDEVSTACK_E2E=1:up→ all three engines health-gated + app's postgres role/db provisionedshared status→ shows eachshared-*db create/list/drop→ the exact path that used to fail with the connect-reset races3 mb/ls/rb→ in-process aws-sdk-go-v2 (nomcneeded)resource list --json→ validqueue createwith no nats engine → fails cleanly (info-probe posture, not a crash)shared expose+ a real TCP dial of the published port (what DataGrip needs) →shared expose --offdownTier 2 —
TestE2E_Commands_LocalStackHealthAndAws(adds localstack), nightly release gate underDEVSTACK_E2E_CLOUD=1:upmust reach[ok](the health-gate regression — this was[failed] "unhealthy after 1 attempt")aws --shim resolves the endpoint + dev creds and lists bucketsCI wiring
DEVSTACK_E2E=1).e2e-cloudjob (DEVSTACK_E2E_CLOUD=1) — heavy localstack image, so it gates releases nightly rather than slowing every PR.Safety
The daemon e2e uses the pinned global
devstack-sharedproject/network and cleans up withdown -v, so it only runs on ephemeral CI runners (and the maintainer's VPS), never a developer's machine. Verified locally: compiles under-tags=e2e, the functional (non-daemon) tests pass, and the daemon tests correctly skip withoutDEVSTACK_E2E=1.🤖 Generated with Claude Code